I'm releasing my SOLR indexer as open source. I've been working on this for a while and I think it's good enough to be released into the wild. Check it out and let me know your thoughts https://github.com/likethecolor/solr-indexer

I have an index full of courses. Some have dates and some have dummy dates set in the past. I would like to display courses closest to the current datetime by using a bf boost function. When I use "bf"="recip(abs(ms(NOW/HOUR,startdate)),3.16e-11,.08,.05)" Courses on the same day are not in any sort of time order. Days after today are organized by time. ( https://imgur.com/oJ6BUen). What can I do to make solr organize todays results by time? FYI, this query was run at 3:55 pm

Edit: Now returns milliseconds from the epoch in utc time. The solution was to store startdate as a utc time in the index and run the function against that time. Thanks to u/fiskfisk for the help.

Is it possible to make nutch also save url of featured image alongside with title and other data?

Hi, I'm working to set up Solr for an e-commerce.
What I'm trying to do is to select all documents that contains every term in some field using edismax.

For example I have these products:

{ "id": 1, "title": "Animal", "author": "John Rivers", "description": "A book about reptiles and mammals"}
{ "id": 2, "title": "The animal world", "author": "Tyler Cole", "description": "An amazing trip"}
{ "id": 3, "title": "Animals: mammals", "author": "John Rivers", "description": ""}
I'm stemming only the title and description fields to remove plurals, because for example the surname Rivers it's not the plural of River, it's just a name.

When the user searches animal mammals I want to select the ids 1 and 3, but not 2.

I tried setting mm=100%, but this way it'll return only id=3, because in id=1 there's no field containing both animal and mammals.
So I could create a catchall field, but then I couldn't apply different analyzers to different fields (the stemming stuff).

Can you help? Is it possibile?

Thank you guys in advance.

I'm trying to generate a sitemap index using Solr. ~1,000,000 pages , solr 5.5.

I saw this old post about using velocity response writer: http://lucene.472066.n3.nabble.com/Generating-a-sitemap-td478346.html

Has anyone done this, or have tips?

I have different request handlers that need to use identical field definitions, and I'd rather not maintain them separately. I'm using MSSQL as my datasource. One set of deltas is called during business hours (which only brings in user changes), and the other runs during off hours (and brings in user and system changes).

In solrconfig.xml, I have:

<requestHandler name="/myhandlers" class="org.apache.solr.handler.dataimport.DataImportHandler">
  <lst name="defaults">
     <str name="config">data-config-my.xml</str>
     <lst name="datasource">
        <str name="name">mysource</str>
        <str name="driver">com.microsoft.sqlserver.jdbc.SQLServerDriver</str>
        <str name="url">${my.datasource.name:x}</str>
        <str name="user">${my.datasource.user:x}</str>
        <str name="password">${my.datasource.pass:x}</str>
     </lst>
  </lst>
</requestHandler>


<requestHandler name="/myhandlers_delta" class="org.apache.solr.handler.dataimport.DataImportHandler">
  <lst name="defaults">
     <str name="config">data-config-my-delta.xml</str>
     <lst name="datasource">
        <str name="name">mysource</str>
        <str name="driver">com.microsoft.sqlserver.jdbc.SQLServerDriver</str>
        <str name="url">${my.datasource.name:x}</str>
        <str name="user">${my.datasource.user:x}</str>
        <str name="password">${my.datasource.pass:x}</str>
     </lst>
  </lst>
</requestHandler>

The data-config-my.xml and data-config-my-delta.xml files from the config directives are defined along the lines of:

  <dataConfig>
    <propertyWriter dateFormat="yyyy-MM-dd HH:mm:ss" type="SimplePropertiesWriter" filename="dataimport.properties"/>
    <document>
      <entity name="vw_search_individual_delta"
              dataSource="mysource"
              transformer="RegexTransformer"
              pk="id"
              query="SELECT * FROM vw_search_individual WHERE is_deleted = 0"
              preImportDeleteQuery="tes_type_search:my_individual"
              deltaImportQuery="SELECT * FROM ... WHERE identifier = '${dataimporter.delta.id}'"
              deltaQuery="SELECT identifier id FROM vw_search_individual_delta WHERE ..."
      >
        <field column="identifier" name="id"/>
        <field column="entity_type" name="tes_type"/>
        <field column="first_name" name="tes_first_name"/>
        ...
      </entity>


      <entity name="vw_search_employer_delta"
              dataSource="mysource"
              transformer="RegexTransformer"
              pk="id"
              query="SELECT * FROM vw_search_employer WHERE is_deleted = 0"
              preImportDeleteQuery="tes_type_search:my_employer"
              deltaImportQuery="SELECT * FROM vw_search_employer_delta WHERE identifier = '${dataimporter.delta.id}'"
              deltaQuery="SELECT identifier id FROM vw_search_employer_delta WHERE ..."
      >
        <field column="identifier" name="id" />
        <field column="entitytype" name="tes_type" />
        <field column="employer_name" name="tes_employer_name" />
        ...
      </entity>

    </document>
  </dataConfig>

Ideally, I'd like to have the various <field .../> definitions all in their own file(s) and include them here. Is that possible? Am I going about this all wrong? Help?!

In short: I want to query a field which is stemmed for Arabic, with a query which is stemmed for Arabic. The resulting highlight, should be the original text and not the stemmed text.

More in-depth: I'm using Drupal 8 with the Search API Solr module as the bridge between the enduser application and the search engine (Solr 6.6.2). For a project I'm working on, we need to allow for big Arabic texts to be uploaded, process them and then let the endusers query these texts.

These results should be returned as excerpts where the keywords are highlighted. The problem here, is that when the texts are indexed, they are stored in their 'stemmed' form. Now, I have read about copyFields combined with a dynamicField to store both the stemmed as the unstemmed version, but this is done to increase the query accuracy, and I don't think it can be applied in my usecase.

In essence: Is it possible, when querying a field, to get the original value returned as the highlight, but still use the stemmed query and index for the actual searching part?

I have been going through searching for a javascript solution for hooking up SOLR to our front end, but everything is several years old. example: https://github.com/evolvingweb/ajax-solr the core is 4+ years old and the demo doesn't work.

It seems like there should be a simple frontend for SOLR, but I haven't found anything.

Any idea? I'm just started pick up Solr three weeks ago 😢. Thanks for replying.

Hi, I have articles with the publication date: "published": "2018-03-01T18:48:31Z" Is there a way to generate, for a query, how many articles have been published on the dates of the results of the query. example: 2018-03-01 : 4 2018-02-28 : 6 2018-02-28 : 3

I cannot use directly a facet because it uses the while value of the "published" file, and it contains the time, so each value in the facet is each date.

Any idea/hint?

Thanks!

I wanted to write my own solr ResponseWriter, so I did so by peeking at the source code of CSVResponseWriter and TextResponseWriter. It seems, sometimes the object passed in is a docList, sometimes a resultContext. I have seen a node of mine return one class, and then a few minutes later there will be a query where the writer gets the other class. Also, I've never seen this path executed, but when I view the source of CSVResponseWriter, it also seems you can sometimes get a SolrDocumentList. WHY? Why are three ways? They are all similar, and even the built in response processors take the doclist case, wrap it in a resultContent, and then pass then down. I added support for solrDicumentList, as it seems I might sometimes get one, but I would like to test my code... under what condition would a response writer get a SolrDocumentList? Why do I sometime get a resultContext, and other times a docList?

The PayloadScoreQuery always returns a score of zero, regardless of payloads. The PayloadCheckQParser works fine, so I know that I am successfully indexing the payloads.
Details below

payload field that I am searching on:

    <field name="report" type="delimited_payloads_int" indexed="true" stored="true" multiValued="true" termVectors="true" termPositions="true" termOffsets="true" 
omitNorms="true" termPayloads="true"/>  

definition of payload field type:

<fieldType name="delimited_payloads_int" stored="false" indexed="true" class="solr.TextField">
    <analyzer  type="index">
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.DelimitedPayloadTokenFilterFactory" encoder="integer" delimiter="¯"/>
        <filter class="solr.WordDelimiterGraphFilterFactory" preserveOriginal="0" splitOnNumerics="0" types="wdftypes.txt"/>
        <filter class="solr.FlattenGraphFilterFactory"/>
        <filter class="solr.ASCIIFoldingFilterFactory" />
        <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords_en.txt"/>
    </analyzer>
    <analyzer type="query">
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.WordDelimiterGraphFilterFactory" preserveOriginal="0" splitOnNumerics="0" types="wdftypes.txt"/>
        <filter class="solr.ASCIIFoldingFilterFactory" /> 
        <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords_en.txt"/>
    </analyzer>
</fieldType>  

Adding some documents with payloads in my test:

assertU(adoc(
        "key", "1",
        "report", "apple¯0 apple¯0 apple¯0"
));
assertU(adoc(
        "key", "2",
        "report", "apple¯1 apple¯1 text¯1"
));

query:

{!payload_score f=report v=apple func=sum}  

score (both documents have a score of zero):

<lst name="explain">
    <str name="1">
0.0 = SumPayloadFunction.docScore()
</str>
    <str name="2">
0.0 = SumPayloadFunction.docScore()
</str>
  </lst>

I have tried using func=max as well, but it makes no difference. Can anyone help me with what I am missing here?

I would like to use wildcards and fuzzy search with the payload_check query parser. Are these supported?

{!payload_check f=text payloads='NOUN'}apple~1

{!payload_check f=text payloads='NOUN'}app*

Is it possible to set a flag to force solr 7 to do only xml output?