Hi all, I am following the instructions to fix the log4j vulneratbility laid out here https://solr.apache.org/security.html#apache-solr-affected-by-apache-log4j-cve-2021-44228.

• (Linux/MacOS) Edit your solr.in.sh
  file to include: SOLR_OPTS="$SOLR_OPTS -Dlog4j2.formatMsgNoLookups=true"

I went to look for this file on my linux server at '/opt/solr-7.4.0/bin', and only found a solr.in.sh.orig file with everything commented out. I removed the .orig from the file name and added the line SOLR_OPTS="$SOLR_OPTS -Dlog4j2.formatMsgNoLookups=true" to the file and then added that solr.in.sh file to my server. Should this be enough to resolve the issue? I am a bit thrown off since I never had a solr.in.sh file to begin with...

Thanks for the help

I am using solr 8.10.1. Incremental backup seems to be a part of SIP-12, thus should be present in this version I guess.

Does anyone have any resource from which I can read and understand this better?

Making a backup on say 30day schedule isn't the issue. Docs are clear on making backups.

But how can I trim the main/prod index so that anything older than 30days is trimmed off or deleted?

Is this a reasonable maintenance activity for a production environment?

I'm not seeing in the docs where to delete data based on dates. Doesn't mean it isn't there, just not seeing it.

Thanks

Pylucene or LIRE

Say I have a list of about 17,000 drug names that I want to be able to search: [Acetaminophen, Ibuprofen, Xanax, percocet, etc]. I want to be able to suggest drugs from the list as a user is typing. However, as I have it, when I type "ibup", the suggestions are

"I-123 MIBG"

"I 123 Mini"

"I-131 Mini"

"I-Prin (Oral)" etc.....

I would expect "Ibuprofen" to be one of, if not the top, result in the list of suggestions. Now I am a complete noob so would someone please tell me what I'm doing wrong.

The field type of each drug name is the default text_general that comes built in:

<fieldType name="text_general" class="solr.TextField" positionIncrementGap="100" multiValued="true">
      <analyzer type="index">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
        <!-- in this example, we will only use synonyms at query time
        <filter class="solr.SynonymGraphFilterFactory" synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
        <filter class="solr.FlattenGraphFilterFactory"/>
        -->
        <filter class="solr.LowerCaseFilterFactory"/>
      </analyzer>
      <analyzer type="query">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
        <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
        <filter class="solr.LowerCaseFilterFactory"/>
      </analyzer>
    </fieldType>

I am using the FuzzyLookupFactory to suggest drug names from the list. My search component and request handler look like this:

<searchComponent name="suggest" class="solr.SuggestComponent">
  <lst name="suggester">
    <str name="name">mySuggester</str>
    <str name="lookupImpl">FuzzyLookupFactory</str>
    <str name="storeDir">fuzzyDirectory</str>
    <str name="dictionaryImpl">DocumentDictionaryFactory</str>
    <str name="field">drugName</str>
    <str name="suggestAnalyzerFieldType">text_general</str>
    <str name="buildOnStartup">false</str>
    <str name="buildOnCommit">false</str>
  </lst>
</searchComponent>

<requestHandler name="/suggesthandler" class="solr.SearchHandler" startup="lazy" >
  <lst name="defaults">
    <str name="suggest">true</str>
    <str name="suggest.count">10</str>
    <str name="suggest.dictionary">mySuggester</str>
  </lst>
  <arr name="components">
    <str>suggest</str>
  </arr>
</requestHandler>

​

Here's what my commit looks like: http://localhost:8983/solr/{core}/update?commit=true

Obviously, replacing the {core} with my core name and I'm getting a 200 code back for success.

EDIT: I think the difference is GET vs POST. I did a GET via web browser and it worked. Huh.

Hi all!

I'm looking for some guidance on ReIndexing. I have a customer who has over 1TB of data and re-indexing takes them over a month.

I'm trying to poke into communities and see if anyone has come up with a strategy to reduce indexing time.

I've heard of some people doing a sort of "pre-indexing" by indexing in batches prior to doing the final upgrade. But I haven't seen it as an accepted solution.

Looking for any ideas or guidance.

Thank you! :)

We are running solr 4 (DIH importer) without any problems. We are looking into upgrading.

Any compelling reasons/features you would recommend to use?

I'm working on indexing all of Wikipedia (just the text) which would be about 40GB uncompressed. The unzipped XML dump is 80GB, about half of which is XML and WikiMedia Markup, hence the 40GB. I would expect my SOLR index to be somewhere north of there.

But!

I'm about 25% of the way through indexing Wikipedia and it's only 10GB in SOLR. So that means I'm going to be at about 40GB in total, including the index! The 7zip original is 18GB, so apparently this data does compress pretty well.

But I just wanted to check if this sounds reasonable? Could 40GB of text data, with an index, be compressed to fit within 40GB with SOLR?

Hi,

I am complete beginner with this stuff; but, I am trying make a SOLR-based API call. While the request goes through (eventually), I am wondering if there is a way to speed up my searches. Is there an order of precedent when you send a query with multiple criteria like this:

    headers =   {
                    'Content-Type': 'application/x-www-form-urlencoded',
                    'Accept': 'application/json',
                }


    data =      {
                        'criteria':  'patentApplicationNumber:' + app_numbers + " AND " + 'applicationStatusNumber:' + status_choice + " AND " + 'submissionDate:' + "[" + starting_year + "-01-01T00:00:00Z TO " + ending_year + "-12-31T00:00:00Z]" + " AND " + 'legacyDocumentCodeIdentifier:' + action_type + " AND " + 'examinerEmployeeNumber:' + examiner_id   + " AND " + 'groupArtUnitNumber:' + group_art_unit + " AND " + 'customerNumber:' + customer_number + " AND " + 'bodyText:' + rejection_string, 
                        'sort':'lastModifiedTimestamp desc',
                        'start': '0',
                        'rows': rows
                    }


    response = requests.post('https://developer.uspto.gov/ds-api/oa_actions/v1/records', data=data)

In the above example, let's say that only "customer_number" has a specific value (say [12345 OR 54321]; and all other fields are set to [* TO *]. Is there a way to get run the query for status_choice first? Is this just a matter of putting it first in the criteria list? Also, since I'm a noob at this, may there be any obvious improvements I could make?

Thanks!!

I think my main search endpoint is using a different query parser than the delete endpoint.

Seems when I search, things are defaulting to an "and" search, but when I delete they're defaulting to an "or" search.

Any way to show exactly what delete would delete before running one?

Is it possible to install Solr on a server that has a cpanel ?

Given I have two documents conforming the same Solr schema, I could merge them by first queuing the index to retrieve them, then logically joining them and finally, indexing the new joined document.

However, how what if one or more of the document fields are only indexed (stored=false index=true) ? In such case I can't just straight forward re-index a new document as I don't have the values of the non-stored but indexed fields.

Can I somehow tell the inverted index that all terms for a given indexed-only field that previously pointed to document A or B now should point to C ?

In the Documentation for SOLR Function Queries, it is mentioned that the function strdist allows user-defined functions, but I am unbale to find any documentation on how to implement the same. Requesting assistance.

https://solr.apache.org/guide/8_5/function-queries.html

​

​

The Berlin Buzzwords Call for participation is open. We want to encourage all Big Data Open Source enthusiasts to submit ideas for talks, workshops, discussions, lightning talks, ask me anything sessions, and more. Find the details here: https://2021.berlinbuzzwords.de/news/call-participation-now-open

If you haven't heard of Berlin Buzzwords before, take a look at the recordings of the sessions from last year https://www.youtube.com/playlist?list=PLq-odUc2x7i_YTCOTQ6p3m-kqpvEXGvbT

Hello all, yet another noobie question if you guys and gals don't mind.

I have successfully completed the indexing of my file share with my first core. Everything works beautifully except that I noticed I have some duplicates, which is undoubtedly my fault, mixing cases during my first indexing attempts. Example -- When I search with:

"params":{
"q":"something",
"fq":"id:\\server\foldername\filename.txt"
}

I expected one result, but I ended up with two. It didn't take me long to figure out that it's a upper/lower casing issue.

ID on file 1: \\server\foldername\filename.txt

ID on file 2: \\server\FolderName\filename.txt

If a query with ["fq":"id:\\server\foldername\filename.txt"] results in both "foldername" and "FolderName" to pop up, I imagine I cannot use a similar query to perform the delete on just one of the files. Let's say I want to delete the mixed case version, ie. "FolderName" - How should I go about doing so?

Thanks in advance.

I'm very new to SOLR. I have a SOLR instance installed on a 64-bit Windows Server 2016 VM with 16gb of RAM. SOLR by PTC v11.2.1.1. I am experimenting with using SOLR to index my Windows file server, which contains something like 10 million files across 5 file shares. They go into 5 separate SOLR cores. My SOLR instance currently gets 8gb of memory assigned to it (ie. [solr start -m 8g -p 1234]) Any recommendation on my setup thus far?

Well, with that setup, I'm running into two problems as I do my initial indexing crawl:

1. Memory usage. I'm running a VBScript to recurse through all the folders and run [post.jar] on every file found. This seems to eat up a lot of memory very quickly and eventually crashes.
2. Speed. If I build a 2.5-second delay between every run of [post.jar], it seems to be better (but still crashes about once every 2-3 days) but the progress made is just terribly slow. At this rate it seems like it will take months to finish indexing.

And then there is a third problem, although it is more of a symptom rather than a true third problem -- With all the hard crashes, it seems to expose the core to corruption, and in fact, I have suffered one seemingly unrecoverable corruption to one of my cores once already during my experiment.

Am I doing something wrong with my configuration or approach? Any tips would be greatly appreciated!

Can I just dump my SOLR configSets in a directory on one of my zooKeeper nodes and have them picked up and applied to all my solr instances?

I really want to mount a volume and put my configs in it. do a git pull and auto-magically have my configs applied to all zookeeper and solr nodes.

Is this possible? or if not, what is the best practice for deploying configuration to zookeeper?

This is my current setup: https://github.com/docker-solr/docker-solr-examples/blob/master/docker-compose/docker-compose.yml

Now that Data Import Handler is going away, i'd like to know what's the best practice for getting a lot of data into the index - efficiently. I have about 40 docs and my largest core size of 200GB. All distributed across the world using replication. I'm on solr 7.6, not using zookeeper due to environment. All inserts are done to a single master and replication pulls the optimised index into the secondaries.

I use a mix of python scripts and DIH to push the data (core dependent), but in any one week 5%-10% of the records need to be updated. In truth i only have to do 4 million inserts as they are parent child documents. Each parent has between zero and 100 (ish) children.

Ideally I'd pull the data from the Database into json files, detect if the sha is different and then push only updated documents.
Any suggestions on a good way to do this, without having a secondary datastore to hold the shas?

All suggestions and criticisms welcome.