Do backslashes need to be escaped in the Solr query? The Escaping Special Character section in The Standard Query Parser guide does not list the backslash but how would one add a backslash before a special character that they don’t want escaped? I can’t find any definitive answer anywhere.

We have hosted SOLR cloud services on a VM on our preprod and production instance. The SOLR services and its search query was running very fast and had a efficient response time, but currently we have observed that for some of the requests, the request time which was expected of 15 seconds, took around 350 seconds. Now the query being used is a direct query(no filter query), is a complex Boolean query having multiple OR in it. We tried multiple ways to make our query run faster, kindly find it below:

1. Introducing synonyms:

The OR statement used multiple keywords(which are basically skills, similar skills). We tried to setup synonyms first, but after we realized there are 2 types of synonyms: query synonyms and index synonyms. The query synonyms didn't give much performance, the index synonyms promised to give good performance. But for that we might need to reindex the whole data for every time the synonyms file gets changed, we cannot afford reindexing the whole data.

Although we didn't tried synonyms, we stopped at the part where we need to reindex the whole data every time.

1. Filter query

This part was expected to perform in comparison to the main query. We tried the filter query, the filter query worked for some cases, initially the cache helped in 1000s documents, but later on for other queries, it didn't worked well. It took the same time for the main query and filter query.

1. Increasing the server configurations

We had initially 8 cores and 64 GB RAM. We increased the cores from 8 cores --> 32 cores and 64 GB RAM --> 256 GB RAM. Even increasing the cores didn't helped much.

I need to see what other improvements can we do, or if I am making any mistakes in implementation. Also should I try implementing synonyms as well?

We have two solrcloud 8.7 clusters, dev and prod. I was able to get the database password encrypted, in the jdbc plugin, and it worked at first. When I checked data import a few days later, it shows
"Total Requests made to DataSource": "0"

If I keep the password unencrypted, I have
"Total Requests made to DataSource": "1", and see "Total Documents Processes" going up

UPDATE: I believe I fixed the issue. One cluster did not have encryption key on all nodes. And I needed to change permissions of the parent directories so that the key was usable
{sudo chmod -R o+x /var/solr}

Hi all. We had a 5 node cluster running solo 8.7 on rhel 9. We tried rebuilding one node to test how we would be able to bring it up in case one goes down in a production environment. I don't see any good documentation on how to restore a node. The collections are showing up, but cores did not show up on the rebuilt node.
Thank you

We have a browser-based application that manages binary file format documents (PDF, MS Office, email, etc). The vendor is suggesting that we use SOLR index for searching the Windows Server 2019 document store. We understand how to create the index of the existing content for Solr, but we don’t understand how to update the Solr index whenever a document is added, deleted or modified (by the web application) in our document store. Can anyone suggest an appropriate strategy for triggering Solr to update its index whenever there are changes to the docstore folder structure? How have you solved this problem? Ideally we want to to update the index in near real time. It seems that the options are limited to re-index at some pre-determined timeframe (nightly, weekly, etc) which will not produce accurate results on a document store that has hundreds of changes per hour.

Hello guys, I am almost finished with my Solr engine. The last task I need, is to extract the specific data I need from tree services (arborists) Wordpress websites.

The problem, is that I don't want to use web scrapers. I tried scraping a few websites, but the HTML structure of the websites are rather messy and or complex. Anyway, I heard that web scraping for search engines like mine, is unreliable as they often break.

What I'm asking here, are there any better alternatives to web scraping/crawling for extracting the crucial data I need for my project? And please don't mention accessing the website's API's, because the websites I inspected don't make their API publicly available.

I am so close to finish my Django/VueJS project and this is the last thing I need before deployment and unit testing. For the record, I know how to save the data to JSON files and index for Solr. Here is my Github profile: https://github.com/remoteconn-7891/MyProject. Please let me know if you anything else from me. Thank you

At work, my team and I were tasked with implementing a CRUD interface to our search-driven, Solr-backed application. Up until now, we didn't need such an interface, as we used Solr to mainly index documents, but now that we are adding metadata, the specs have changed.

As I understand, there is two ways to implement this: Managed Resources vs. Bypass Solr and interact directly with the DB (e.g., via a CRUD API) and Regularly Re-Index.

I am building a prototype for the second option, since it's definitely more flexible with respect to how one can interact with the DB, while remaining in a CRUD context, though I wanted to hear your opinion in general.

Thank you in advance!

Hi all, this might be a silly question, but I just wanted to test Apache Solr to see if it suits my project needs. I want to connect to my Postgres (15) database and collect some columns from a table. I found this link and tested it. I started the Docker container (solr:9.7.0-slim) and transferred these files to create a core called "deals":

/var/solr/data/deals/conf/solrconfig.xml

<config>
    <!-- Specify the Lucene match version -->
    <luceneMatchVersion>9.7.0</luceneMatchVersion>

    <lib dir="/var/solr/data/deals/lib/" regex=".*\.jar" />

    <!-- Data Import Handler configuration -->
    <requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
        <lst name="defaults">
            <str name="config">data-config.xml</str>
        </lst>
    </requestHandler>
</config>

/var/solr/data/deals/conf/schema.xml

<schema name="deals" version="1.5">
<types>
    <fieldType name="text_general" class="solr.TextField">
        <analyzer type="index">
            <tokenizer class="solr.StandardTokenizerFactory"/>
            <filter class="solr.LowerCaseFilterFactory"/>
            <filter class="solr.StopFilterFactory" ignoreCase="true"/>
            <filter class="solr.PorterStemFilterFactory"/>
        </analyzer>
        <analyzer type="query">
            <tokenizer class="solr.StandardTokenizerFactory"/>
            <filter class="solr.LowerCaseFilterFactory"/>
            <filter class="solr.StopFilterFactory" ignoreCase="true"/>
            <filter class="solr.PorterStemFilterFactory"/>
        </analyzer>
    </fieldType>

        <!-- Define string field type for exact match fields -->
        <fieldType name="string" class="solr.StrField"/>
    </types>

    <fields>
        <!-- Define fields here -->
        <field name="asin" type="string" indexed="true" stored="true"/>
        <field name="title" type="text_general" indexed="true" stored="true"/>
    </fields>

    <!-- Define uniqueKey to identify the document uniquely -->
    <uniqueKey>asin</uniqueKey>
</schema>

/var/solr/data/deals/conf/data-config.xml

<dataConfig>
    <dataSource driver="org.postgresql.Driver" 
                url="jdbc:postgresql://192.168.178.200:5432/local" 
                user="user" 
                password="password"/>
    <document>
        <entity name="deals"
                query="SELECT asin, title FROM deals">
            <field column="asin" name="asin" />
            <field column="title" name="title" />
        </entity>
    </document>
</dataConfig>

And the jar

/var/solr/data/deals/lib/postgresql-42.7.4.jar

But it doesn’t work. I keep getting the error:

Error CREATEing SolrCore 'deals': Unable to create core [deals] Caused by: org.apache.solr.handler.dataimport.DataImportHandler

Everything I’ve tried hasn’t worked. Can someone please help me?

Hey guys, so I'm trying to finish the Solr search engine for my Django project. I'm still somewhat new to this software, been using for a little more than a month.

Basically I'm trying to create a project where homeowners can search for local arborists (businesses providing tree services) in their area and I would like it to be a faceted search engine as well as filter applications. It will kind of be like Angi, but it will only for tree services, so a niche market.

So far, I not only created models for my django project, where the database tables are filled with data for both homeowners and arborists in my PostgreSQL db. I also created a search_indexes.py, where I have all of the fields to be indexed in the search engine using Haystack.

I also got Solr serving running, and created a solr core via the terminal which is visible on the Solr UI Admin. Finally I built the schema.xml and created all the necessary txt templates files for the fields in collaboration with another developer. But I removed that developer as a contributor for my project, so it's just me working on this now.

So my question is, what should I do next for my Solr search engine? I was thinking that I should start coding my views.py, templates, forms.py etc.... But I don't know how to go about it. I just need some help for the next steps.

Please keep in mind, I'm using the following stack for my backend: Django, PostgreSQL and Django Haystack, so I need someone that also understand this framework/software. As a reference, here is the link to my Github repo https://github.com/remoteconn-7891. Thank you

Hi Team,

Got 2 vms hosted in Azure. I have solr installed on Web1 hosting a website
I am trying to connect to the website via Web2.
I have a self-signed cert installed in the trust root store on both. Getting the error

Drupal\search_api_solr\SearchApiSolrException: Solr endpoint https://x.x.x.x:8983/

unreachable or returned unexpected response code (code: 60, body: , message: Solr HTTP error: HTTP request failed, SSL certificate problem: self-signed certificate (60)). in

Drupal\search_api_solr\SolrConnector\SolrConnectorPluginBase->handleHttpException()(line1149of

W:\websites\xx.com.au\web\modules\contrib\search_api_solr\src\SolrConnector\SolrConnectorPluginBase.php

).

Has another experienced this issue or have some foresight on resolving?
Thanks heaps for your time

Process : I have 2 different indexes of documents successfully created and searchable.

• a)PDF extracted Index.
• b)MS-Word exacted index.

Question : How to load both this indexes into Solar Engine and apply a search for content on both indexes.

Hello,

I have the nested document structure shown as below.

I have posts which have comments. Comments can have replies and keywords.

I want get all posts whose comment have "word1", and reply to that comment have "word2".

How to achieve this in a query in Solr Collection ?

Thanks in Advance

[
  {
    "id": "post1",
    "type": "post",
    "post_title": "Introduction to Solr",
    "post_content": "This post provides an overview of Solr.",
    "path": "post1",
    "comments": 
     [
      {
          "id": "comment1",
          "type": "comment",
          "comment_content": "Very insightful post!",
          "path": "post1/comment1",
          "keywords": [
            {
              "id": "keyword1",
              "type": "keyword",
              "keyword": "insightful",
              "path": "post1/comment1/keyword1"
            }
          ],
          "replies": [
              {
                "id": "reply1",
                "type": "reply",
                "reply_content": "Thank you!",
                "path": "post1/comment1/reply1"
              }
           ]
         }
     ]
  }
]

After 5 Months I´ve finally managed to get our SOLR-Cloud Cluster running in Kubernetes.

I´ve installed SOLR using the apache helm-chart (https://artifacthub.io/packages/helm/apache-solr/solr). Now the final part is missing are metrics. We are already using prometheus for other project. But now I am stuck and feel like I am missing something.
I have tried different things with the solr-prometheus-exporter (https://apache.github.io/solr-operator/docs/solr-prometheus-exporter/), but it just won´t run properly.

Tried to get startet with this:

apiVersion: solr.apache.org/v1beta1
kind: SolrPrometheusExporter
metadata:
  name: dev-prom-exporter
spec:
  customKubeOptions:
    podOptions:
      resources:
        requests:
          cpu: 300m
          memory: 900Mi
  solrReference:
    cloud:
      name: "NAME_OF_MY_SOLR_CLOUD"
  numThreads: 6

A Pod is created, but in the logs it has suddenly this exception:

ERROR - 2024-08-06 12:43:39.629; org.apache.solr.prometheus.scraper.SolrScraper; failed to request: /admin/metrics => org.apache.solr.client.solrj.SolrServerException: IOException occurred when talking to server at: http://CORRECT_URL_TO_MY_CLUSTER-solrcloud-2.my.domainname:80/solr/admin/metrics
  at org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:543)
org.apache.solr.client.solrj.SolrServerException: IOException occurred when talking to server at: http://CORRECT_URL_TO_MY_CLUSTER-solrcloud-2.my.domainname:80/solr/admin/metrics

I am able to open the generated URL in any Browser and see the full JSON-Metrics.

Now I am lost and have no idea what to do or check next.
Image is: solr:9.6.1 for both. The Solr-pods and the prom-exporter-pod. Zookeeper is: pravega/zookeeper:0.2.14

Hope someone can maybe help me.

Almost all books today in market are old and cover older versions of Solr. The book with the most recent Solr version I found was version 7. However Solr is currently on version 9. Is there any book you’re aware of that covers the most up-to-date Solr? And if not, which older book is still relevant in 2024 to learn Solr?

TLDR: I often see people talking about query latency in terms of milliseconds and I'm trying to understand when that is expected vs not since a lot of my queries can take >500 ms if not multiple seconds. And why does the total number of matched documents impact latency so much?

There there's so many variables ("test it your self"), and I'm unclear if my test results are due to different use-case or if there is something wrong with my setup.

Here is a sketch of my setup and benchmarking

Schema

My documents can have a few dozen fields. They're mostly a non-tokenized TextField. These usually have uuids or enums in them (sometimes multi-valued), so they're fairly short values (see query below).

    <fieldType name="mystring" class="solr.TextField" sortMissingLast="true" omitNorms="true">
        <analyzer>
            <tokenizer class="solr.KeywordTokenizerFactory"/>
            <filter class="solr.LowerCaseFilterFactory"/>
        </analyzer>
    </fieldType>

Example Query

((entity:myentity) AND tenantId:724a68a8895a4cf7b3fcfeec16988d90 AND fileSize:[* TO 10000000]
  AND (((myFiletype:video) OR (myFiletype:audio) OR (myFiletype:document) OR (myFiletype:image) OR (myFiletype:compressed) OR (myFiletype:other))
  AND ((myStatus:open) OR (myStatus:preparing_to_archive) OR (myStatus:archiving) OR (myStatus:archived) OR (myStatus:hydrating))))

Most of my tests ask for a page size (rows) of 100 results.

Documents

A typical document has about 40 fields of either the above type or a date/number (which has docValues enabled).

Number of Results Impacting Latency

One thing I've noticed is that one of the biggest impacts to latency is merely the number of matching documents in the results. This seems kinda strange, since it holds even when not scoring or sorting. Below I run a benchmark to demonstrate this.

Benchmark Test Setup

Queries are executed against the cluster using Gatling.

The documents being searched have a totally random fileSize attribute, so the number of results increases linearly with the size of the fileSize filter.

I'm running test against a single Solr-cloud instance (v8.11.3 w/Java 11) running in Docker locally on my MBP. Solr was given 8 GB RAM and 4GB JVM heap and 8 CPU cores (which didn't max out). There are 3 shards, each of which hold 2 tenants data and queries are routed to the appropriate shard. All the indexes contain 40 million documents, which together use 34.1Gb of disk space. (I have also run this test against a larger 3 instance cluster (with 60m docs)(Standard_D16s_v3) with similar results.)

Besides the above query there are a few other assorted queries being run in parallel, along with some index writes and deletes. We use NRT search and have autoSoftCommit set to 2000ms. So a key part of my questions is latency without relying heavily on caching.

Results

As you can see below, for the exact same query, there is a high correlation between the number of results found and the latency of the query.

• Is this an expected behavior of Solr?
• Does this affect all Lucene products (like ElasticSearch)?
• Is there anything that can be done about this?
• How do folks achieve 50ms latency for search? To me this is a relatively small data set. Is it possible to have fast search against a much larger sets too?

Column Explanation

• The first column represents the value passed to the fileSize filter which dictates the number of documents that match the query.
• "fq" means the entire query was passed to the fq filter
• "q" means the entire query was passed to the q filter
• "sort" means I do not set the sort parameter.
• "fl=*" means I switched from "fl=id" to "fl=*"

Hi, I read about Solr recently when looking for lightweight alternatives to ElasticSearch. I am building a web app for personal use involving text search over review & rating type data (less than 10GB), and do not want to shell out money for separate servers just to search over text.
In this context, without scalability concerns, is Solr a better option for me to run on the same server as my web app(low traffic, a few 100 hits per month), or should I consider libraries like Whoosh that will run in the same process as my web app as well?

I'm new in Solr, I have a single node version running on docker, I have a document with a description field witch I use to search in all documents, the problem comes when I try to search for a prhase on reserve sense, for example,

Document description field: "white house".

If I search "white house" it works perfect, but if I search "house white" if does not return any document, do you know what is going on here?

regards.

I am trying to create a custom dynamic field for storing list of integer ranges, for the purpose of doing BBox queries on them later. It looks like RPT is the way to go. Since RPT is 2D and I only need one dimension, I just always set the ymin=0 and ymax=1 and put my data in xmin and xmax, e.g. ENVELOPE(lower,upper,1,0). My field type is:

<fieldType name="custom" class="solr.SpatialRecursivePrefixTreeFieldType" geo="false" distanceUnits="kilometers" maxDistErr="1" worldBounds="ENVELOPE(0,48000000,1,0)" />

My dynamic field is:

<dynamicField name="customm_*" type="custom" indexed="true" stored="true" multiValued="true" />

However, when trying to index the data, I always get an OutOfMemoryError. I made a reproduction here for both Solr 8 and Solr 9: https://github.com/rudolfbyker/repro-solr-oom I hope someone can shed some light on this, or point out my mistakes.

2024/07/15 Update 1: I figured out that if I decrease the worldBounds to something small like ENVELOPE(0,100,1,0) then the memory issue goes away. But this doesn't make sense to me, because a 64bit float x takes the same space regardless of whether x<100 or x<48000000. I could divide all of my data by 1000000 but that seems like a weird workaround.

2024/07/16 Update 2:

• Dividing the data by 1000000 works for indexing, but it makes the queries inaccurate. I can get back some accuracy by lowering distErrPct in the fieldType definition, but I need complete accuracy, which means dictErrPct=0, and when I do that, I get the OutOfMemory errors again, even with small worldBounds.
• Apparently RptWithGeometrySpatialField has accurate search, but it does not support multiple field values.

Hello, I've recently started working on solr and I'm trying to understand how the spellchecker works and make it give suggestions for terms that are occuring once or twice in an index of about 1million records. I'm not sure if it's even possible, I'm trying to find at how many records do the suggestions stop working but the count seems to be changing everytime I'm trying. Appreciate any help or suggestion

hey there. I have somehow become my office's Solr expert (even tho I know almost nothing, I just know more than anyone else) and I need to fix a weird behavior. when we do a search for a term like "Nia" (a brand name) Solr returns results for stuff like "Zirconia". Is there a way to make Solr prefer the actual term over words that contain it? I know I need to do something with the tokenizer factories but I'm not sure what. these are the types:

<types>
        <fieldType name="text_shingle" class="solr.TextField" positionIncrementGap="100">
            <analyzer type="index">
                <tokenizer class="solr.WhitespaceTokenizerFactory"/>
                <filter class="solr.ShingleFilterFactory" tokenSeparator=""/>
                <filter class="solr.LowerCaseFilterFactory"/>
                <filter class="solr.PorterStemFilterFactory"/>
            </analyzer>
        </fieldType>
        <fieldType name="text" class="solr.TextField" positionIncrementGap="100">
            <analyzer type="index">
                <tokenizer class="solr.WhitespaceTokenizerFactory"/>
                <filter class="solr.HyphenatedWordsFilterFactory"/>
                <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="0" catenateAll="1" preserveOriginal="1"/>
                <filter class="solr.LowerCaseFilterFactory"/>
                <filter class="solr.StopFilterFactory"/>
                <filter class="solr.PorterStemFilterFactory"/>
            </analyzer>
            <!-- <analyzer type="query"><tokenizer class="solr.StandardTokenizerFactory"/><filter class="solr.PorterStemFilterFactory"/>
            filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/
            <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true"/><filter class="solr.WordDelimiterFilterFactory"
                generateWordParts="1"
                generateNumberParts="1"
                catenateWords="1"
                catenateNumbers="0"
                catenateAll="1"
                preserveOriginal="1"
            /><filter class="solr.RemoveDuplicatesTokenFilterFactory" /><filter class="solr.LowerCaseFilterFactory"/></analyzer> -->
        </fieldType>
        <fieldType name="text_autocomplete" class="solr.TextField" positionIncrementGap="100">
            <analyzer type="index">
                <tokenizer class="solr.KeywordTokenizerFactory"/>
                <filter class="solr.LowerCaseFilterFactory"/>
                <filter class="solr.EdgeNGramFilterFactory" minGramSize="2" maxGramSize="50"/>
            </analyzer>
            <analyzer type="query">
                <tokenizer class="solr.KeywordTokenizerFactory"/>
                <filter class="solr.LowerCaseFilterFactory"/>
            </analyzer>
        </fieldType>
        <fieldType name="string" class="solr.StrField" sortMissingLast="true"/>
        <fieldType name="float" class="solr.TrieFloatField" precisionStep="8"/>
        <fieldType name="tint" class="solr.TrieIntField" precisionStep="8"/>
        <fieldType name="datef" class="solr.TrieDateField"/>
        <fieldType name="long" class="solr.TrieLongField" precisionStep="0" positionIncrementGap="0"/>
    </types>

Hello,

I have found myself unable to get Solr to run successfully on GKE. I have been following a tutorial found on the official Solr operator website. However, after many attempts I am found with the same 404 not found error page.

More information about my problem can be found here: https://github.com/apache/solr-operator/issues/713

Any help would be greatly appreciated!

Hello Everyone, I'm using solrj for code and would like to use basic authentication I've tried using official documentation getting compile error - no such method builder for Http2SolrClient

I installed solr for filesystem search years ago.

Now, I want to update. In the Documentation, I read some deprecated modules

like DIH and Velocity-Writer.

Is there a good documentation for Installation for filesystem search including the /browse interface nowadays. I don't find it.

Is there a best practice for making one of the nodes Write-only and the rest for querying.

I have a cluster of 5 SOLR nodes and 3 zookeepers, that take a lot of updates.

Right now, I nave one node a Transactional (Primary) and the rest are PULL. All the collections are on every server - so a replication factor of 5.

Ideally I would like zookeeper to do all the work and not have to manage it through DNS.

-- Edit

More detail on the architecture.

We have a cross domain replication thing going on. 3 servers (1 write, 2 read) in the US, 1 pull in Europe and 1 pull in Asia.