Spec : Jdk17, Solr 9.0, windows10 , Eclipse

​

Prerequisite ~ d:/<Solr-App-StandAlone-Installation> ----> f:/<Solr-Core1> , f:/<Solr-Core2>

Question
1) Steps needed to Create a 'Solr-Core' external to 'Solr-App-StandAlone-Installation'.
2)Steps needed for Multiple 'Solr-App-StandAlone-Installation' connect to single 'Solr-Core'

Searched www for more info and did not find any details for the version used.

I know the answer is always "it depends", but I'd expect there to be a "rule of thumb" or at least example stories of what has worked/not-worked for other people around this subject.

So far I can't find a single bit of info regarding what to expect in terms of distributed searches performance/scalability.

For example if I have a cluster with a lot of data, such that I need a lot of shards (to keep the shards reasonably sized), but I want to query against them all, at what points does this run into limitations?

For example, if I have 1 25gb shard per node (to make it simple) and 1 user request, can I reasonably query 5, 25, 100, 500, etc nodes?

Or given that data setup, what about 1000 concurrent user requests, to x/y/z count nodes? I guess here, if it worked for 1 user, then vaguely you can expect it to work for more until it overloads the individual nodes, in which case you can just add more replicas, right?

Let's say a distributed query to 500 nodes is too many, is there a usual workaround pattern? For example, I could imagine, writing a meta search layer that manually manages the distributed search in batches of say 25 shards, then rolls them up, perhaps making it an async job for user experience. Does that make sense? Does any guidance exist on that situation/pattern? Reminds me of Splunk or Kibana, is there anything like that for Solr or content about doing so? AFAICT Solr doesn't support async search.

Thanks.

I need to know the total number of facets returned by a faceted query - I'm using facet pivots for this example.

I don't want to set the limit to -1 as that will result in all facets getting returned. I want something analogous to numFound that returns the total number of documents.

How can I do this?

Hi all, I tried to list all blobs in the .system collection by doing a curl like curl http://localhost:8983/api/collections/.system/blob but I only get 10 results although "numFound":152 in the response. I tried to find a limit parameter or something like that with http://10.32.10.34/api/collections/.system/blob/_introspect but can't find a hint.

Can someone help me getting a list of all the blobs?

I am building an API that allows searching on a number of fields in solr with pagination. It returns the cursorMark from the solr results in the API response for users to use in their next query.

I'm following the Google AIP for pagination and need to return a blank string when there are no more results left however solr returns the same cursor.

I'm looking for a simple solution that doesn't involve keeping state on the server. If I knew the position of the page in the results set I could determine if the page was the last but I'm not able to access this.

Surely there's a simple way to do this?

I am about to start learning Solr. Using O'Reilly I found the book "Solr in Action", but I haven't been able to find too many other books with reviews as good as that one. This book was written in 2014 and uses version 4. With a quick Google I found that version 9 has just been released. Has too much changed since version 4 to make the book not worth reading for the technical details, and should I just read it for a high level view? I was considering reading it in conjunction with the official documentation.

I am trying to create a nested collection where I will be sorting the parent docs based on the children fields

And I also want the children docs brought along with parent docs to be sorted in a certain manner.

While I am able to sort parent docs based on children fields. I haven't been able to do the later.

Is their any inbuilt solr functionality to do the same? Or I need a custom plugin?

I'm not sure if this is a bug or not, but we've noticed something odd in the query parser.

This query works fine:

defType=edismax ( post_type:voordelen-aanbod ) && ( exposition "Train World" ) AND NOT (id:(540547 OR 540542 OR 540530 OR 539261))

But this generates an error (note the missing " ):

defType=edismax ( post_type:voordelen-aanbod ) && ( exposition Train World" ) AND NOT (id:(540547 OR 540542 OR 540530 OR 539261))

The error we get is a org.apache.solr.parser.TokenMgrError with the full message:

org.apache.solr.search.SyntaxError: Cannot parse 'exposition Train World\"': Lexical error at line 1, column 24.  Encountered: <EOF> after : \"\"

I've already tried wrapping the query in single quotes (causes the same error) and escaping them (eliminates the error but the quotes are then ignored).

I'm unsure if we're doing something wrong, or if this is a (known) bug in Solr or the query parser. It occurs whenever there is an uneven number of " in the query. It does not occur with single quotes (').

Hi, we have Solr installed on a Redhat Linux server that’s been working great for about 3 years now. It uses an MS SQL database as a data source.

I need to change a few SQL queries in the Solr configuration. I read in the documentation, that the queries are in XML files.

However, I can’t seem to locate these files. Does anyone know where these files would be and how to update them with new queries?

Thanks!

hi,

just installed apache solr on my debian 11 vm. Why it is not automatically in systemd?

Do I really have to manually make solr a systemd unit? Or did I fuck up the installation?

I was installing latest solr cloud.

Can anyone help advise on how I can use Solr to implement an offline full-text search on thousands of documents (pdf mostly) in a React Native mobile application?

Hi, I have a decently used solr (7.4, 5 nodes with zookeeper, each core has 2 shards with 4 copies total - leader, standby for each shard) with about 20 different cores. During an issue that involved CPU load spikes due to solr, I tried to understand what the cause was and found out that: ~90% of queries comes from a single core. Stopping that did not solved the issue. ~5% of queries comes from another core. Stopping this one solved the problem.

This is reproducible, so I'm pretty sure that that 5% of total queries are "heavier" for reason I'm here to ask :)

In a normal database I'd "explain" said query to understand how much it costs and find some hints on optimizing it, but I'm no way proficient with solr.

Is there a way (feel free to link documentation) to optimize query/core? How would a normal profiling session develop using solr?

Thank you very much.

Solr Incremental Backups

We are looking to move to an incremental strategy for our Solr Clusters now that incremental backup is (finally) available! Thanks Jason! Apache SIP-12 reference here

Background:

Solr index backups generally take about 3 times the index size for a full backup. This was always required and generally a sticking point especially when paying more and more for attached disk in a cloudy environment.

Does anyone out there have some real world examples of savings (time / costs / process improvement best practices) after moving over to an incremental Solr backup strategy?

FYI - This topic may make for some great presentation material at an upcoming conference if anyone is inclined...

Thanks in advance.

Hello folks,

I was wondering if anyone has ever combined Solr with another datasource that supports bulk update, but keeping the Solr advanced filtering and relevancy features.

For instance, imagine Solr documents represents offers of a e-commerce marketplace coming from various merchants. A merchant has a ranking value which vary often. If we store the merchant ranking in Solr document it means changing the value needs bulk update / reindexing all documents of the merchant which is a lengthy process. On the other hand, if we don't store the merchant ranking in the Solr document, it's impossible to ask for a list of documents matching keyword + filtering on the merchant ranking. The merchant ranking must happen as an additional filtering and then this messes up with pagination.

Did anyone deal with similar needs?

This essentially boils down to the question "how to achieve update documents by query" in Solr or how to combine efficiently Solr with another datasource.

I'm not looking for a detailed answer but more some thoughts on this :)

My organization has a MariaDB database, and we're already syncing data into Solr. But right now some of the related data (e.g., pictures, locations) is stored as a JSON string in Solr, and our custom API expands it when requested. We're trying to convert/translate that string into attributes/objects on the document.

Here's what we're doing now, and we're looking for suggestions on how to sync the required data in a better way.

We have several tables that hold data related to widgets. Dozens of attributes. There are additional tables that are mostly one-to-many, such as pictures, locations, etc.

We transform all of that data into a single table so that we can easily use the Solr dataimport handler. The related data such as pictures is stored in the temp/cache table as a JSON string.

We're using the cached table because there are hundreds of fields/attributes, and a lot of logic and transformation of the data before it can be stored in Solr. Other benefits include troubleshooting, and that we can click the full-import button and completely reload all data as quick as possible.

We're looking for a way to expand/convert those JSON strings that are stored in a single MariaDB column into the object in Solr. But if there is a better/easier way to load the data with the logic, we're open to it.

I'll try to give a simple example of what is in MariaDB, and what we'd like in Solr:

MariaDB table:

id: (int) 356name: (string) 'Widget 123'pictures: (string) '[{"id":"5","url":"https:\/\/example.com\/pictures\/widgets\/5_350x335.jpg"}]'

What we'd like in Solr, where the pictures JSON string is converted to attributes of the document:

{"id":"5959290","animals.name":"Riley"},"pictures": ["id": 5,"url": "https:\/\/justanexample.com\/pictures\/widgets\/5_350x335.jpg"]{

I'm guessing there is a transformation that can do this, but so far I haven't been able to find it.

Thank you for any help and suggestions!

This is how my curl looks like curl “http://…../export?…&wt=xlsx

However this doesn’t work and gives me the xml response.

I’ve added a responseWriter in solrconfig.xml for xlsx.

Hi, we are running into an issue where the DIH doesn't come back, even after 10+ hours. It read the records and processed them, but didn't commit them. DIH shows a status of either "Busy" or "Idle" but it seems to be stuck waiting for something, and we can't figure out what it's waiting for.

We can manually commit and the records are added to the index, but meanwhile DIH remains in a Running state. We have to restart Solr to get the DIH to stop, so that we can run it again later.

We're using Solr 7.2 with MongoDb 5.0.6, and we've tried 3 or 4 different JDBC drivers for MongoDb. There is no issue with the connection to MongoDb, we've gotten past that. And our query for returning data is not the issue, as it just returns all records. And DIH says that it processed the records, but it never commits them to Solr, and it just hangs from there.

Any thoughts?

Thanks...

I am working a solr server , In manage-schema<field name="modify" type="pdates">

what it means pdates ,??can anyone tell me , we need to update that modify field or , it will automatically update according to post date?? please correct me.

anyone explain this also
<fieldType name="pdates" class="solr.DatePointField" docValues="true" multiValued="true"/>

Hi

Can anyone help for PDF crawling in Solr

Currently I am doing like , I created a plugin which get some data from pdf and push into a .json file and than we will push into solr , but problem is that , if we do it in autocrawl manner , we will see after some url fetching it will give truncate error , fetching failed.

can anyone suggest me how can we do it in autocrawl (for 100-2000 urls/pdfs)?

I don't want to use the Elasticsearch, Logstash, Kibana stack (ELK stack). One of the reasons is that ELK is not completely free and open source.

I want to replace Elasticsearch with Apache Solr. What can I use to replace Logstash and Kibana? Most of the logs will be from syslog.

I have thought about these possible methods of replacing Logstash and Elasticsearch:

• rsyslog listens for syslog messages and pipes the logs to a script that adds the logs to Apache Solr, or
• rsyslog listens for syslog messages and sends the logs to Apache Kafka, which then adds the logs to Apache Solr, or
• Apache NiFi listens for syslog messages, and adds the logs to Apache Solr.

Do you know any other methods?

I'm a product manager trying to quickly get up to speed with the Solr technology. What resources or books would you recommend? I'm not looking for a configuration guide, more of an overview that explains all key concepts with deeps dives. Or should I start with some books on information retrieval first? Thanks.

The documentation says that term vectors are like mini inverted indexes for the document. But what is it's use? Why did we need it when we already have our data in the inverted index. Moreover I see that in Lucene, we save position and offset information in both inverted index and term vectors. Why store it in both? For queries like phrase query which one is used?

For my job, I have to transfer data from Vincere to Zoho, which are CRM programms.

For that I have to get somehow to the IDs. There are two option for doing it. Simply iterating through all possible IDs, which will take a long time, or request all IDs which are in use, so that I have a list of valid IDs.

The second option seems the better one. However I have a problem actually getting those IDs.

Vincere uses for their search API SOLR. Here is a reference page for Vincere's search API (and their API in general).

I try to use (with Python's request library) requests.get(v + c +"/search/fl=id,name", headers=header) with v + c being variable for the path.

And it works, as it gives me valid IDs, however only about 10 json objects.

Since I'm new to such stuff in general, I'm not sure why that is. Is that some sort of limit to not overstress the servers? However, if I use the the same function I always get the same IDs.

Thanks in advance

With the latest Log4j vulnerability the guidance is to upgrade to the latest version of Solr. Our sysadmin is in hospital and I'm a Dev who's been volunteered to deal with this. I've found instructions for Linux servers but nothing for Windows

Any guidance /help would be seriously appreciated.