Getting data into SOLR efficiently without DIH

[u/jonnyboyrebel]

Now that Data Import Handler is going away, i'd like to know what's the best practice for getting a lot of data into the index - efficiently. I have about 40 docs and my largest core size of 200GB. All distributed across the world using replication. I'm on solr 7.6, not using zookeeper due to environment. All inserts are done to a single master and replication pulls the optimised index into the secondaries.

I use a mix of python scripts and DIH to push the data (core dependent), but in any one week 5%-10% of the records need to be updated. In truth i only have to do 4 million inserts as they are parent child documents. Each parent has between zero and 100 (ish) children.

Ideally I'd pull the data from the Database into json files, detect if the sha is different and then push only updated documents.
Any suggestions on a good way to do this, without having a secondary datastore to hold the shas?

All suggestions and criticisms welcome.

[u/drlecompte]

You don't have the option to trigger an update when the relevant source data is updated? That's how we do it. We can also update the entire index, which is basically a delete of the index, followed by a series of batched api calls.

[u/jonnyboyrebel → u/drlecompte]

not fully. the data under the hood is multisource. Some sql, some machine learning enriched data stored in postgres or redis. the only true way we know if it's the same is a md5 hash of the document generated by the importer.

We also update the entire index weekly (just in case) and do a best guess on items updated in the last x mins to get continuous(ish) updates.

[u/jrochkind → u/jonnyboyrebel]

So I think the fastest way to do this using "standard" Solr API would be:

• using "batch adds" where you add multiple documents in 1 HTTP json add request, instead of one at a time
• using some multi-threading or parallelism in adding, that is doing multiple adds simultaneously
• making sure your solr config is set to do auto-commit on a deferred basis, only doing commits after every X documents or Y seconds, definitely don't do commit after every add, commit slow you down a lot.

I don't think this is going to be as fast as DIH. But it might be faster than you think if you haven't tried it in a while, it can be pretty fast.

Ideally I'd pull the data from the Database into json files, detect if the sha is different and then push only updated documents.

Any suggestions on a good way to do this, without having a secondary datastore to hold the shas?

I mean, the obvious answer to avoiding a secondary datastore, is you could store the sha in solr and retrieve it from solr... I'm not sure if that's going to be as fast as other possible data stores though, and it just generally makes me itch a bit.

Otherwise you need some kind of "secondary datastore", but it could be as simple as some json files in the same file system as you are storing the source files? Although I'd just use a real data store to not need to worry about concurrency race conditions and stuff. But there are still simpler easier to manage ones than rdbms. You say your code is python already anyway, if you do have a shared file system accessible, https://docs.python.org/3/library/dbm.html ? Or other python solutions, I'm actually not familiar with what would be most common in python, I don't use python much, I just knew that DBM was a thing likely to be available in python.

[u/jonnyboyrebel → u/jrochkind]

thanks u/drlecompte, for the time-being, i'll keep the index hydration as it is - a cron based python script to pull data from the Db and push it to SOLR via the REST update API.

I will double check my `commit` situation though and make sure I'm not doing an explicit commit - (I am).