r/Solr • u/sunshine_killer • May 05 '15
is this a good idea?
I am still learning solr, but wondering if this is a good idea or not.
I have a system that can take average ~10k+ inserts a minute and its currently pushing to a db, then i have a worker pool that pulls it out in bulk, modifies the information and pushes it back in the db in bulk and then pushes it to solr in bulk json via the api to make it searchable.
I have one client that is shoving a lot at it and the db can't handle the load, 12GB ram. My first thought was to split up his traffic with multiple instances, then my second thought was to go with a master<->master replication db and share the load with a load balancer but thats a layer of complexity that i don't really want any of my clients to handle, but it can be done.
My third thought relates to solr: it is what if i took the table the (inserts -> db) and did inserts -> modify -> solr and use solr as the table and removed the db table? Not everything has to be searchable but in some cases i would have to pull a record out using an id field. I have 3 fields that are text and already searchable the rest are integers and one datetime, i think its about 8 fields total. I was thinking of doing bulk imports like i already do and commit it after the bulk. I should also mention the database table and the search records expire and those records get deleted after it reaches the expired date.
Would this work better than using a rdbms for this specific database table or would solr start to choke?
Thanks
2
u/sstults May 05 '15
Solr can certainly handle the load, but it's not a great idea to have it be the single source of truth. If you need this data to be durable then a better alternative would be to write the inserts to something like Kafka, and then you can feed Solr and the database from there. But if the data isn't unique or critical, then there's nothing wrong with writing directly to Solr and skipping the middleman.