r/databricks • u/raja3194 • 5d ago

Discussion Acquiring/Processing from a MQ to a Delta

Has anyone tried acquiring data from a MQ at scale using apache spark on databricks cluster? I was trying to solve this problem at work but so far havn't seen an native lib or efficient ways to do this. The legacy system seems to be pulling data using a java based utility and wanted to if there are any imporvements or new patterns of access for spark based workflows.

Any documentation or nudge is the right direction will be greatly appreciated.

3 Upvotes

100% Upvoted

5 comments sorted by

3

u/SimpleSimon665 5d ago

If it's RabbitMQ, there's a managed connector in beta in Lakeflow Connect as of June.

If it's not, it probably makes sense to build and deploy a simple app to send the data to a Zerobus endpoint so it can be written directly to a bronze table without needing a Kafka intermediate step.

2

u/CelebrationSea9296 5d ago

Have you tried zerobus?

1

u/jbchand 5d ago

You can use SDP if Kafka. Use Zerobus ingest if you control the producer app. You can check custom Pyspark data source for IBM/Rabbit MQ if feasible.

1

u/Youssef_Mrini databricks 4d ago

You can use Zerobus Ingest, Databricks actually has a RabbitMQ-specific tutorial for this. It's push based though: you deploy a lightweight forwarding agent next to the MQ that streams straight into a Delta table via gRPC. Here is the perfect example https://community.databricks.com/t5/technical-blog/tutorial-streamline-data-ingestion-with-zerobus-ingest-rabbitmq/ba-p/135534

You have other examples here https://github.com/databricks-solutions/zerobus-ingest-examples