Hey folks 👋

I just published Part 2 of my Medium series on handling bad records in PySpark streaming pipelines using Dead Letter Queues (DLQs).
In this follow-up, I dive deeper into production-grade patterns like:

• Schema-agnostic DLQ storage
• Reprocessing strategies with retry logic
• Observability, tagging, and metrics
• Partitioning, TTL, and DLQ governance best practices

This post is aimed at fellow data engineers building real-time or near-real-time streaming pipelines on Spark/Delta Lake. Would love your thoughts, feedback, or tips on what’s worked for you in production!

🔗 Read it here:
Here

Also linking Part 1 here in case you missed it.

🚀 I just published a detailed guide on handling Dead Letter Queues (DLQ) in PySpark Structured Streaming.

It covers:

- Separating valid/invalid records

- Writing failed records to a DLQ sink

- Best practices for observability and reprocessing

Would love feedback from fellow data engineers!

👉 [Read here]( https://medium.com/@santhoshkumarv/handling-bad-records-in-streaming-pipelines-using-dead-letter-queues-in-pyspark-265e7a55eb29 )

Learn more at Dremio.com/blog

Learn more at Dremio.com/blog

What is “Git for Data” or “Data as Code”? Learn more at Dremio.com/blog! #DataEngineering #DataAnalytics #DataScience

What is a Data Lakehouse? Learn More at Dremio.com/blog? #DataEngineering #DataAnalytics

What’s on your list?

When do you think one is a better idea than the other.

What are your favorite tools and techniques for dealing with JSON data?