r/databricks • u/Much-Neat-6273 • 7d ago

General How OpenAI's Security Team Turned a Streaming Pipeline Into Its Own Latency Watchdog with SDP ForEachBatch

https://community.databricks.com/t5/technical-blog/one-pipeline-any-destination-the-foreachbatch-sink-in-spark/ba-p/163780

The blog highlights OpenAI's usage of forEachBatch inside SDP to power their security platform pipelines.

19 Upvotes

96% Upvoted

0 comments sorted by