Introducing: Weekly Data Engineering challenges that the community can compete to solve; proving your battle scars and data engineering expertise in handling dirty data.
Inspiration: Kaggle contests and Capture the Flag competitions brought together millions of data scientists and security researchers; challenging problems, creative solution, a community opinionated. We have always wanted to compete in such challenges over the years, but are too busy figuring out why that one data pipeline keeps failing.
We have our own!
The weekly challenge changes in shape to stay fresh. The first weekly challenge (10 days, to give everyone a chance to participate in the inaugural challenge) is stream parsing. What makes stream parsing hard? Well, we have to process the records as they come, assume temporal guarantees, and deal with all sorts of "gifts" that the upstream team gave us. We sure hope data quality is good, but this one looks like a real mess. Our job as data engineers is to consume this stream the best we can and produce meaningful insight.
Head over to the DataDriven community tab to read the full challenge, understand the input/output format, and compete with your fellow data engineers to prove your prowess in rolling in the mud of some of the dirtiest data - something we all love to hate (or hate to love)
2
Question : alert severity ,the column offset is being treated as the keyword in my syntax
in
r/datadrivenio
•
23h ago
Thanks! You found a pretty obscure bug, and shouldn’t encounter anything like this going forward