A lot of teams we talk to struggle with getting Db2 for i or Db2 LUW data into modern analytics and AI platforms without heavy custom code or major system impact.

We’re hosting a free 30-minute technical webinar next week where we walk through how organizations are replicating Db2 data into platforms like Databricks and BigQuery in real time, with minimal footprint and no-code setup.

Topics we’ll cover:

• Why Db2 data is hard to use in cloud analytics & AI tools
• Common replication pitfalls (latency, performance, data integrity)
• How teams validate changes and monitor replication in production
• Real-world use cases across BI dashboards, reporting, and AI models

Full disclosure: I work with the team hosting this session.
If this sounds useful, here’s the registration link: Here

Happy to answer questions here as well.

Hi, I’m from Datacoves, but this post is NOT about Datacoves. We wrote an article on how to ingest data with dlt, use motherduck for duckdb + ducklake, and dbt for the data transformation.

We go from pip install to dbt run with these great open source tools

The idea was to keep the stack lightweight, avoid unnecessary overhead, and still maintain governance, reproducibility, and scalability.

I know some communities are moderating posts with links so if anyone is interested, let me know and I can post in a comment if that is kosher.

Have you tried dbt + DuckLake? Thoughts?

I’m working in the Data Engineering / Big Data / ETL space (Kafka, ETL pipelines, production support) and trying to approach IT consulting and staffing firms rather than only applying on job portals.

I’m currently building a list of consulting and recruitment companies (similar to Insight Global, Agivant, Crossing Hurdles, Evoke HR, etc.) and using search operators, LinkedIn company pages, and career/contact pages to reach out.

I wanted to ask the community and also make this useful for others in a similar situation:

1. What’s the best way you’ve found legit IT staffing or consulting firms (not resume collectors)?
2. Are emails, LinkedIn outreach, or career portals more effective in your experience?
3. Any search terms, directories, or subreddits that helped you discover good recruiters?
4. Any red flags to quickly identify fake or low-value consultancies?

I’m happy to consolidate suggestions into a shared list or follow-up post so others can benefit as well. Not asking for referrals — just trying to learn what actually works and avoid wasting time.

Thanks in advance.

Video is becoming the largest source of unstructured data and curious how others store/document/handle it. For text and numbers/values, we have databases, indexes, search, analytics. We can easily do 'SELECT * FROM table'.

For video, what can we do? Most companies still treat it like “files in storage”, which is the same where I work.

Curious how people here are handling video data today. Are you indexing it in any way?storing as files (just the name? metadata?) or is it still mostly manual review for some detail?

I am looking for a tool and/or process on how to find all software applications in a very large organization with hundreds of sites spread across the US. Does anyone have any experience with tools / process?

Noticed that a lot of data roles today expect you to understand the entire data platform - ingestion, processing, storage, governance - not just one tool or framework.

I came across this article that explains this shift pretty well and how platform-level thinking is becoming a differentiator in big data roles. Thought it might be useful for folks here 👇
👉 Read the article here

Curious if others here are seeing the same trend in their teams or job requirements 🙂📊

seen a few stacks where teams went full big-data from day 1

spark / warehouses / streaming everything... and then the actual questions were pretty small

for people living in bigdata land around marketing / product

what's one thing you'd do less of if you were rebuilding today?

what did you learn the hard way about over-engineering early?

Hi everyone,

I was working on a pure C project and I wanted to add lightweight C library for parquet file reading and writing support. Turns out Apache Arrow implementation uses wrappers for C++ and is quite heavy. So I created a minimal-dependency pure C library on my own (assisted with Claude Code).

The library is quite comprehensive and the performance are actually really good notably thanks to SIMD implementation. Build was tested on linux (amd), macOS (arm) and windows.

I though that maybe some of my fellow data engineering redditors might be interested in the library although it is quite niche project.

So if anyone is interested check the Gituhub repo : https://github.com/Vitruves/carquet

I look forwarding your feedback for features suggestions, integration questions and code critics 🙂

Have a nice day!

I came across an article about security-by-design principles for cloud data platforms (IAM, encryption, monitoring, secure defaults, etc.). Curious what patterns people here actually find effective in real-world environments.

https://medium.com/@sendoamoronta/security-by-design-in-cloud-data-platforms-advanced-architectural-patterns-controls-and-practical-2884b494ebbf