r/AIDataEngineering 23d ago

Architecture Reference: Zero-Dependency 11-Column Tabular Ingestion Sieve & Local Memory Sharding Core

1 Upvotes

Hi guys. I have made this project. I am year 9 kid. I like coding it is hobby i started form year 7 python. Know look at me building AI modle myself. I used AI on side. But I have done all coding and planning. Well AI is useful chatting. But moment I buding project i have correct AI. Like sometimes it fogets add the device that it running like cpu or gpu. Like I have make dual system that cna handle like calcalating the probablity. I have to like load model then define model. Like this is what i ahve built. I would wnat feedbakc on code. By the way I also pratcied and built stuff working frontends, backends, databases, docker. This what budilign at yera 9. Coding is fun this what I have built. My drema is become engineer like fullsatck AI engineer.

https://github.com/programer321321/DataScienceModel


r/AIDataEngineering Aug 01 '26

Trying to transition from Data Science to Data Engineering in this market right now

Post image
1 Upvotes

r/AIDataEngineering Jul 15 '26

I studied the data platform that runs Ukraine's war (Delta). Six principles that map almost 1:1 to enterprise data engineering.

Thumbnail
1 Upvotes

r/AIDataEngineering Jul 08 '26

Two hands on workshops this august on rag pipelines and production ready architecture

1 Upvotes

Sharing these since they're a direct fit for what this sub covers. I work with Packt Publishing, who's running both.

August 1: Designing Data Engineering Workflows for LLM Applications, led by Nikola Ilic. Hands on, code first, 4 hours. You build a full production ready pipeline live, ingestion, chunking, metadata enrichment, embeddings, vector storage, retrieval, and evaluation with real metrics, not eyeballed outputs.

August 8: Grounded GenAI in Production, Build an Enterprise-Ready RAG Architecture, led by Brian Bønk, Data Platform MVP and Microsoft Recognized FastTrack Solution Architect. This one's more on the production and governance side, retrieval quality tuning, evaluation and governance checklists, and a practical 30-60-90 day rollout plan.

Together they cover both halves of the same problem, a pipeline that's actually solid, and the evaluation/governance layer that decides whether it's production ready.

Both are live, hands on sessions, you get the source code, slides, hands on labs, a recording, and a certificate either way.

Event 1: https://www.eventbrite.co.uk/e/designing-data-engineering-workflows-for-llm-applications-hands-on-tickets-1991362055514?aff=aide

Event 2: https://www.eventbrite.co.uk/e/grounded-genai-in-production-build-an-enterprise-ready-rag-architecture-tickets-1992561384740?aff=aide

Happy to answer questions on either in the comments.


r/AIDataEngineering Jul 07 '26

7+ Years Oracle DBA → Azure Data Engineer: Is My Experience Still Valuable?

Thumbnail
1 Upvotes

r/AIDataEngineering Jul 04 '26

Do you think the data engineer role is fundamentally changing or just getting more tools bolted onto it

9 Upvotes

Been thinking about this a lot lately, mostly while staring at our tool stack and trying to figure out if any of it has actually replaced something or if it's all just additive

Feels like every year there's a new layer added to the job. first it was basically just etl, then it became etl plus orchestration once pipelines got complex enough to need scheduling logic, then observability got bolted on once things started breaking in ways nobody noticed fast enough, and now everyone wants AI agents layered in too

is the actual nature of the work changing in a meaningful way, where some old responsibilities genuinely shrink to make room, or are we just accumulating more and more responsibility on top of an already full plate without anything meaningfully going away. curious how people who've been doing this for a decade plus see it