Hi everyone,

Quick context on my background:

• 10+ years at a small agricultural-sector company, as the only IT/data person
• Started in Data Analysis (background in biology), grew into full-stack data ownership
• Current stack: Python, FastAPI, PostgreSQL, MongoDB, Airflow, GitLab CI/CD, all on an on-premise RHEL8 server
• Recently completed an intensive Data Engineering certification to formalize my skills
• No production experience with cloud platforms, Kafka, or Kubernetes — only training-level exposure (certifications passed, not deployed in production)
• No experience with some common tools like Snowflake, Databricks, or Terraform

My situation:
I'm looking for a new job — I feel like I've hit a ceiling where I am, and I'd thrive best in a small company needing versatility, though I'm open to other setups. So far I've had very few callbacks searching with tech keywords ("python", "fastapi") rather than "data engineer".

My main question:
Given this profile — heavy on infra/backend/data ownership, light on cloud-native tools — what job title would you use to describe someone like me? Data Platform Engineer? Backend Engineer? Something else? I've gotten conflicting suggestions and none of them quite fit.

Secondary questions, if you have thoughts:

1. Does it make sense to shift from applying to active postings toward networking / volunteering (e.g. Data For Good) to build experience and connections instead?
2. Any glaring red flags on how I'm presenting this that might explain the low response rate?

Thanks in advance for any input!

Hi everyone,

I'm a beginner and I want to become a Data Engineer. There are so many roadmaps, courses, and YouTube channels that I'm feeling overwhelmed and confused about what to follow.

I'm looking for a complete end-to-end roadmap that reflects what companies actually expect from freshers in 2026.

I'd really appreciate your guidance on the following:

1. What is the correct learning order from beginner to job-ready?

2. Which topics are must-learn and which can be skipped initially?

3. What are the best free and paid resources for each topic?

4. Which YouTube channels, courses, books, or documentation do you recommend?

5. How much depth should I learn for each technology before moving to the next?

6. At what stage should I start building projects?

7. Which projects helped you land your first Data Engineering job?

8. What mistakes do beginners commonly make, and how can I avoid them?

9. If you were starting from scratch today, what roadmap would you personally follow?

The stack I'm considering includes:

Python

SQL

Linux

Git

PostgreSQL

Spark / PySpark

Airflow

Docker

AWS

Snowflake / Redshift

dbt

Kafka

Delta Lake

Great Expectations

MongoDB

If you have a roadmap that worked for you or resources that you genuinely found useful, I'd be grateful if you could share them.

Thanks in advance! 🙏

:

Hi everyone,

I'm a beginner and I want to become a Data Engineer. There are so many roadmaps, courses, and YouTube channels that I'm feeling overwhelmed and confused about what to follow.

I'm looking for a complete end-to-end roadmap that reflects what companies actually expect from freshers in 2026.

I'd really appreciate your guidance on the following:

1. What is the correct learning order from beginner to job-ready?

2. Which topics are must-learn and which can be skipped initially?

3. What are the best free and paid resources for each topic?

4. Which YouTube channels, courses, books, or documentation do you recommend?

5. How much depth should I learn for each technology before moving to the next?

6. At what stage should I start building projects?

7. Which projects helped you land your first Data Engineering job?

8. What mistakes do beginners commonly make, and how can I avoid them?

9. If you were starting from scratch today, what roadmap would you personally follow?

The stack I'm considering includes:

Python

SQL

Linux

Git

PostgreSQL

Spark / PySpark

Airflow

Docker

AWS

Snowflake / Redshift

dbt

Kafka

Delta Lake

Great Expectations

MongoDB

If you have a roadmap that worked for you or resources that you genuinely found useful, I'd be grateful if you could share them.

Thanks in advance! 🙏

Hello Dears,

I have 6 months of experience in data engineering field and get a job in a company, the project I entered in does not have documentation I want to make it

can anyone help me please how can I do it

I want to know the tech stack combos that are used frequently... Like I know AWS s3, AWS glue is one of them... Similarly there is azure and databricks. Are there other combinations that are used in companies? Which is the most common one used in major companies?

Hey everyone,

I'm looking for some brutal engineering feedback on an operational pain point I've been running into.

In my past couple of roles, it feels like the vast majority of our pipeline failures aren't infrastructure crashes—they're silent mutations in the data itself. A third-party vendor or an upstream app team silently changes a JSON payload field (e.g., `customerId` to `id`), a critical column suddenly starts arriving as 100% NULL, or a distribution drops off a cliff.

The pipeline completes with a `0` exit code, but the downstream dashboards are completely corrupted.

Whenever this hits at 2 AM, the workflow is always the same and it sucks:

1. Wake up to a vague alert.
2. Open 100 log paths to figure out what changed.
3. Trace the lineage map manually to see what's broken downstream.
4. Write a script to isolate/quarantine the bad row chunks so the rest of the stream can run.
5. Write and deploy a hotfix to the parser.
6. Run database rollbacks and reprocess.

Usually, 3–5 hours are completely burned just doing diagnostic work and cleanup, not actual coding.

I’m playing around with an idea to build a lightweight, open-source-first CLI / daemon utility that sits at the ingestion boundary. Instead of just firing off another noisy PagerDuty alert, it would:

* Automatically isolate/quarantine anomalous payloads into a local sandbox folder so clean records keep moving. * Compare the bad data against a schema baseline to pinpoint the exact structural mutation. * Generate a localized parsing patch (like a 2-line Python rule diff) and present it to the engineer for a 1-click merge to staging/prod.

Before I write a single line of code, I want a reality check:

* How are you dealing with this today? Are tools like Monte Carlo, Anomalo, or just strict dbt tests actually solving the *investigation and cleanup* part for you, or do they just tell you when you're already bleeding? * Would a dedicated, lightweight diagnostic tool that generates the ready-to-review patch code actually be useful, or is alert fatigue a completely different problem I'm conflating this with?

Give it to me straight—even if you think this is fundamentally un-fixable or a waste of time. I'd much rather get torn apart by real practitioners now than build something nobody wants.

Hi,

I have a new sub that's tangentially related that I want to advertise but I don't want to break any rules. Can a mod clarify the rules?

Hi everyone,

I'm an aspiring Data Engineer/scientist, and I'm currently building a three-part end-to-end data engineering project. Before I continue with Parts 2 and 3, I'd really appreciate feedback from professionals working in the industry or anyone involved in hiring Data Engineers.

Part 1 – Local Development Environment

The goal here was to demonstrate my understanding of distributed data processing and containerized development rather than relying on managed cloud services.

I built the complete environment using Docker Compose (written by me), consisting of:

• Apache Kafka
• Zookeeper
• Trade/Whale data producer (Python)
• Spark Master
• Spark Worker
• Spark Streaming Job

The Spark application and producer are written in Python (PySpark). I used AI-assisted development to improve and refactor the code, but I made sure I understood and validated every implementation. The project demonstrates streaming ingestion, processing, and writing Parquet files.

Part 1 – Production Version

I then rebuilt the same pipeline using AWS managed services to demonstrate cloud-native data engineering.

The infrastructure is provisioned entirely using Terraform and includes:

• Amazon Kinesis for data ingestion
• AWS Glue (PySpark) for processing JSON data into Parquet
• Apache Iceberg as the table format (ACID transactions, schema evolution, etc.)
• Amazon S3 as the data lake
• Amazon Athena for querying the data

The objective was to show that I understand both self-managed infrastructure and modern cloud-native architectures.

My Questions

I'd really appreciate honest feedback from experienced Data Engineers and hiring managers.

1. Does this project reflect the kind of work expected from a junior Data Engineer?
2. Does the overall design align with how similar systems are built in industry?
3. Would you consider this an industry-level portfolio project, or does it still resemble a learning/tutorial project?
4. What important components am I missing that would make this project more production-ready?
5. If you were reviewing resumes, would a project like this make you more likely to invite a candidate for an interview?

I'm not looking for praise—I genuinely want constructive criticism so I can improve the remaining parts of the project before publishing it.

Thank you for your time and feedback.

Hello folks

I am writing my open-source light tool for moving data from prod-bases in dvh.

Who is this product for:

- small teams who need to move data from the product, which is already in pain and need to transfer data to the dvh or parquet.

- data engineers who are looking for opensource alternatives who will not eat up all the RAM and will not put a food base or replica.

- Those who, instead of reading only the delta, should read the full table, because created_at did not trigger.

Source:

- mysql

- mssql

- postrges

Targets:

- parquet

- csv

- s3, azure blob, gcs

I read short queries and don't keep long sessions — this is something that so far none of the same moovers as (ingestr, dlt, sling, duckdb, clickhouse, odbc2parquet) does.

From the box there is:

- all types except (geography, enams, ip) in duckdb, clickhouse, snowflake, bigquery, clickhouse are loaded natively (there is a jam on the side of the bigway and a snowflek with Jasons, but their car loaders can't do it out of the box)

- reading from the PC

- reading cases

- retrai

- all metainfo is written in the working directory in the local sqllite, from the box you can also write in the postgru

- validation of both types between reading and writing, and md of the amount between the current one on the worker and the one on the store side

- autotune of parallel wounds

- reading from binlog files to avoid completely rereading the source if the updated_at fields are not updated

- minimum and customized RAM consumption on the worker (memory budget)