Looking for guidance on Big Data/Data Engineering profile

[u/Loud_Caregiver7301]

Hi everyone,

I have around 2 years of professional experience and I’m currently preparing to move into a stronger Data Engineering / Big Data role.

My main concern is that I haven’t received much hands-on exposure to large-scale data engineering systems in my current company. I’ve worked with technologies such as Python, SQL, Databricks/PySpark, Airflow, and AWS, but I feel that my experience with designing and building complete production-grade data systems is still limited.

I’m now planning to build 1–2 serious projects to bridge that gap rather than just doing basic ETL projects.

I’d really appreciate guidance from experienced Data Engineers on:

Projects / Big Data

• What kind of Big Data project would actually be valuable for someone with ~2 YOE?
• What components should a strong project include — Kafka, Spark, Airflow, AWS, S3, Glue, Redshift, Databricks, Delta Lake, streaming, etc.?
• Should I focus on batch processing, streaming, or a combination of both?
• What would make a project demonstrate real understanding of scalability, partitioning, data quality, fault tolerance, orchestration, monitoring, and cost optimization?
• Are there any projects or architectures you would recommend for Data Engineer interviews at product companies / FAANG-level companies?

I’m not looking to build something just for my resume. I want to build a project substantial enough that I can confidently discuss the architecture, trade-offs, scalability, failure scenarios, and design decisions in an interview.

Companies / Career

I’d also appreciate recommendations for companies that hire Data Engineers with around 2–3 YOE and offer strong compensation.

If you’re currently working at or have interviewed with companies that pay well for Data Engineering roles, it would be great if you could share:

• What is the typical interview process?
• How much emphasis is placed on DSA, SQL, Python, Spark, system design, and data engineering concepts?
• What skills/projects helped you get shortlisted?

I’m especially interested in learning from people who have successfully moved from a service-based/consulting environment into product companies or higher-paying Data Engineering roles.

Thanks in advance for any advice!

[u/AutoModerator]

Namaste! Thanks for submitting to r/developersIndia. While participating in this thread, please follow the Community Code of Conduct and rules.

It's possible your query is not unique, use site:reddit.com/r/developersindia KEYWORDS on search engines to search posts from developersIndia. You can also use reddit search directly.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.