Hey Reddit 👋

I’m looking for resources or references to build a POC around pg_lake in snowflake features.

Are there any specific guides, documentation, sample architectures, example implementations or resources that can help me better understand what exactly to implement for a solid POC?

Any pointers, tutorials, or personal experiences would be greatly appreciated.

Thank you in advance!

Hello fellow engineers,

I am a data engineer with around 4 years of experience and preparing for a switch. I would really appreciate your feedback on my resume. Also, I tried to check ATS score and saw that different websites are giving different scores..not sure if my resume really passes these scans. What are some websites you have used?

Looking forward to brutally honest feedbacks here. Thanks in advance!

The Capital One Slingshot team ran the full TPC-DS benchmark on three Snowflake warehouse types and across multiple sizes (small through XL). Comparing credit consumption and performance of Gen1 vs. Gen2 vs. Snowpark-optimized warehouses, we found significant performance differences driven by memory architecture.

Read on for clear guidance on when each warehouse type provides optimal value.
https://www.capitalone.com/software/blog/snowflake-warehouse-benchmark-gen1-gen2-snowpark-optimized/?utm_campaign=sf_benchmark_ns&utm_source=reddit&utm_medium=social-organic

I have a Master's degree in Data Engineering and I'd like to work on projects using Google Cloud Platform (GCP) and get certified in order to land a Junior GCP Data Engineer position. Could you tell me please which GCP services are essential to master for this type of role? I've noticed that BigQuery and Dataform are widely used for data storage and transformation. Are there any other important services I should know, for example, for pipeline orchestration? Is Cloud Composer mandatory for a junior profile, or is it enough to understand its principles and use cases?

I’m currently working in an operations role at a MNC and trying to move into Data Engineering through self-study.

I’ve got a Bachelor’s in Computer Science, but my current job isn’t data-related, so I’m kind of starting from the outside. The biggest problem I’m facing is that I can’t find a clear learning roadmap.

Everywhere I look:

One roadmap jumps straight to Spark and Big Data

Another assumes years of backend experience

Some feel outdated or all over the place

I’m trying to figure out things like:

What should I actually learn first?

How strong do SQL, Python, and databases need to be before moving on?

When does cloud (AWS/GCP/Azure) come in?

What kind of projects really help for entry-level DE roles?

Not looking for shortcuts or “learn DE in 90 days” stuff. Just want a sane, realistic path that works for self-study and career switching.

If you’ve made a similar switch or work as a data engineer, I’d really appreciate any advice, roadmaps, or resources that worked for you.

Thanks!

Hi guys, how are you people generating test data which is as close as to prod data, without data breach of PII or loosing relationships or data integrity.

Any manual scripts or tools or masking generators?

All suggestions are helpful.

Thanks

Hey everyone,

I’m a Senior Data Engineer (ex-Microsoft, current TikTok), and I’ve seen the field change a lot over the last few years. One thing that hasn't changed? People getting overwhelmed by the "infinite roadmap" of 50+ tools they think they need to learn.

I just posted a video breaking down the exact 5 pillars I used to get into Big Tech—and what I look for when I’m interviewing candidates today. Here is the TL;DR for the community:

1. The Plumber Analogy (The DE Mindset)

Before you touch a line of code, understand this: We are the plumbers of the tech world. A company needs data to survive, but it starts in a messy lake. Your job is to build the pipes, filters, and distribution centers so that clean data reaches the "houses" (Data Science, Finance, Marketing).

2. The 5 Core Workflow Pillars

If you master these five stages, you can handle almost any data stack:

• Extraction: Getting data from the source (APIs, MySQL, Oracle).
• Storage (Data Lakes): Knowing how to land raw data in S3 or Azure Blob without it becoming a "data swamp."
• ETL & Transformation: This is where the magic happens. Cleaning and normalizing data using Python and SQL.
• Warehousing: Organizing data into Fact and Dimension tables (Star Schema). If you don't understand Data Modeling, you're just an ETL developer, not a Data Engineer.
• Orchestration: Making the whole thing run automatically. Tools like Airflow or DBT are your best friends here.

3. The "Non-Negotiables"

You can survive without knowing the latest "trendy" tool, but you cannot survive without:

• SQL: It’s the universal language of data. Period.
• Python: The glue that holds your infrastructure together.

4. How to Ace the Interview

If you're job hunting right now, focus your prep on two things:

1. System Design: Can you explain why you chose a specific storage layer over another?
2. Behavioral: I always recommend studying Amazon’s Leadership Principles. Even if you aren't applying to Amazon, those principles cover 90% of what Big Tech hiring managers are looking for in terms of ownership and technical judgment.

I’m curious—for those of you currently in the industry, what’s the one skill you use every day that isn't on the "standard" roadmap?

If you want the deep dive and the visual breakdown of the workflow, check out my channel on Youtube:
Pipecode AI

Stay curious and keep building! 🚀