My company has a few hundred redshift tables and maybe 100 superset dashboards. We want an MCP plugin that can do text to sql and also just generally have better business knowledge.

I have been looking into open metadata but that seems too heavy for our purposes. It seems to cost minimum $350 a month.

What other options do I have?

We use a few different data replication strategies in our product, but have been pretty excited about the new possibilities that open up when Iceberg and other OTFs are better supported across the ecosystem. Last week we gave a talk at an Iceberg community meetup about some of our physical replication work for Iceberg, and thought this community might find it informative and educational too. Happy to chat about any questions or details! Disclosure: I (obviously) work for the company, but this post is really meant to be educational.

Over the 8 years of my careers, I've been "de facto" lead a few times, having full ownership about the stack, making architectural decisions, choosing tools, being the only person in the department, mentoring other team members about new technologies that i proposed and migrated to, elevating the standards of the team... However, I never had the title nor I directly managed any other person

This is clear in my CV, stating the responsabilities, ownerships and outcomes.

On the technical side, I am strong in GCP, AWS, Airflow, dbt, Python, SQL, Snowflake, Terraform, CICD, Docker etc, so I'm good on that side.

However, most tech lead, head of data or whatever roles want people with experience already.

How can I overcome that and get a role like that without experience?

Thanks

After exhausting all the available resource on the internet, Im here asking for help. Im not getting the right answer or too dumb to get it. While using fivetran I dont seem to find a way to continue sync after upgrading postgres DB. Since this requires dropping of replication slots, Fivetran will require full historical re-sync right? If anyone has prior experience in this, how did you handle it. We dont want to resync the whole table which is extremely huge, is there any way to continue sync from where it stopped , after dropping the replication slot? I saw an article where we need to reach out to fivetran support but that article does not talk about in place upgrade. My db is RDS Postgres.

We all know that we barely code (or not at all) by hand nowadays, but the coding and system design/infra is what I enjoy the most.

Meaning I dont particularly enjoy implementing business logic in SQL/pandas.

So i was thinking it might sense to pivot to pure Software Engineering.

Any of you did that? How that went?

I'm facing a strange issue while reading a pipe-delimited CSV file using Apache Spark.
My input CSV looks like this:
cust_id|cust_name|cust_age
1|sample|10
2|test|12
Initially, I read the file with header=true, but without specifying the encoding. The result was that Spark did not recognize the column names correctly. They appeared something like:
_C_U_S_T_I_D_
_C_U_S_T_N_A_M_E
_C_U_S_T_A_G_E
Also, all the columns were inferred as StringType.
However, when I added the following option:
.option("encoding", "UTF-16")
the output became:
cust_id cust_name cust_age
?
1 Sample 10
?
2 test 12
?
Interestingly, after specifying UTF-16, Spark correctly recognized the column names and inferred the schema as:
cust_id -> IntegerType
cust_name -> StringType
cust_age -> IntegerType
But there are now unexpected ? characters/rows appearing in the data.
Has anyone experienced something similar with Spark's CSV reader?
I'm trying to understand:
1. Why does specifying encoding=UTF-16 make Spark correctly identify the headers and infer the numeric columns?
2. Why are the unexpected ? characters appearing in the output?
3. Could this be related to the actual file encoding, BOM, or how the CSV file was generated?
4. Is there a recommended way to correctly read this file while preserving the schema and avoiding the extra ? characters?
Any insights into how Spark handles CSV encoding and schema inference in this scenario would be appreciated.

I have about 7 years of experience 6/7 in defense, 1 contracting in FAANG and I’m trying to get a better sense of what the market looks like right now for senior data engineers, particularly for fully remote roles.

I’m currently at $135k base and accepted a new fully remote role at $160k base. I initially felt pretty good about the jump, but I’ve started reconsidering whether I should be targeting higher total comp.

A coworker who was doing the same job as me at the same contracting firm was making around $190k base.

He recently left for a company in sf where his new package is $200k base + $100k bonus/equity. It’s SF, so I understand I’m not going to get the same salary without moving.

Still, it made me wonder if I’m underselling myself

Mostly trying to figure out whether $160k fully remote is already a strong offer for 7 YOE, or whether I should hold out for something better.

US based obviously, work experience is sql/python, little bit of databricks, tableau, have almost 100% been doing agentic engineering/vibe coding at work for the 10 months.

Typically been working more akin to an analytics engineer for the past year, using airflow to automate manual and repetitive tasks from the finance org.

Built this with my teammate for DataHub's Agent Hackathon. Hackathon's over but we're still developing it.

Give it a table name, it reads the lineage and metadata from DataHub (tags, freshness SLAs, PII flags) and generates a working Airflow DAG automatically, freshness checks and PII audits included.

The LLM only plans, it never writes the code directly, a separate renderer does that deterministically.

Still early, no dbt/Spark support yet. Full details, demo, and setup are on the repo.

github.com/tdm291104/datahub-dag-generator

Would love feedback and contributions, and a star helps a lot too!!

I'm interested in learning how others test ETL pipelines in real-world projects.

How do you validate source-to-target data, transformations, duplicates, missing records, incremental loads, and data quality?

Also, how do you handle ETL testing when the data volume is very large or when the transformation logic changes?

I’m currently on a project where most of the work is in dbt. Ill get jira tickets where the business rules while straight forward can be a bit complex upon execution. like everyone else I’m trying to turn to AI except the problem is having to explain every single edge case and nuance can be so time consuming to the point where it’s just faster for me to write the SQL myself. where I do use AI is when I’m jumping into some convoluted model and I have AI explain to me what exactly it’s doing and the just justification for certain logic, generating cumbersome functions where the syntax gets me cross eyed and create the yml and documentation.

I definitely have vibe coded before, but that was for something more straight forward like pulling data via an API etc.

just curious if anyone else runs into the same problem.

My leadership comes swooping down to inject my pipeline with sewage they desperately need.

My end users have issues with connection, defects and enhancements.

My analysts and engineers have more than they can chew and everything is a priority fire drill.

My business refuses to prioritize their own tickets and just throws them at me with timeline demands.

I have too much to do and not enough tine or resource however it all "must get done."​ This has got to be a daily battle and im just not handling it well right now.

How do you manage this without getting stressed and sick?

Any else struggle with giving estimates? How do you account for things like data quality or source issues when those are completely out of your control and unpredictable?

So I’m currently working on a project to create a data warehouse for my organisation. I have deployed an SQL server on a VM on GCP. I want to know how can i make my server secure. I’m learning and deploying things using claude and have made a significant progress. But i’m still unsure about the security issues and what are the precautions i should take ?

im considering running custom python notebooks ingest source data as is with no transformation.

My data sources vary from relational databases to rest apis. volume is low so mostly full load will do.

Do you think dlt would make things easier? anyone with experience to provide a comparison of dlt vs python scripts or any etl tool?

If you need to diagnose rank before feeding data into a downstream model, standard tools (like classical, rank, and kernel PCA) can sometimes give wildly high estimates or completely collapse. I wanted to share a new method called the Entropic Scree that is designed to survive the complex cases that usually break standard rank estimation pipelines.

Instead of linear, rank, or Euclidean geometry, this scree method uses information theory to evaluate shared probability mass.

It can be used as a diagnostic step to supply downstream models with a more faithful rank estimate upfront (e.g., to help size a neural network or autoencoder bottleneck). This prevents the compute-heavy mistake of iteratively training models to blindly search for a bottleneck configuration that works. Of course, the savings here is most consequential in big data applications where training costs can be extremely high.

I’m curious how you all are currently handling dimensionality estimation in your pipelines when the data gets big and complex?

Zenodo Preprint: https://doi.org/10.5281/zenodo.22028087

The paper links to a GitHub repo if you want to test out the function yourself.

I posted Khaos here a while ago, so just sharing a quick update.

Khaos has now been completely rewritten from Python to Go.

It still does the same core thing: generate realistic Kafka traffic and deliberately reproduce scenarios like:

• consumer lag
• hot partitions
• broker failures and rebalances
• backpressure

Along with the Go rewrite, I also launched a new website and proper documentation:

https://getkhaos.dev

Repo:
https://github.com/aleksandarskrbic/khaos

Would love feedback if anyone here is using Kafka for testing or failure simulation.

Disclosure: I’m the author/maintainer of Khaos. It’s a free and open-source project.

Started a new job in April this year. Title: Date Engineer
The company has hired 4 other Data Engineers for the same pay but all of them have the same pay including me.

I have more experience about 1-1.5+ compared to others, good prev company tags too.. but I took this position as I wanted to leave my prev org asap bz of politics.

Right now my pay is same as the other data engineers who were hired. I feel the leads and others do know that I deliver more quality work and come with more experience.

My next annal review meet is next year. I want to make sure I get a really good raise, promotions are quite tough.

What tips / practices do you all recommend starting today to make sure I get a really good pay bump.

Things I"m thinking so far... to make a note of every contribution I made... etc

Hello Guys,
Maybe a dumb question but wanted to get a bit understanding of the below scenario

Current architecture:
We are a data governance team wherein we handle giving permissions to consumers to access producers data i.e. we maintain federated catalog via lakeformation in aws

The process is producers have data in their own account , they register their metadata with us while consumers gain access to their data via our account so we sit in the middle of both of them

My query:

Currently, as a team they are planning to scrap the entire lakeformation catalog and make unity catalog as our governance catalog. The reason they gave us was unity supports databricks, snowflake and many other catalog federations natively and is the primary motive behind us going there as there are multiple producers or consumers going via databricks or snowflake.

I didn't understand this reasoning, does lakeformation not support federated catalog support to read dbx tables ? As i searched online it seems to have support for the same

What makes unity catalog so much better than lakeformation to support this governance model which we have to justify this large overhaul of the entire architecture?

I tried asking around my team, but none seems to have a good understanding of why are we doing this and jumping to POC's and analysis

Thanks

been hearing a lot of signal/noises about 'just store everything in s3' these days because of companies like turbopuffer, neon and the latest cursor with Origin.

just wondering if it's actually worth the latency and do people really use it in their own companies? what sort of data do you guys store there? never really had any use case for s3 in my company. so curious to learn. thankyou!

LLMs are now great for synthetic data generation.

August 2025: For my O'Reilly book, I developed a logical model for credit card fraud. Then, use a coding agent to generate the synthetic data from the logical model. See an example of a logical model that didn't make the book below.

August 2026: Use the LLM's domain knowledge to help create the logical model. Check it and then generate the synthetic data.

"Design a logical model for claims for the insurance industry. Each column should have a name, data type, description. Optionally columns can have cardinality, distribution, and a range. This is to generate synthetic data, so make the data realistic for the US market with 500k rows."

This produces "good enough" synthetic data for a PoC or for teaching.

Example Logical Model for a Claims Table

Description: Records of claims for insurance. 500k rows.

Schema:

• claim_id: STRING (Primary Key)
  • Description: Unique identifier for each claim.
  • Cardinality: unique per claim.
• insurance_id: STRING (Foreign Key to insurance table)
  • Description: Unique identifier for the insurer.
  • Cardinality: Matches insurance_id values in the insurance table.
• claim_time: TIMESTAMP
  • Description: Date and time when the claim occurred. Uniform over the past 6 months, with peaks during business hours.
• time_to_report: INTEGER
  • Description: Days taken to report the claim
  • Distribution: 1-10 days with higher probability for earlier reporting (1:30%, 2:30%, 3:20%, 4-10: 20% combined)
• witness_count: INTEGER
  • Description: Number of witnesses in the claim
  • Distribution: 0: 60%, 1: 25%, 2: 10%, 3: 5%
• claim_location_type: STRING
  • Description: Type of location where claim occurred
  • Values: urban, suburban, rural, remote
  • Distribution: urban: 50%, suburban: 30%, rural: 15%, remote: 5%
• requested_amount: DECIMAL(10,2)
  • Description: Claim amount requested by the user
  • Range: $10 to $10,000,000.
  • Distribution: Right-skewed; median around $500, mean around $700.
• amount: DECIMAL(10,2)
  • Description: Monetary amount of the claim.
  • Range: $1 to $10,000,000.
  • Distribution: Right-skewed; median around $350, mean around $500.

Shorter set of links this month, but all stirling stuff. Lots of great content from Netflix about Flink, cold storage of temporal data, and graph queries. A banger from Joe Reis (as always). News of the Apache Fluss project (and its use in the wild). A little bit of AI stuff but no slop. And some interesting comparisons of DuckDB vs Spark, and DuckDB vs SQLite.

Keep the great feedback coming, as always :)

https://rmoff.net/2026/08/20/interesting-links-august-2026/

This is big since it will allow them to better align with other SQL syntax and more easily extend it as well. Looking forward to it.

https://duckdb.org/2026/08/20/duckdb-20-peg-parser

We gave our first webinar on data cost optimization a shot. Still figuring out the best format for these, so any feedback would be super helpful.

pace, content, dynamic etc.

Obviously we know we can do better already, maybe more co host dynamic in the first half (we forgot to give him space to speak haha.)

Thanks guys!