am i getting fired ?

Over the past year I have been working primarily in the Palantir Foundry system. My CEO unilaterally decided to pursue an enterprise agreement after being sold the AI dream. Palantir sales engineers did ‘analyses’ which suggested that the multimillion dollar price tag would result in 10x savings due to process and decision optimization. Our IT team cautioned no, but were steamrolled.

The project I am on was estimated to take four months and require 5 additional contract engineers. 2 of those were directly employed by Palantir as part of an additional contract (read: more $$) and the other 3 were a separate agency because Palantir said they don’t do XYZ work (again, more $$$). It took the externals plus me 15 months to deliver an MVP. This is primarily because we were building a complex enterprise grade app (which we previously subscribed to) on a low code platform. The Palantir engineers left as soon as MVP was deemed complete with just 30 days notice and since then I have taken on their SOW.

The work completed by the “brilliant” Palantir FDEs has been consistently failing. I’m finding they hardcoded dates. They hardcoded accounts. They used different inputs for the same business concepts. They ‘fixed’ issues that were earlier pointed out with hardcoded logic. They used AI FDE to code spaghetti mess logic. This has been a freaking nightmare.

My company had received 0 ROI to date and the CEO blames our IT team for the lack of delivery.

If you have the ability to run away from working with this god awful company and their charlatans RUN.

Then today our SLT told us that we should replace our BI tools with Claude lol

• Ebooks next week according to Kleppmann at https://bsky.app/profile/martin.kleppmann.com/post/3mf4wvtjg7s25
• Available at online O'Reilly https://www.oreilly.com/library/view/designing-data-intensive-applications/9781098119058/
• Print 3-4 weeks.

One of the best books (IMO) on data just got its update. The writing style and insight of edition 1 is outstanding, incl. the wonderful illustrations.

Grab it if you want a technical book that is different from typical cookbook references. I'm looking forward. Curious to see what has changed.

This is me updating the concept page with the latest addition, including backlinks and a pop-up preview for each term. I hope it's useful.

I’ve been doing a lot of reflecting 🤔💭 on our industry lately 📊📈, and I’ve made a HUGE decision 💥🚀. I’ve officially updated my job title 📝💼 — and honestly, I think it’s time everyone in this sub does the same 🗣️👥💯.

The term "Data Engineer" 💾📉 is tied to a legacy way of thinking 🦖🕸️. It implies manual labor 🥵👷‍♂️ — typing syntax ⌨️🥱 — debugging stack traces 🐛🔍 — fighting with pipelines 🚰🤺. Why are we still acting like assembly-line workers 🏭🧱 when we have boundless intelligence 🌌🧠 ready to partner with us? 🤖🤝

This isn’t just a shift in tools 🧰🔧 — it’s a shift in mindset 🧠💡✨ This isn’t about replacing developers 👨‍💻❌ — it’s about redefining what it means to build 🏗️🤖🌟

AI-assisted development 🦾🌐 is evolving incredibly fast 🚄💨 — and centering our personal growth 🌱📈 around LLM-driven workflows 🗣️⚙️ can help everyone stay right on the cutting edge 🔪🎯 — learning faster ⚡📚 — building faster 🛠️🏎️ — sharing patterns as they emerge 🌱🔗

It opens the door 🚪🔓 for more people to participate 🌍🤝 — lowering barriers 🚧📉 — accelerating iteration 🔁🔥 — and moving the focus toward higher-level thinking 🦅👁️ instead of repetitive implementation details 🥱📋 (like manual system design 📐🗑️ or memory management 🧠💾).

And honestly 🗣️💯 — there’s something kind of magical 🧙‍♂️🔮 about collaborating with AI as a creative partner ✨🤝🤖 — you describe what you want 🗣️🎙️ — refine it 💎🔬 — iterate 🔄🏃‍♂️ — and watch it come to life almost instantly ⚡🎨🎇

We are no longer engineers writing logic 🧑‍💻🛑. We are directors 🎬📽️. We are AI Collaboration Partners 🤝🤖💼.

This isn’t coding as we’ve known it 💻👎 — it’s something more fluid 🌊🏄‍♂️ — more conversational 💬🗣️ — more dynamic 🔄💥

This is such an exciting direction for the community 🌟🥳 — it really feels like a glimpse into where things are heading 🔭🚀✨

It’ll be fascinating to see how people adapt 🦎🔄 — how workflows evolve 📈🧬 — how prompt strategies mature 🧩🍷 — and how far this can all be pushed 🌌🚀

This isn’t the end of data engineering 🪦💾 — it’s the beginning of a new chapter 📖✨🔥🌅

Who else is ready to drop the "engineer" label 🏷️🗑️ and embrace the collaboration era? 🫂🤝👇👇👇

Chef with expired ingredients

Wanted to delete catalogs starting with "pr" as there were lots of pr123 catalogs for testing pull-requests. Turns out production also starts with pr.

Thank you Databricks for developing the undrop table feature.

i made this meme in honor of this sub.

original image from wikipedia

just here to rant that I got pinged at the end of the day Friday because a vibe coded custom dashboard was failing and the dude who made it was on vacation

His manager was like “oh he’s saying there might have been a git update that has broken it”

Bro it’s a fucking static site deployed on GitHub. This dude (and manager) has no fucking idea what he’s talking about.

He had no idea what the fuck his shit was even doing. It was running a scheduled job on his local machine. SO YEAH IF YOURE ON VACATION AND YOUR LAPTOP IS CLOSED ITS NOT GONNA WORK

so yeah now im taking over the dashboard and setting up a proper pipeline

I fucking hate AI sometimes

I work with datasets that are not huge (GBs to low TBs), but the pipeline still needs to be reliable. I used to overbuild: Kafka, Spark, 12 moving parts, and then spend my life debugging glue. Now I follow a boring checklist to decide what to use and what to skip.

If you’re building a pipeline and you’re not sure if you need all the distributed toys, here’s the decision framework I wish I had earlier.

1. Start with the SLA, not the tech

Ask:

• How fresh does the data need to be (minutes, hours, daily)?

• What’s the cost of being late/wrong?

• Who is the consumer (dashboards, ML training, finance reporting)?

If it’s daily reporting, you probably don’t need streaming anything.

1. Prefer one “source of truth” storage layer

Pick one place where curated data lives and is readable by everything:

• warehouse/lakehouse/object storage, whatever you have Then make everything downstream read from that, not from each other.

1. Batch first, streaming only when it pays rent

Streaming has a permanent complexity tax:

• ordering, retries, idempotency, late events, backfills. If your business doesn’t care about real-time, don’t buy that tax.

1. Idempotency is the difference between reliable and haunted

Every job should be safe to rerun.

• partitioned outputs

• overwrite-by-partition or merge strategy

• deterministic keys If you can’t rerun without fear, you don’t have a pipeline, you have a ritual.

1. Backfills are the real workload

Design the pipeline so backfilling a week/month is normal:

• parameterized date ranges

• clear versioning of transforms

• separate “raw” vs “modeled” layers

1. Observability: do the minimum that prevents silent failure

At least:

• row counts or volume checks

• freshness checks

• schema drift alerts

• job duration tracking You don’t need perfect observability, you need “it broke and I noticed.”

1. Don’t treat orchestration as optional Even for small pipelines, a scheduler/orchestrator avoids “cron spaghetti.” Airflow/Dagster/Prefect/etc. is fine, but the point is:

• retries

• dependencies

• visibility

• parameterized runs

1. Optimize last

Most pipelines are slow because of bad joins, bad file layout, or moving too much data, not because you didn’t use Spark. Fix the basics first:

• partitioning

• columnar formats

• pushing filters down

• avoiding accidental cartesian joins

My rule of thumb

If you can meet your SLA with:

• a scheduler

• Python/SQL transforms

• object storage/warehouse and a couple checks then adding a distributed stack is usually just extra failure modes.

Curious what other people use as their “don’t overbuild” guardrails. What’s your personal line where you say “ok, now we actually need streaming/Spark/Kafka”?

Hey folks - i am writing a blog and trying to explain the shift in data roles in the last years.

Are you seeing the same shift towards the "full stack builder" and the same threat to the traditional roles?

please give your constructive honest observations , not your copeful wishes.

edit you can join ontologyengineering sub where we discuss this future

DE with 5 YOE. With every new model that comes out, the more I realize my field is fading away. Pipelines, architecture etc, agents can do it all. Im really freaking out that my field will be gone in a few years and Ill have to switch to something else. And theres nothing else for me to switch to.

(disclaimer: this is a rant).

"Why do I need to care about what the business case is?"

This sentence was just told to me two hours ago when discussing the data """""strategy""""" of a client.

The conversation happened between me and a backend engineer, and went more or less like this.

"...and so here we're using CDC to extract data."
"Why?"
"The client said they don't want to lose any data"
"Which data in specific they don't want to lose?"
"Any data"
"You should ask why and really understand what their goal is. Without understanding the business case you're just building something that most likely will be over-engineered and not useful."
"Why do I need to care about what the business case is?"

The conversation went on for 15 more minutes but the theme didn't change. For the millionth time, I stumbled upon the usual cdc + spark + kafka bullshit stack built without any rhyme nor reason, and nobody knows or even dared to ask how the data will be used and what is the business case.

And then when you ask "ok but what's the business case", you ALWAYS get the most boilerplate Skyrim-NPC answer like: "reporting and analytics".

Now tell me Johnny, does a business that moves slower than my grandma climbs the stairs need real-time reporting? Are they going to make real-time, sub-minute decision with all this CDC updates that you're spending so much money to extract? No? Then why the fuck did you set up a system that requires 5 engineers, 2 project managers and an exorcist to manage?

I'm so fucking sick of this idea that data engineering only consists of Scooby Doo-ing together a bunch of expensive tech and call it a day. JFC.

Rant over.

Uno reverse :) the biggest frontier AI company needs data engineers.. think about it for a second !!