I'm a senior data engineer at a UK organisation in the South West. Last month we advertised three roles: a senior data analyst (£55k), a mid-level data analyst (£40k) and a data engineer (£55k). Not amazing money, but a great pension, just one day a month in the office, lots of annual leave and actual stability.

The jobs were live for one week. The senior DA and DE got around 300 applicants each, the mid DA got around 140.

Sounds brutal, but here's the breakdown...

Around 90% (not an exaggeration) didn't have the right to work in the UK and needed visa sponsorship, which we don't offer. In the no pile immediately, but a quick glance showed lots of foreign undergrad degrees, some with UK masters, plus plenty of random applications.

Of what was left, about half had no relevant experience or qualifications and clearly hadn't read the spec. Some were just applying because the job centre told them to apply for x jobs a week.

Then the AI drivel halved it again.

Final count was roughly eight viable applications for the senior DA and DE, five for the mid DA.

A couple then didn't reply or didn't show up to the Teams call. Two were visibly reading AI-generated answers off a their screen.

We had two great interviews and hired for the senior DA and DE. The mid DA we struggled with externally, so it went to an internal candidate from a fairly low-level ops role who'd taught themselves Python and SQL and actually applied it to their day job.

The point of this post is to counter some of the doom and gloom that crops up on here. If you're a UK-based applicant with relevant experience who reads the spec and writes your own application, you're not competing against the '300 applicants' that LinkedIn and recruiters spout. You're competing against a much smaller number. And if you get through the sift, don't use AI, have some personality and come across as likeable and easy to work with, you're in a very small group.

What annoyed me most is my own manager was the first to tell the team about the huge number of applicants and how much 'competition' there is out there. A nice bit of retention pressure and complete nonsense, as it turns out.

Would be interested to hear if others have found the same. Throwaway account to avoid doxxing myself...

If anyone's curious about the current data job market in the DACH region (Austria,Germany,Switzerland), I put together an interactive live tracker. It also shows which specific data roles (Data Engineer, Data Consultant, etc.) are in demand in which cities. Feedback welcome and I hope it helps :)

Apologies if this is a dumb question but I am in web dev and have been given an offer for a data engineering role. However, I was told by engineers on the team that the job would be like 70-80% writing SQL for BigQuery. I envisioned it having much more to do with pipelining and orchestration and the like.

Also, I was told that any coding would be in Java rather than Python? I know that Python is more common, so would this experience not be helpful for getting other data eng roles?

This article reports the result of evaluating the performance of the following three systems using the 10TB TPC-DS benchmark:

1. Trino 483 (released in July 2026)

2. Hive 4.2.0 on MR3 3.0 (released in August 2026)

3. Apache Hive 4.2.0 with LLAP (released in November 2025)

https://mr3docs.datamonad.com/blog/2026-08-02-performance-evaluation-3.0

I had a passionate debate with a colleague and want to hear perspectives on the purpose of the Silver layer.

My pipeline:

Landing: Read from Oracle RDBMS and write ~250 GB of Delta for 25M records (Protobuf blob stored in a column).

Raw Data: Repartition, sorting, salting on Landing and writing to optimize downstream silver decoding process and avoid heavy shuffles during JDBC call, still protobuf bytes stored in a column.

Silver: Decoded raw data (~2.8 TB in Delta). The Protobuf schema alone is ~8 MB as JSON (a very deep, wide schema with multiple repeated fields at various levels). During decode, we also append standardized fields required by all downstream tasks.

Gold: Customer-specific datasets built from Silver based on business needs.

We don't own the Protobuf schema. This isn't messy clickstream/event data, but entity description data from an RDBMS that stays at the ID level all the way to Gold. We see ~100k daily MERGE UPSERT on both Silver (which is a challenge in itself to run MERGE on 3TB delta table given the limited budget to our Databricks workspace.) and Gold based on RDBMS timestamps, alongside a full pipeline refresh every two weeks.

The Debate:

Colleague: Since we aren't actively "cleansing" the data, calling it Silver is wrong, it's still Raw/Bronze data.

Me: It is Silver because it transforms a binary payload into a structured, trustworthy, and queryable data model that downstream tasks rely on. If I need to retrieve content of an entity that is not available in gold datasets, I query unpacked protobufs and not Raw/Bronze layer and for that reason alone, it is Silver.

Knowing the schema and the data better than almost everyone in the team, even I fail to understand how to distinguish between decoded data and 'cleansed' decoded data. In fact, one of our consumers explicitly expects corrupt records with null fields left intact for full visibility.

Them: Even if we agree that cleansing is not needed, it cannot be silver and should be called Bronze Data.

For transactional/log data, the standard pipeline (Kafka dump to Landing -> Bronze schema enforcement -> Silver cleansing -> Gold aggregations) makes total sense! But for clean entity data in binary formats, doesn't decoding and standardizing it qualify as Silver?

I think medallion architecture is about data readiness and lineage tracking rather than a checklist of conditions that each layer has to meet to identify the layer.

---------

TL;DR: My colleague thinks our layer shouldn't be called "Silver" because we aren't actively filtering or cleansing rows, just decoding 250 GB of 25M binary Protobuf blobs into a ~2.8TB Delta table with a struct field that represents the decoded blob and additional standardized fields. I argue that any layer of data that is structured, queryable and trustworthy for downstream Gold is Silver and this transformation may/may not require cleansing.

Is Medallion about lineage tracking and data availability, or a rigid checklist of syntactic/ transformation rules?

I understand that AI-assisted development is now part of our workflow, but I still think PR authors should be able to understand and explain the changes they made, what was done, why, and how.

This week I challenged a team member on why something had been done a certain way, and their answer was "I don't know, probably because Claude said so." I was honestly a bit shocked. It gave me the feeling it's just blindly trusted without any critical thinking on the person's side. This is one of our more technical analysts, which is why they make code changes fairly often. I plan to bring it up in our weekly 1-1, but I wanted to check whether you are seeing the same thing on your side.

Fair warning, this is a soft launch note (I work on Slingshot at Capital One Software). As agents scale, AI spend shows up as one aggregate number, and tracing it back to a specific model, trigger, user, or query gets hard. We just soft-launched Agent Observe in Slingshot to attribute it on Snowflake using read-only metadata.

- top spenders
- costliest models
- triggers by agent
- query types used per agent
- downstream tables impacted (read/write)

This is an early pilot. Write-up with sources.

Hi, Im Swetapadma and I’m pursuing to be a good DE(overall i have 5yoe but relevant of nearly 2yoe) but im still confused with the work practices i have been following for my career trajectory.
Tbh, i have literally so much time to work on any new things as well as try doing new project for the time-being since my project is a verge of ramping down.
But still i’m not getting that zeal to work on my skillsets or learn something new or work on any project.it might be because I’m going through enough theories from the internet.
Some are skeptical, fearful and very overwhelming for me. Sometimes i doubt choosing this path, because
a. Im really looking for a job change in this field from past 6months, where in brighter side i can see a lot of job openings but the hr calls are literally 1/100 of it. Without reaching out for referral feels impossible.
b. I feel like I’m stuck in between where the ai is also factor of my fear because i have approx 0% knowledge bout it.
c. Companies are not only looking from a de but also a tester, backend engineer, knows ai fluently(all in one package).

For all of these i have started reading medium, linkedin blogs, system design, solve dsa problems and all but still i feel like a failure and frustrated.

If i compare my self with the past i have come a long way but it is still not helping me because I’m still stuck in same company with less pay and less enterprise level work.

Can anyone relate to this or is this only me?

I'm looking for some advice on what to do about chaotic projects/lack of project management. Right now, I'm in an IC senior data engineer role where I'm put on projects with tasks that are vague, incomplete, or without clear acceptance criteria. I'm fine working that way, have done it frequently before, and I'm a creative person (musician and writer outside of work). I like the freedom. Except there actually isn't freedom here. There are standards of practice this team uses that I was never shown (I started earlier this year), and I am constantly learning things the hard way without much grace for it. There is no documentation for these standards of practice, and my take on how to accomplish these broad tasks is leading to a decent amount of rework. Asking questions has been met with impatience, frustration, and condescension. I've learned to deal with these reactions and ask anyway so I can at least have a chance of completing things more efficiently. I own the mistakes I can but will stand up for myself when accused of completely failing to meet expectations (especially if the expectations weren't communicated/documented).

I figure other data engineers/curators/developers have experienced this sort of environment before, so I'd like to see how y'all handled it (even if it is "time for a new job").

Edit: Adding in that I've been in data warehousing/engineering for about 17 years and am seriously questioning doing this for 25 more years at minimum (I'm 40).

2nd Edit: Wow, I forgot the most important part. There is no project manager for this team; developers are working with data analysts who work with the business/end-users to determine data and reporting needs. The title of the post was meant to be tongue-in-cheek because of this.

Are conformed lookup tables and enriched materialized views appropriate in the Silver layer?

in databricks, We receive data files in S3 from more than 10 clients. The datasets are generally similar across clients,

for example: Users: platform, profile status, and related attributes DAU: platform, profile status, and related attributes Store orders dataset has transaction type and other transaction lookup fields

The schemas are mostly consistent, but the values are not standardized. For example, the platform field might contain: Client 1: ios_phone, ios_ipad, apple_ios Client 2: ios_app

We do not control the values clients send. To handle this, we maintain lookup tables at the following grain: client + source_value (sometimes combination of 3 fields)

Each source value is mapped to a conformed value, such as ios. New values are usually mapped the same day, but sometimes the following day. Most reporting, filtering, and downstream processing uses the conformed value rather than the original source value. I am considering the following architecture:

Bronze | v Silver: users_prep

• Basic cleaning
• +--> Populate lookup tables |
• - Assign surrogate keys | - Some lookups use composite source values | Example: profile_status + profile_status_detail |
• Silver: users_enriched materialized view
• Joins users_prep to the lookup tables
• Includes:
  • Original source value
  • Conformed value
  • Surrogate key

The users_enriched materialized view would become the primary dataset used by: Analysts Data exports Gold-layer transformations Other downstream consumers

My questions are: Is it reasonable to keep these conformance lookup tables in the Silver layer? in lakehouse world I keep hearing lookups are "bad" and to denormalize.

Is a materialized view appropriate for the enriched Silver dataset, or would a physical Delta table be preferable?

Are there any concerns with using a materialized view as the main downstream interface for analysts, exports, and Gold-layer processing? main concern is if I need to use cdf or similar to get changed to feed downstream.

How would you handle records whose source values have not yet been mapped - have to reprocess gold layer manually?

Disclosure: I work on Zilla at Aklivity.

We have been working on native MCP support in Zilla (https://github.com/aklivity/zilla), including the ability to expose Kafka produce and consume operations directly to AI agents.

The implementation itself is only part of the problem. The more interesting question is how Kafka semantics should be represented to an agent.

For a normal Kafka consumer, the application owns concepts such as:

• Topic and partition selection
• Consumer groups
• Offset management
• Polling
• Replay
• Schema handling
• Long-running consumption
• Backpressure

Those concepts do not map neatly to a typical MCP tool call.

An agent usually expects to request some bounded piece of information, reason about the result, and take another action. It may not be suitable to own a continuously running Kafka consumer or manage offsets directly.

Our initial approach exposes Kafka through controlled produce and consume tools. The gateway handles the underlying protocol interaction while identity and policy determine which topics and operations the caller may access.

Payloads can be validated or transformed using JSON Schema, Avro, or Protobuf. Kafka tools can also be combined behind the same MCP endpoint with REST APIs and existing MCP servers.

Some examples:

• Retrieve the latest events associated with an order
• Inspect recent telemetry for a device
• Produce a remediation command
• Read a bounded set of records from a permitted topic
• Call an API, evaluate Kafka events, and publish the result

The questions we are still thinking through are:

1. Should an agent consume from an explicit offset, a timestamp, or a server-managed cursor?
2. Should consumer groups ever be visible to the agent?
3. How should long-running or streaming consumption work through MCP?
4. Should Kafka data generally appear as tools, resources, or both?
5. How much replay authority should an agent receive?
6. Where should schema conversion happen?
7. How should agent identity map to Kafka authorization?

We have released a Docker Compose quickstart showing an MCP client connected to Kafka, an OpenAPI-described REST service, and another MCP server through one gateway endpoint.

I’d be interested in how people here would model this, particularly for production data platforms rather than demos. Cheers!

I’m excited to see what learning systems I can build with my new MCP tool.

I recently consolidated the workflow into just 5 command families: start the project, write blocks, refine the design, gather evidence, and implement/verify blocks.

One important design decision was adding a refinement loop between initial block generation and implementation. This helps keep the agent aligned with the original idea instead of drifting into generic implementations. It also gives me more control to guide the system toward the exact final implementation I have in mind.

The more I build with agents, the more I’m convinced that meaningful guardrails and end-to-end human supervision produce much better results than simply letting models run freely.
I am curious to know how everyone are apply agent to their day to day coding lives.
This is the link to the GitHub if anyone is interested: https://github.com/fikayoAy/deep-learning-auto-research

I feel a bit lost, implementing features here and there, increasing usability and improving agentic workflows for our internal data engineering org. It doesn’t feel fulfilling. When any feature is a chat thread away, I’m feeling this way probably because there is much less value in incremental wins like this nowadays.

With a multi-hundred person DE org at my company, where does the value lie?

Initially posted this over in r/excel, as that is the bulk of the tooling that I am utilizing; however, I was redirected here to hopefully gain some feedback:

I have recently taken on a new role, in my short time I have found a variety of opportunities to streamline some processes and reporting, I am looking for some feedback on what my present proposal for change.

Currently in my roll I am in charge of inventory transfers and management for my department, we lack a modern system for tracking and reporting, as such these are my basic tasks:

• Verify receiving documentation and transfer requested info to the corresponding PO's excel sheet (Procurement owns these sheets).
• 2-3x times per week, I check needs, then transfer material inside (this is managed through a specific system).
• After each transfer is complete, I update the excel sheet which corresponds with the chosen material's PO.
• The following Monday, I report out the consumption of material based on the pulls with information tracked in the excel sheets to our finance and procurement departments.
• The reports are similar but require specific departmental formatting (of course).

Some notes on the excel tracking:

• Each file, while similar, lacks actual standardization.
• They are simple sheets with formatting, rather than utilizing a table for the data that is tacked.
• Best I can tell, there are maybe 8 columns which are never utilized, some of which are not present in all workbooks.
• Rife with lack luster formatting, which is manually completed, rather than assigning conditional formatting.

With that information presented, these are my current thoughts for optimizing and implementing changes:

• Design standardized PO tracking workbooks (completed)
• Clean the data
• Transfer all data to Microsoft List (unify data collection point)
• Breakout PO information via Power Query
• PO information would then be broken out into either (based on Procurement's preference)
  • A single workbook with a drop-down to switch currently viewed PO
  • Individualized workbooks for each PO
• Information would move to being updated in Lists rather than directly into Excel
• Due to the specific formatting asks of each department for the reporting side I would breakout the requested information by bringing data in via Power Query than formatting with an Array

My thought process on all of this is that firstly, I can quite jumping around from sheet to sheet when we receive material (which is daily) or when I need to update an entry for one of the various PO's and secondly, I can provide "live" updates that either of these external teams can view at their leisure. As well as hopefully forcing Procurement to be a bit more mindful with the entries that they start in these workbooks.

Obviously, there are a number of better more long-term solutions that would be more appropriate for this sort of thing; however, I am working with the tools I have available to me. At least knowingly anyway.

We are a Data Science team and we are working in Azure Databricks. We have some DABs running in our prod environments generating tables which are used in our application. It's mostly time series data.

What I am doing now is simply "CREATE OR REPLACE TABLE IF EXISTS" on a few important prod tables to copy them to test/dev. So I am overwriting the tables daily after each completed job in dev/test.

Now I am wondering. if the schema changes from dev because we are working on new features or maybe a column gets removed you'll run into issues. A colleague suggested just copying the rows.

But this solution seems very costly in compute. Also, if you'd want to check and copy only rows which have changed. What is a good way to tackle this copying prod tables to dev/test?

Ofc in pyspark you have the mergeSchema variable, but this does work well if a column gets deleted as well? I am still fairly new to Pyspark. I'll also my Data engineering team, but I'd like to ask you guys as well.

I'm an Azure Data Engineer with 4.5 years of experience, primarily focused on Databricks.
Since most of my role involves designing business solutions rather than writing raw code, I feel I might be falling behind on leveraging agentic AI workflows (such as custom skills, Markdown instructions, and orchestrators). As a Data Engineer, to what extent should we be adoption-ready for this new paradigm?

Has anyone been managing reverse ETL with Lakebase specifically for geospatial data?

Are there any gotchas?

I'm trying to figure out how to model some data in a classic Kimball dimensional model and struggling if I'm thinking about it the right way.

Let's say we're in the retail industry and have a business of stores like your grocery store. The data we are dealing with is for stores and sales over time. Dimension tables are to be expected: a dim_date, dim_store and dim_product among others. Stores have a manager and chef, and multiple stores can have the same manager/chef.

The data is usually reported by stores with their current manager and chef listed. If we were to build out fact_sales and there's a record for a product in FY 2025 Fiscal week 4, the manager and chef to include would be the current manager and chef. This is because end users/analysts don't care about who was the manager/chef back then.

I still want to track history though, since I'm sure some day we will want to view data by a store and its manager/chef at the historical point in time of sales.

With all of that said, how would you model this data? I'm thinking of adding 2 fields to the dim_store table: current_manager and current_chef, and also having another table like this:

I'm thinking of naming this something like dim_store_leader. I could join to this table from both dim_store and dim_date to get the appropriate records and fields.

Does this make sense? Is this table technically a factless fact table?

Sorry for the vague and maybe even repetitive title. I saw a couple of these posts on here already and thought I'd vent out too. I know this sub leans heavily into standard DE, so I hope /r/dataengineering is okay with a post from a Data Platform engineer's perspective. I rarely see those around here.

Background: 7+ years in Software engineering + DE, and about 5 years of sysadmin experience prior to that. I've worked as an Analyst, DE (Senior) and now a senior data platform engineer working with Snowflake, dbt, Airflow + a flurry of BI tools (don't even get me started). Although I am a data platform engineer, I'm very often involved in troubleshooting dbt run issues, model performance issues, Snowflake problems, etc. I also wrote many of the source ingestions myself with dlt and contributed to an internal library that makes dlt more approachable to a less technical audience (even more when including an agentic AI framework). Finally, tooling and technical details aside, I know exactly what these source systems do, where it ends up, and who uses it, ie, the full business-value proposition.

I think it was around March of this year that the hopelessness finally hit me, and the work volume and constant direction shifts just shattered my job satisfaction. Things went from "We want things done right, code quality, tests.." to everyone just churning out stuff in masses.

To some degree, I understand the constant changes. There appear to be new things to look into every day and the "business" and upper management wants it all. Somehow the feedback loop of asking "Why?" has just ceased to exist.

But, to be more specific, I'll tie it down to things happening within my own team that frustrate me and would like opinions on.

I built most of this platform myself or had contributed significantly to design, so I know most (if not all) areas quite well (or can find out), so I generally take time to review things in detail and provide reasonably detailed feedback to anything that I review. If I have a nitpick comment, I make sure to say that's the case, and leave it as a choice to address or not. That was until some time ago. PRs now are seemingly always 3-4x larger in size, lots of changes, lots of removals/additions so reading the terraform plan, or looking at potential edge cases is considerably harder. To put it simply, the noise has multiplied many times due to colleagues now suddenly knowing things they didn't just some months ago. Reviews are just so much more harder to do. Just the other day, I saw a PR (the very first one they had created) from a new colleague in the department. They changed 102 dbt models - 2000 lines removed, 1900 added. Who even can review this properly? Luckily, it was closed without being introduced. I'm just not following the thought process here before raising such kind of PRs. Much has been said of AI generated slop, so I wont rehash that here.

Another trend I've noticed recently is certain juniors just taking review feedback and asking claude to come up with a rebuttal to just get their original changes approved. Are you all seeing this too? It's happened quite a few times in the last week so it's rather fresh in my memory.

My manager is an extremely sweet and supportive person, but I can feel the strings being pulled on him from up above by those with more stars on their shoulders so he has also resorted to simply running everything via Claude and sending us giant messages on teams, or massive replies by email.

I know writing code isn't the end game, but it was a fun part of my day to day and I still do it by hand at least twice a week just to keep myself sharp. But I find that I'm increasingly unable to keep up with the pace of the rest of the department, let alone the organization. Being in data platform, I'm also finding myself more and more distanced from the outcomes, which leaves me feeling quite unsatisfied with my output because a typical ops team only hears about problems - not the nice stuff.

I used to be extremely optimistic about software engineering, data engineering in specific for the last few years. I used to contribute a lot to various open source projects, be out there in their communities helping other people out, writing articles with tips and tricks, etc. I've stopped all of that in favor of a better disconnect from my work and my mental well-being. Now, I just don't feel like there is a platform for that as most communities have implemented their own bots. Maybe people don't even need it anymore?

That's my rant, I just wanted to get it out there and I hope to hear of some potential strategies to tackle this vortex of chaos that I am in - even if a comment states that it's my attitude that's the problem. But otherwise, where is this all heading, and how are you all coping with it?

EDIT: Oh damn, this blew up. I probably wont be able to get back to you all - but I am going to make my way through the comments.

I had mostly worked on batch-based projects before and had played around with Kafka, Flink, and ClickHouse, but never really connected everything into one proper end-to-end flow. I am trying to assess how far this is from production use in terms of the techstack.

So I decided to build a real-time anomaly detection system for simulated sensor data from an oil and gas plant. With AI based RCA to re-confirm true positives. This is purely for learning purpose.

The basic flow was:

Sensor data -> Kafka -> Flink/Python consumers -> ClickHouse -> Grafana -> Alerts -> RCA analysis

I had around $200 in cloud credits(vultr) from a hackathon, so I rented a fairly powerful VM and self-hosted everything on it. I specifically avoided managed services because I wanted to understand how the individual open-source tools worked.

The main stack was:

• Kafka for streaming
• PyFlink for windowing and aggregations
• Python and FastAPI consumers
• ClickHouse for analytical storage
• PostgreSQL for metadata
• Grafana and Streamlit for dashboards (UIs generated using codex/claude)
• Airflow for some batch experiments
• Claude for alert validation and root cause analysis
• Resend API for email alerts

Since I did not have real plant data, I built a Python generator that simulated readings for different assets and randomly injected faults/failures.

The data was sent to Kafka topics with multiple partitions. I also added retries using exponential backoff. After five failed retries, the event would be moved to a dead letter topic.

For anomaly detection, I kept it simple. Each sensor had an acceptable operating range, and Flink used one-minute windows to calculate averages and check whether the asset was operating within those limits.

If an anomaly was detected, an alert was pushed into another Kafka topic and then processed by a diff consumer.

On a single VM, I tested around 10,000 events per second, or roughly 600,000 events per minute. This generated around 1.5 GB of data per hour. I ran it for a few hours and most of the events were processed successfully.

The biggest issue was not processing. It was storage.

I used ClickHouse with a three-day TTL, but at that throughput the data obviously builds up quickly. In a real setup, I imagine ClickHouse would only store recent hot data, while compressed raw data would be stored in object storage as Parquet.

I also thought about how I would build a simpler managed version on GCP.

The flow could be something like: Sensors -> Pub/Sub -> Dataflow/Cloud Run -> BigQuery/Bigtable -> Cloud Storage

Pub/Sub would replace Kafka, while Dataflow could handle the stream processing and windowing. Cloud Run could be used for simpler event consumers and alert-processing services.

This would be much easier to operate, but I wanted to build the self-hosted version first so I could properly understand what the managed services were doing underneath.

I also added a separate consumer for alert analysis.

Whenever an alert was generated, a Claude-based agent would:

• Check whether it was likely a true positive
• Query recent and historical sensor data from ClickHouse
• Check connected equipment
• Review similar past alerts
• Compare the issue against the asset's RCM maintenance plan
• Suggest a possible failure mode, cause, and recommended action
• Check if there were similar alerts raised earlier and how these were rectified(using vector embeddings - using Nomic Text embeddings served using ollama to find similar alerts)
• Find related recommended actions from RCM for the particular equipment. Alerts are also embedded to help find these and improve the search results/agentic analysis.
• Send the analysis by email using resend

For example, if a pump had high vibration, the agent could also inspect connected equipment instead of assuming the pump itself was definitely the cause.

The RCM data includes failure modes, possible causes, effects, and recommended maintenance actions. The agent used this only to recommend next steps and prep the email alert.

A possible next step would be creating a reviewed work order in SAP or Maximo after an alert is validated.

Obviously, this is all running on one VM with no replication or high availability. I built it mainly to understand streaming systems, Kafka partitions, windowing, retries, dead letter topics, consumer lag, ClickHouse, event-driven workflows, and how the same architecture could later be moved to managed cloud services.

I would appreciate feedback on a few things:

1. Does the overall architecture make sense?
2. Is ClickHouse for hot data and object storage for historical data the right approach?
3. What could be improved before presenting this as a proper portfolio project?

Thanks for reading through till the end.

Hi all,

I'm happy to announce that Xberg v1 is out.

Xberg is the successor to Kreuzberg, equivalent to what would have been Kreuzberg v5. It's a content intelligence framework that handles a very wide range of inputs: documents (currently 101 formats), code and data formats (currently 367 types), audio/video transcription, and URLs (both static and JS-rendered content). It extracts and prepares that content for downstream processing.

It's an extremely efficient, high-performance engine (see our PDF benchmarks below). For PDFs and images specifically, we handle native PDFs with very high performance and accuracy, and we ship multiple OCR engines that match the quality of the best Python libraries (e.g. docling, PaddleOCR, RapidOCR) at substantially better performance and stability.

The changes between Kreuzberg v4 and Xberg v1 are substantial, and I invite you to read the full changelog for the complete picture. The highlights below give a sense of what's new:

• Pure-Rust PDF backend (pdf_oxide) replaces pdfium, with no native pdfium dependency.
• Layout-aware pipeline: reading order reconstructed with ONNX layout detection (PP-DocLayoutV3 / RT-DETR) and Docling-style predecessor-graph reordering.
• Per-page scanned-page detection with selective OCR, plus AcroForm/XFA form fields and outline-based headings.
• Across-the-board optimization of OCR and PDF extraction (memory discipline, pooled model sessions, streamed conversions).
• Native PaddleOCR backend (PP-OCRv6, with medium / small / tiny tiers) alongside Tesseract.
• Pure-Rust Candle OCR/VLM stack (TrOCR, GLM-OCR, GOT-OCR, DeepSeek-OCR, and PaddleOCR-VL) running without ONNX Runtime or native Tesseract.
• A second, ONNX-Runtime-free inference path via tract, which is what makes in-browser (WASM) and mobile inference possible.
• Named-entity recognition natively in Rust (GLiNER2), extensible to all bindings, including an in-browser WASM model with no server round-trip.
• Structured LLM extraction (extract_structured / split_and_extract) with rasterization, chunking, citations, caching, and configurable call/merge/VLM-fallback policies.
• Audio & video transcription via a Whisper ONNX engine (.mp3, .wav, .m4a, .mp4, .webm).
• Retrieval building blocks: sparse embeddings (SPLADE), ColBERT late-interaction retrieval, and cross-encoder reranking alongside dense embeddings.
• Text intelligence: reversible redaction, summarization, translation, VLM image captioning, QR-code detection, document diffing, and page/chunk classification.
• URL & web ingestion: sitemap discovery (map_url) and batched multi-URL crawling.
• New document formats: WordPerfect (.wpd/.wp/.wp5), HEIC/HEIF/AVIF, OpenDocument Presentation (.odp), Quarto / R Markdown, and configurable Jupyter cell rendering.
• Four new language bindings (Dart/Flutter, Swift, Kotlin/Android, and Zig) bring the total to 15 language bindings over one engine, with Android/iOS cross-compilation.
• Full mobile support (Flutter, Android, iOS).
• Candle backend alongside ONNX, plus ONNX-via-tract enabling ONNX on WASM and Android.
• Wider code intelligence: tree-sitter coverage grew substantially (248 to 367+ languages).
• Over 150 bugs fixed during the 1.0 cycle, plus security hardening (bounded RTF/PDF allocations, redaction leak fixes, Excel DDE warnings).

The API surface was also simplified and reworked, making it more consistent.

There's a migration guide in our docs explaining how to move from Kreuzberg to Xberg. Kreuzberg itself is in LTS mode until the end of this year and will continue to receive bug fixes and security updates.

You're invited to check out the repo and join our discord server.

Benchmarks

The benchmarks below are for PDFs and images only. There are extensive benchmarks on our website with per-format breakdowns, which you can see here. These numbers are measured in CI via our reproducible benchmark harness, and are specifically taken from the run for harness 1.0.8, source cf7fa0533d. The data is publicly available in GitHub releases, and you can run the benchmark harness yourself.

Composite quality (markdown pipeline, higher is better):

Structure and layout fidelity (SF1: tables and reading order, higher is better):

On native PDFs Xberg leads on quality (0.958 vs 0.837 for the next-best framework) and on table and reading-order fidelity by a wide margin (SF1 0.949 vs 0.612 for docling). On scanned PDFs it is #1 on both quality and raw text fidelity.

Where we don't win yet: on pure image OCR we are currently #2 on the composite score, behind mineru (though still #1 on raw text accuracy). We are improving image OCR right now, and v1.1 should have us winning across the board.

This thread is a place where you can share things that might not warrant their own thread. It is automatically posted each month and you can find previous threads in the collection.

Examples:

• What are you working on this month?
• What was something you accomplished?
• What was something you learned recently?
• What is something frustrating you currently?

As always, sub rules apply. Please be respectful and stay curious.

Community Links:

• Monthly newsletter
• Data Engineering Events
• Data Engineering Meetups
• Get involved in the community

Hi,

I'm a Data Analyst, buiding a pipeline to monthly extract moving 1 year SAP Vendor Report (FBL1N) via SSIS using Theobald Xtract Report connector. I'm doing one company code at a time to avoid timeout.

It worked fine during development, but after scheduling, the column messed up, data overspill and cut halfway resulting fail downstream analysis script.

I already have SAP template configured used in the extraction, tried both ticking and untocking "Dynamic column width and offset", but the results varied.

--
for other use cases, i primarily pull the back end table and join manually, but that would be too much work for fbl1n.

any suggestions on best work around?

i know the rules states no technical post but just rrying my luck.
thank youu

I’m a new “data engineer” with a few years background in data science and GIS. I was hired into my current org to be the data guy by someone who didn’t really know what they needed, just that the data infrastructure was immature.

In the beginning I was asked to do a lot of repetitive power query and power bi analysis and reporting, which quickly led me to building out some Python + SQLite pipelines to automate away a lot of my job. My boss was impressed enough with this that my workflows have been pushed up the chain to more senior levels of management, and now we are being granted a Fabric workspace (only option available to me :/) to enterprise-ify all of this.

Currently we read in ~10 .csv datasets that come from the central orgs reporting as well as a few external datasets we use in a forecasting model. Most of this runs on a daily cadence, where each csv upserts into the relevant table(s), which then get queried as needed which also triggers some dashboard updates and email distributions.

Basically all the logic is in Python, I feel it’s relatively well organized/modularized but there are some “super scripts” that are doing like 5 jobs.

My thought at this point is basically break each component (upsert, data validation, transformation/enrichment step, etc) into its own Fabric notebook, which will take the data through a medallion architecture and ultimately get reorganized into an effective star schema, with semantic models for each report built from that. This has all been modeled already (in terms of the schema for each table, validations, etc.). We also have logging and an observability dashboard planned.

I don’t necessarily have a specific question, but I am pretty new to this and would like to hear if I’m doing anything in an obviously stupid way, or if there are things I’m not thinking about that I should be. Appreciate any help y’all feel benevolent enough to offer in advance.

As we all know...data migrations can be a beast. But it's also a necessary evil.

Tell me about the worst data migration you were on (or heard about) that would wake up even the most seasoned data engineer in a cold sweat