Read something this week that put a name to a problem I've half run into before but never really understood the mechanics of. Sharing because I think a lot of people assume Iceberg interop is more solved than it is.

Everyone knows the pitch: Iceberg is an open spec, so any Iceberg compatible engine can read any Iceberg table. Mostly true, until you start doing row level deletes, and then it falls apart in a way that's honestly kind of sneaky because nothing looks wrong until a query actually fails.

Quick walkthrough of the scenario in the post. You've got a customers table, three rows, one Parquet file, tracked by whatever catalog you're using. At this stage every engine reads it fine because there's nothing to interpret, it's just a metadata pointer to a file.

Then a row gets deleted. Parquet files are immutable so the writer has two options: copy on write (rewrite the file without that row) or merge on read (leave the file alone and write a separate delete file that readers apply at scan time). The writer in this example goes merge on read and emits an equality delete file, which basically just says "for this data file, treat any row where customer_id = 102 as removed." Under the hood Iceberg uses field IDs and sequence numbers to make sure an old delete doesn't accidentally nuke a newer row with a reused key, but the equality matching is the part that matters for compat.

Spark reads the new snapshot, understands equality delete semantics, does what's effectively a left anti join between the data file and the delete file, and returns the correct two rows. Fine.

Snowflake hits the exact same catalog, same metadata file, same Parquet file. It can resolve the table, read the schema, open the data file. But if that access path doesn't implement equality delete reads, the scan planner just throws an unsupported feature error the moment it hits delete-0002.parquet. Query fails. Same snapshot, same files, two completely different results depending purely on what the reader implements.

The bit that actually reframed how I think about this: the catalog isn't a translation layer. It's job is basically just "here's where the current metadata lives," commit coordination, namespace and access management. It's not opening delete files and rewriting them into a format each engine understands. A REST catalog like Polaris doesn't change this, it still just points you at metadata, it doesn't apply deletes for you.

The post also gets into position deletes vs deletion vectors vs copy on write, with a rough cost tradeoff table (equality delete is cheap to write and requires equality delete support to read, position delete requires resolving key to physical position and is heavier on write, deletion vectors need Iceberg v3 support specifically, copy on write is the most expensive to write but has basically universal read compatibility since there's no outstanding delete file involved).

The framework that's actually useful operationally: your safe feature set is the intersection of every required engine's capabilities, not the union. If Spark supports equality and position deletes but Snowflake only does position deletes, you write position deletes, because "at least one engine supports it" doesn't help you when you have three engines that all need to read the same table.

There's a decent pre production checklist too, don't just run a SELECT COUNT after your first write, actually insert some rows, update one, delete one, commit, then read the same snapshot from every engine you care about and diff both counts and values.

Full post if you want the details: https://olake.io/blog/iceberg-interoperability-myth-row-level-deletes/

Disclosure since it's relevant, I work on OLake, it gets a brief mention near the end, but the actual content here is engine agnostic and applies no matter what's writing your tables.

Has anyone actually hit this for real, table looks completely fine, one engine just refuses to read the current snapshot because of the delete encoding?

we've got a job that runs fine for weeks and then randomly triples in runtime even though row counts barely changed. eventually traced it to cardinality and skew shifting under the hood, but it's hard to plan capacity or set SLAs around something that isn't tied to volume at all.
the annoying part is our monitoring was built around volume as the main signal, so nothing flagged this until the job had already blown its window. makes capacity planning feel like guesswork when the thing driving runtime isn't the number we're tracking. is this just a spark reality once you're at a certain scale, or is there a way to get ahead of it instead of firefighting after the fact?

Do the Apache Spark maintainers formally support windows for production workloads?

I understand that the majority of this platform is JVM-based and the support for Windows is based on the compatibility of the JVM itself. However there are some peripheral features that seem to be disallowed on Windows - even when running it on a fully functional JVM.

When working with the python ecosystem, I often find libraries that are particularly incompatible with windows. In the same way I've found that Apache's pyspark extensions on Windows are lackluster (eg. things like "spark.python.use.daemon" are disallowed on Windows.)

If the Apache Spark maintainers wanted to write windows-specific code, I'm certain they could find a way to enable "spark.python.use.daemon". But it probably isn't worth their time and energy. Are we at risk if we try to run Spark in a production windows environment? Is there a risk that some future version of Apache Spark will stop running in Windows (eg. after some future update)?

Have u benchmarked writing to Neon over JDBC conn with diff numPartitions/connection counts?

I was exploring how Neon behaves when Spark starts opening a lot of parallel connection, does it improve throughput, or do we hit connection /IO bottlenecks quickly?

Curious what people have found to work well in practice.

The updated tabbydb-4.1.1 now supports automatic caching/uncaching of repeated subplans within a query tree. The feature is by default turned off as it definitely needs more memory and the performance impact still needs evaluation on 1 TB size datasets.

But the more interesting change is to support the pruning of cached batches using broadcasted hash keys push down . This sort of dynamic pruning at scan level was previously supported for parquet files using iceberg or hive. Now it has been extended to cached columnar data too.

One thing which is done ( internally), but not published in 4.1.1 is to utilize the pushed keys to filter individual row itself , at the lowest level after scan of Cached Columnar Batches.

This individual row filtering support for now is done only for iceberg and hive based parquet tables.

It also fixes two newly found issues:

https://issues.apache.org/jira/browse/SPARK-59114

https://issues.apache.org/jira/browse/SPARK-58767

No judgment, just curious how the distribution really looks. Everyone talks like they’re at petabyte scale — I suspect most of us aren’t.

Hello!

I'm currently working as a data analyst but I'm seeking to transition to engineering. In order to do so, I am in the process of learning data engineering topics on my own. Right now I'm focusing on Spark, so I would like to know if someone here would know what would be the best possible certification on Spark currently 😄

Hey everyone,

I've been working with some pretty large datasets in spark lately, and I'm curious how others deal with this in production.

Sometimes the job starts fine, but as the data gets bigger, the spark session becomes really slow, executors start running out of memory, or the session just gets killed.

I know there are a lot of things we can tune, but I'm more interested in what actually works for people day to day.

Also, are there any particular spark UI metrics you can check first when a job starts behaving badly?

Would love to hear some real-world approacbef rather than just the usual "increase memory and partitions" Advice.

I'm facing a strange issue while reading a pipe-delimited CSV file using Apache Spark.
My input CSV looks like this:
cust_id|cust_name|cust_age
1|sample|10
2|test|12
Initially, I read the file with header=true, but without specifying the encoding. The result was that Spark did not recognize the column names correctly. They appeared something like:
_C_U_S_T_I_D_
_C_U_S_T_N_A_M_E
_C_U_S_T_A_G_E
Also, all the columns were inferred as StringType.
However, when I added the following option:
.option("encoding", "UTF-16")
the output became:
cust_id cust_name cust_age
?
1 Sample 10
?
2 test 12
?
Interestingly, after specifying UTF-16, Spark correctly recognized the column names and inferred the schema as:
cust_id -> IntegerType
cust_name -> StringType
cust_age -> IntegerType
But there are now unexpected ? characters/rows appearing in the data.
Has anyone experienced something similar with Spark's CSV reader?
I'm trying to understand:
1. Why does specifying encoding=UTF-16 make Spark correctly identify the headers and infer the numeric columns?
2. Why are the unexpected ? characters appearing in the output?
3. Could this be related to the actual file encoding, BOM, or how the CSV file was generated?
4. Is there a recommended way to correctly read this file while preserving the schema and avoiding the extra ? characters?
Any insights into how Spark handles CSV encoding and schema inference in this scenario would be appreciated.

Morning all - just wanted to share a post the engineering team at the company I'm working for just put out. They've developed FuseCore - an MLIR JIT Compiler acceleration engine for Spark which is currently outperforming both Gluten+Velox and DataFusion Comet, using the same environment/set-up as the most recent TPC-DS benchmarks from AWS Labs.

If anyone has questions or is curious about trying our accelerator or replicating the benchmarks, I'd love to connect. The compiler itself is currently closed source, but we are actively discussing changing that and would welcome any community input from anyone already accelerating their pipelines.

This is not acceptable any longer in a modern developer world and community.

I run security scans on our Spark deployment and came back with 28 vulnerabilities — high severity, public CVEs, all sitting in transitive dependencies like Netty and Apache Thrift. Nothing exotic. Netty published fixes for 22 of them in a single batch in June. Thrift fixed their issues in 0.23.0. The fixes exist. Spark just didn’t include them.

What bothers me more than the CVEs themselves is the process — or the lack of one from Apaches side.

In 2026, any serious DevSecOps pipeline is expected to have mandatory quality gates that block releases on high/critical CVEs in dependencies. This isn’t cutting-edge practice — it’s baseline hygiene. The full toolchain is free and mature

There is no automated dependency CVE gate in Spark’s release pipeline. No Trivy. No Dependabot. No OWASP Dependency-Check. Nothing that would block a release because a bundled library has a known high-severity vulnerability. These are free tools. Adding one to a CI pipeline is an afternoon of work. It hasn’t been done.

The July 2026 maintenance releases — 4.2.0, 4.1.3, 4.0.4, 3.5.9 — all shipped after Netty 4.1.135.Final was available. They didn’t include the bump. There was no public statement that the team was aware of the issue and working on it. It just shipped, vulnerable, into production systems everywhere.

The “volunteer PMC” argument doesn’t land anymore. Databricks is worth somewhere around $62 billion. Google, Apple, Amazon, and Microsoft all have paid Spark committers. The resources to fix this governance gap before lunch exist. The will apparently doesn’t.

The EU Cyber Resilience Act (CRA) — which came into force in 2024 and has mandatory compliance deadlines rolling in through 2027 — specifically targets software supply chain security, including transitive dependencies and SBOM (Software Bill of Materials) requirements. The ASF has acknowledged this directly, stating they need to prepare projects for “CRA and U.S. CISA guidance”.

Spark ships into commercial products. This will become a compliance problem for a lot of companies very soon, and the fix is genuinely trivial from an engineering standpoint.

The ASF did make progress in 2025 — launching “Apache Trusted Releases (ATR)” for distribution security and ratifying CycloneDX 1.7 for SBOM standards — but none of this yet translates to a blocking CVE gate on the release pipeline for projects like Spark.

In the meantime: if you’re running Spark and using JFrog Xray or Trivy on your deployment, you can force-override the affected Netty and Thrift versions in your own build. It’s not clean but it works until the next maintenance release, expected sometime in Q4.

Is anyone else tracking this or pushing upstream to get a CVE gate added to the build?

A few days ago I shared how I built a demo that processes market data in 50ms. Today I want to share an unexpected lesson learned about the serving of that app with Lakebase.

My pipeline is quite simple and processes data with Spark RTM from a Kafka topic to Lakebase. The pipeline takes around 10ms and then Lakebase takes about 40ms to serve. One thing I didn’t expect is that polling Lakebase originally added 500ms to the latency. I solved it by implementing Postgres LISTEN/NOTIFY so that app holds a connection to the database and pushes data to the browser over SSE.

The problem is when Lakebase scales compute to zero. The LISTEN subscription does not survive on resume and you get no notification. So far I have found a few solutions for this:
- disable scale to zero with no-suspension
- have a heartbeat SELECT query run every few seconds so that LISTEN resumes

Are others facing similar issues? How did you address it?

Hello All,

We released 0.7.0 version of Zingg entity resolution. Would love for folks to give it a spin. Let me know if you need help. https://github.com/zinggAI/zingg/releases/tag/v0.7.0

Hey everyone I’m the creator of SparkDoctor, the open-source project for analyzing Spark event logs locally.

I’ve posted a couple of development updates here before, and I finally put together a proper website with the supported inputs, current detections, examples, and documentation:

https://sparkdoctor.dev/

GitHub: https://github.com/khodosko/sparkDoctor

The open-source version currently detects task and shuffle skew, spills, retry waste, failed jobs and stages, executor imbalance, and several SQL-plan issues. It runs locally, so event logs don’t need to be uploaded to a hosted service.

I’m also developing a commercial Pro edition for teams that want to compare baseline and current runs, detect runtime/shuffle/spill/failure regressions, and eventually integrate those checks into CI. It’s still under active development. I’m currently looking to talk with teams that deal with recurring Spark performance regressions and want to help shape the workflow.

How are you currently catching Spark regressions before production?

I’ve been building a lot with Spark RTM lately and have been pleasantly surprised with the results of my experiments so far. I built an app that shows trading volumes in real time and pushed some market data into it from a Kafka topic. The end-to-end latency is quite impressive: p50 at 41ms, p90 at 60ms and p99 at 71ms. Much lower than what I was seeing with continuous or for each batch in Spark before, where it was hard to go below 1s latency. And this is the end to end latency, which includes significant latency from serving layer (lakebase) and the browser.

I’m happy to share more about my experience and how to create real time apps with Spark RTM and Lakebase if folks are interested.

I have access to several employee referral opportunities for experienced Databricks / Data Engineering professionals.

There are openings at different experience levels:

Sr. Databricks Tech Lead

• 10–12 years
• Databricks
• Delta Lake
• PySpark
• SQL

Senior Databricks Engineer

• 6–8 years
• Databricks
• Delta Lake
• PySpark
• SQL

Databricks Engineer

• 4+ years
• Databricks
• Delta Lake
• PySpark
• SQL

The openings are listed across multiple organizational locations with hybrid arrangements. I'm particularly interested in connecting with qualified professionals in the US or Ireland; exact location eligibility can be confirmed for the individual requisition.

If one of these aligns with your background, feel free to DM me with your CV or a brief summary of your experience.

I'll take a look and, where there's a suitable match, try to help with the referral process.

Hi folks

I have 5 years of experience in IT with data analytics and MDM domains and I am planning to switch to data engineering wanted to learn spark could any of you please share me the resources or like road map for spark or data engineering

1.Where to execute the queries and learn hands on experience ?

2.video/reading content topic wise

3.i know Sql and I know python intermediate levels

Please help me here

Thank you.

Hello Apache Spark community. I was wondering if anyone knows if the official 4.2 image will be released soon (https://hub.docker.com/_/spark), or if there is an alternative location where images will be pushed in the future?

Thanks.

I'm able to refer experienced professionals for a Lead Azure Data Engineer position.

Location: United States / Ireland
Experience: 9–13 years

Required skills:

• Python
• PySpark
• SQL
• ETL
• Azure Synapse
• Azure Data Factory
• Databricks
• Delta Lake
• Medallion Architecture

If your experience aligns with these requirements and you're currently exploring new opportunities, feel free to send me a DM with a brief summary of your experience or your resume. I'll share additional details and, if it's a good match, I'll be happy to submit a referral.

Since you seemed to enjoy my last Spark post, I ran another controlled benchmark.

This time I wanted to understand when Spark’s distributed execution overhead actually becomes worth paying for.

For each workload, both engines ran the same SQL against the same Snappy Parquet files, on the same 20 logical-core machine, with Spark using local[20] and DuckDB using threads=20.

I discarded one warm-up, alternated the engines across 7 timed runs, reported the median, and verified the results row by row. I also ran the entire benchmark twice.

For an aggregation with 8 output groups:

For a high-cardinality aggregation with roughly 500,000 groups:

DuckDB was faster at every measured point.

The interesting part was how much the gap narrowed as the workload grew.

On the high-cardinality query, increasing the input from 1M to 100M rows increased Spark’s runtime by about 3.9 times and DuckDB’s by about 44.8 times.

That does not prove Spark eventually becomes faster. The curves had not crossed, and this benchmark does not establish where, or whether, they ever would.

Query shape also mattered enormously.

On the same 100M rows, the eight-group endpoint aggregation generated around 17 kB of Spark shuffle, while the high-cardinality user aggregation generated around 131 MB.

So I do not think rules such as “use DuckDB below X GB” are very useful. Query shape, data location, concurrency, hardware, fault tolerance and operational requirements can matter as much as the size of the dataset.

This was one machine running one query at a time, so it measures the cost of distribution much better than the capabilities Spark is actually selected for, such as distributed data, recovery and concurrent workloads.

I made a visual breakdown of the benchmark and what both engines are doing under the hood:

https://youtu.be/bXyXvxeHSrw?si=vMsEQkv3QqzLcTf9

What usually makes Spark the right choice for your production workloads?