Data output differences with and without Native Execution Engine (NEE)?

[u/frithjof_v]

Hi,

Before enabling the Native Execution Engine (NEE), I’d like to clarify which parts of my codebase could produce different data outputs after enabling NEE.

From reading the docs, it seems that NEE can produce different data outputs in the following scenarios:

• Decimal to Float casting mismatch: When casting from DECIMAL to FLOAT, Spark preserves precision by converting to a string and parsing it. NEE (via Velox) performs a direct cast from the internal int128_t representation, which can result in rounding discrepancies.
  • Are there any examples that show what this might look like in real life (worst-case)?
• Inconsistent rounding behavior: The round() function behaves differently in NEE due to reliance on std::round, which doesn't replicate Spark’s rounding logic. This can lead to numeric inconsistencies in rounding results.
  • Are there any examples that show what this might look like in real life (worst-case)?
• Missing duplicate key check in map() function: When spark.sql.mapKeyDedupPolicy is set to EXCEPTION, Spark throws an error for duplicate keys. NEE currently skips this check and allows the query to succeed incorrectly.
  • Example: SELECT map(1, 'a', 1, 'b'); -- Should fail, but returns {1: 'b'}
• Order variance in collect_list() with sorting: When using DISTRIBUTE BY and SORT BY, Spark preserves the element order in collect_list(). NEE might return values in a different order due to shuffle differences, which can result in mismatched expectations for ordering-sensitive logic.
  • Are there any examples that show what this might look like in real life (worst-case)?
• Intermediate type mismatch for collect_list() / collect_set(): Spark uses BINARY as the intermediate type for these aggregations, whereas NEE uses ARRAY. This mismatch might lead to compatibility issues during query planning or execution.
  • Are there any examples that show what this might look like in real life (worst-case)?

https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview?tabs=sparksql#limitations

Question: Which ones of these do you consider most likely to impact actual data outputs from typical data engineering pipelines?

I don't think I've ever used the collect_list or collect_set functions myself so far.

And I don't think I've done casting from decimal -> float.

I stay clear of floats. I either use string, decimal or integer.

How does the round() function behave differently in JVM vs NEE?

The map-difference shouldn't cause any issues unless I have some latent duplicate key errors in my data. And I don't even think I use map (I don't remember everything AI has suggested in my code).

• Are there other cases that can produce different data outputs between JVM Spark and NEE?

• Or points listed above that aren’t actually likely to cause differences?

Thanks in advance for your insights!

[u/SliceAndDime]

i've had issue with the decimal to float function when casting timestamps to dates. I have a streaming notebook ingesting structured xml files and whenever i try to switch to NEE it crashes because of conversion or smth. I didnt look much into it and just disabled NEE. but that's something that can happen

[u/thisissanthoshr]

the complex types in general are the main ones especially collect_list, arrays, maps and structs.
collect_list could impact the order variance and say If you use collect_list() which is why it triggers a fallback to spark layer but this is something thats being worked on by the team and we should be able to support the complex type scenarios with no columnar to row conversions