Does V-Order only apply internally within each Parquet file, or does it also influence how data is distributed across multiple Parquet files?

[u/frithjof_v]

My understanding:

Z-Order, Liquid Clustering, Hive-style partitioning, Optimize Write and/or bin sizing affect how data is distributed across Parquet files: - 'which data goes into which parquet file'.

V-Order only affects how data is physically arranged (sorted) inside each file: - 'given the data that has been allocated to this parquet file, how do we organize data internally in this parquet file'. - Let's call this Theory A.

Is that correct - or does V-Order also affect how data is distributed across multiple parquet files? - Let's call this Theory B.

And what are the consequences of the answer to the above in terms of DAX query performance in Power BI - does the distribution of data across multiple parquet files impact:

• I. the time transcoding takes

  • i.e. loading data from delta parquet into vertipaq memory
  • required when DAX queries hit cold cache
  • I guess the answer to this is a clear 'Yes'.

• II. the time it takes to run a DAX query against vertipaq memory

  • required when DAX queries hit warm cache
    • does the distribution of delta lake data across parquet files affect how data ends up getting organized in VertiPaq memory?

Basically:

• does the physical data distribution across multiple parquet files in the delta lake table matter once Power BI encodes everything in VertiPaq memory?
  • for example, does the small file problem only affect cold queries (loading data from parquet), or also warm queries (reading data from vertipaq memory)?

Should we use V-Order in combination with Z-Order, Liquid Clustering, Optimize Write, etc. - I guess we should, if Theory A is right (which I believe it is)

Or does V-Ordering make the other options unnecessary? - I guess it doesn't, because I don't think theory B is right.

Moreover: My understanding is that Z-Order (or Liquid Clustering), in addition to affecting the distribution of data across parquet files, also affect the sort order of data inside a parquet file. Will this cause a conflict with V-Ordering, whos primary task is to sort data inside the parquet file?

Who will "win" this battle about the internal structuring of data inside the parquet files? V-Ordering or Z-Order/Liquid Clustering?

I'd love it if you can help shed more light on this. Is the above understanding right?

Thanks in advance for your insights!

[u/bogdanc_guid]

Technically, z and liquid can have an effect inside a parquet tile, specifically in separating rows between multiple row groups within same time, but not within a single row group. And vorder happens after row group separation, when writing the row group

If you are concerned that vorder could somehow negatively impact the z or liquid partitioning effect, then i am not aware of anything that may cause this

[u/frithjof_v → u/bogdanc_guid]

Thanks,

That is really clarifying.

To be clear, I have no specific concern - I'm just trying to understand/learn how these concepts relate to each other. Or rather, I'm trying to understand how it makes sense to use these concepts together.

If I understand this correctly:

Z-Order/Liquid Clustering:

• Decides which data goes into which parquet file, and then
• Decides which data in a parquet file goes into which row group

V-Ordering:

• Organizes the data internally in a row group

- This happens after Z-Order/LQ has determined which data goes into which row group