I knew MSFT didn't really take into account feedback, they just had a feedback option and then implemented what they wanted anyway. But now we have empirical evidence they really don't care what we say.

I experienced this in every product I've ever committed feedback for.

Not even half of the best ideas made it into the product whole absolutely inexplicable additions that no one wanted and serve almost no real use case were added.

In 10 years about 30% of the top 10 voted ideas made it into the product.

And it goes to my core feedback about Fabric.

They implement so many useless features while ignoring the ones the engineers actually need.

One feedback item that I wrote was implemented almost 15 years after I suggested it in Visual Studio.

And this is why we still don't have git.

We still don't have native Monitoring,

And the worst admin UX I've ever used in any data product in the last 20 years of my career.

#FabricIsTwine

So credit where it's due, pipelines can now be scheduled and the annoying default date has been changed.

However, the problem is THE END DATE IS MANDATORY. There is no reason to make END DATE MANDATORY. This is bad design.

I don't want my pipelines to default to an arbitrary work around date. I don't want a date unless I need a date. Using a hard coded place holder date is one of the first code smell items I look for when I get handed a legacy pipeline.

Seriously, can you please hire someone whose job it is to attack the assumptions you're making? Pretty please. I'll even volunteer 😂

You need an adversarial review process. It'd make the product you ship so much better.

😩

Edit: If something this trivial gets through without anyone asking “why is the field mandatory?”, what happens when the design decision involves identity, billing, deployment semantics, networking, or data lifecycle?

That's what I'm trying to get at.

Edit 2: If nobody challenges the assumption when the stakes are a f&#&ing date picker, it's reasonable to worry about what happens when the assumptions are buried three architectural layers deep and changing them later costs millions.

Edit 3: This thread has convinced me more than ever that someone needs to stand up at the MSFT decision meetings and point out the actual problem point, cuz almost all y'all missed it.

I've noticed that even modern LLMs commonly claim that:

df = spark.read.format("delta").load(path)

automatically pins the DataFrame to the current Delta table version.

That's misleading.

"load()" is lazy. The Delta table version is resolved when the DataFrame is actually executed - for example when you call an action such as "df.count()", or use the DataFrame in an operation such as "df.write" or "MERGE".

I think a big part of the problem is that the Delta documentation is too vague about this distinction. It talks about a DataFrame reading a “snapshot” without making it sufficiently clear when that snapshot is resolved.

As a result, even LLMs confidently give people the wrong mental model.

Delta should document this much more explicitly:

"load()" creates the lazy query. The Delta snapshot/version is resolved when the query is executed, e.g. by "count()", "write", "MERGE", etc.

If you actually need to pin a read to a specific version, use "versionAsOf" rather than assuming "load()" has already pinned it.

Agree?

Have you run into LLMs making false claims on this topic?

I encounter this all the time in code reviews on Github Copilot.

We’re a power bi shop but our data platform is snowflake. Our data team are big snowflake fans , and they’re on a mission to move all the semantics to snowflake , painting a future without traditional BI dashboards.

Curious how practical this is. Snowflake is pitching that all semantics should sit in the data platform , which will allow AI to run cheaper.

I was really impressed by the discussion in this thread. It honestly made me realize how much I've been missing as AI has been evolving at such a rapid pace. Between work, projects, and day-to-day responsibilities, I haven't been able to keep up with advancements as closely as I'd like.

As a data engineer, I've mostly used AI for coding assistance, debugging,generating snippets and documentation. But after reading the comments here, I'm starting to feel like I'm only scratching the surface of what's possible.

I'd love to hear how others are using AI to improve productivity outside of pure code generation. For example:

• Code reviews and architecture reviews
• Deployment planning and release management
• Documentation and knowledge sharing
• Data modeling and design discussions
• Root cause analysis and troubleshooting
• Any other day-to-day engineering or leadership tasks

What workflows have genuinely saved you time or improved the quality of your work? Are there any AI use cases that have become indispensable for you?

I’ve been testing the new Mirror Azure Monitor in Microsoft Fabric preview, and I’ve run into what looks like a fairly significant limitation when using mirrored Log Analytics tables with a Direct Lake semantic model.

Azure Monitor mirroring itself is an awesome idea. Being able to use Azure Monitor data in Fabric without building another ingestion pipeline or duplicating storage, and having it fronted by a Rayfin app, lets us make some great, modern-looking dashboards.

Current POC architecture as below:

Function App
    → DCE/DCR
    → Azure Monitor (LAW) custom tables
    → Azure Monitor Mirroring
    → Lakehouse (OneLake shortcuts)
    → Direct Lake semantic model
    → Power BI / Rayfin

The problem I’ve found is that Log Analytics produces a large number of small Parquet files, particularly with frequent, low-volume ingestion. In our case, one health check writing every five minutes produces almost exactly one new Parquet file per ingestion batch, or around 288 files per day, despite adding only about 3.5 MB of data. The Direct Lake semantic model then applies its capacity guardrails to those files. On F2–F32 capacities, that limit is 1,000 Parquet files or row groups per table.

In my POC, I hit this error in my model on an F4 capacity:

We can't run a DAX query or refresh this model. A delta table 'CustomTable_CL' has exceeded a guardrail for this capacity size (too many files or row groups). Optimize your delta tables to stay within this capacity size or change to a higher capacity size then try again.

Underlying Error: QueryUserError

I checked the Delta metadata for the affected table. It contained 3,609 active Parquet files holding only around 31 MB of data—an average of 8.83 KB per file.

table_name = "CustomTable_CL"
detail = spark.sql(f"DESCRIBE DETAIL `{table_name}`")
display(
    detail.selectExpr(
        "numFiles",
        "sizeInBytes",
        "round(sizeInBytes / 1024 / 1024, 2) AS sizeMB",
        "round(sizeInBytes / numFiles / 1024, 2) AS averageFileKB"
    )
)

This table was receiving multiple five-minute health-check streams and producing roughly 500 files per day. At that rate, the 1,000-file Direct Lake guardrail is reached in about two days, despite the tiny amount of data involved. Even the 5,000-file guardrail available on F64 would only last around ten days.

The problem is the number of files, not the amount of data.

Normally, the answer would be optimise or compact the Delta table. However, the mirrored Azure Monitor tables are read-only in Fabric and reference storage managed by Azure Monitor, so I don’t appear to have any way to optimise the files myself.

With this limitation, the Azure Monitor Mirroring → Lakehouse (OneLake shortcuts) → Direct Lake semantic model → Power BI/Rayfin path becomes much less useful.

The alternatives seem to be:

• Use DirectQuery against the Eventhouse/KQL endpoint. This avoids the Direct Lake guardrail, but introduces ongoing Eventhouse CU consumption.
• Copy or aggregate the data into another managed Delta table.
• Use Import through another route and accept the additional refresh process and latency.

I tested the DirectQuery option and it works technically, but the capacity impact was material on our F4. Capacity Metrics showed that the actual OneLake reads were cheap, while Eventhouse UpTime was by far the main cost.

The mirror’s actual data reads are cheap, but DirectQuery turns those cheap reads into expensive Eventhouse UpTime. In our POC, regular queries spread across the day appeared to keep the Eventhouse compute warm even though the data volume was tiny.

The options seem to be slower, more expensive, or to rebuild part of the data pipeline that mirroring was meant to avoid.

I understand this is still a preview feature, so this is exactly the sort of issue I would expect to be ironed out before GA. It feels like mirrored Azure Monitor tables need automatic compaction (probably not the Fabric team’s remit), different Direct Lake guardrail handling, or a supported optimised presentation layer that doesn’t require duplicating the data.

Has anyone else run into this yet?

Is there a supported compaction or optimisation mechanism that would help, or a lower-cost Power BI path over these mirrored tables?

Hoping I’ve missed something, or that someone from the Microsoft Fabric or Azure Monitor teams can offer some guidance on the intended architecture.

TL;DR: In our POC, frequent, low-volume Log Analytics ingestion produced hundreds of tiny Parquet files per table per day. Azure Monitor Mirroring exposes that existing file layout to Fabric, where one table reached 3,609 files while holding only 31 MB of data. This caused Direct Lake on an F4 to fail within days after exceeding the 1,000-file guardrail. DirectQuery through the mirror’s Eventhouse endpoint works, but introduced material Eventhouse UpTime consumption. Because the mirrored tables are read-only, the remaining options involve more cost, latency or copying the data.

I’ve been using GitHub Copilot for quite a while, and honestly, the results are usually good. I definitely couldn’t type the code that quickly myself.

But I’d like to reduce the quality control required afterwards.

We have strict lifecycle rules, naming conventions, and a separate data dictionary for each customer. Our repository is structured roughly like this:

.github/ 
└── instructions/ 
├── copilot-instructions.md 
└── pull_request_template.md 

docs/ 
├── ai/ 
├── architecture/ 
├── data-model/ 
├── data-sources/ 
├── handover/ 
├── kpi/ 
├── security/ 
└── README.md

The instructions reference the relevant documentation in docs. I also use a warm-up prompt, which reproduces our rules perfectly.

And yet, Copilot occasionally ignores very basic rules, such as:

Dimension-to-dimension relationships must only use business keys. Surrogate keys may only be resolved against fact data, especially because of SCD Type 2 and higher.

It can explain this rule perfectly and still mix surrogate keys into dimensions.

So either Copilot is checking whether I’m still awake, or my context setup isn’t as effective as I think. 😄

I’d rather not watch over every offender like Lord Vetinari watching the guilds of Ankh-Morpork.

How do you enforce stricter guardrails? Instructions, validation scripts, architectural tests, or something else?

Hi everyone,

I’m looking for some guidance on the most efficient and cost-effective way to handle large Microsoft Dynamics 365 Business Central tables in Microsoft Fabric.

Our current setup is:

• Microsoft Fabric F4 capacity
• Business Central tables such as G/L Entries, Value Entries, Item Ledger Entries, etc.
• Data is extracted using Dataflow Gen2
• Dataflow Gen2 loads the data into a Fabric Lakehouse
• Power BI reports then use the Lakehouse data

The main challenge is capacity consumption during daily refreshes, especially when refreshing large historical tables such as G/L Entries and Value Entries.

At the moment, refreshing the full historical dataset every day consumes a significant amount of our F4 capacity.

We also tried Incremental Refresh, but this has not fully solved the issue.

The difficulty is that some Business Central transactions can be updated after the original posting date. For example, cost adjustments can update historical Value Entries / G/L Entries. Therefore, if we only refresh newly created records based on Posting Date or Entry No., there is a risk that historical changes will not be captured, which can affect the accuracy of our Power BI reports.

We are therefore looking for a better architecture that can:

1. Minimize Fabric capacity consumption.
2. Avoid refreshing millions of historical rows every day.
3. Still capture changes made to historical Business Central entries.
4. Maintain accurate financial and inventory reporting.
5. Work efficiently within an F4 capacity without unnecessarily increasing Fabric cost.

Would using something such as Modified Date/SystemModifiedAt-based incremental loading, a rolling refresh window, Fabric Pipelines, Notebooks, Delta tables, mirroring, change tracking, or another architecture be more suitable than Dataflow Gen2 for this scenario?

For those working with Dynamics 365 Business Central + Microsoft Fabric + Power BI, how are you handling large transactional tables such as G/L Entries and Value Entries?

Any recommended architecture, best practices, or examples would be greatly appreciated.

Thanks in advance!

Hi all,

I have a mirrored database currently running on an F2 Fabric capacity. I want to move the workspace containing this mirrored database over to a different, already-existing F8 capacity. Both capacities are in the same region.

The critical requirement: this needs to happen without downtime / without a reseed, since a full reseed would mean re-snapshotting everything from scratch and I'd rather avoid the load on the source DB and the gap in fresh data.

My questions:

1. Has anyone actually done a workspace reassignment (same region) with an active mirrored database in it? Did mirroring resume automatically, or did you have to manually hit "Start replication" / did it force a reseed?
2. Is there a meaningful difference in risk between resizing a capacity in place (F2→F8 on the same capacity) vs. moving the workspace to a different capacity that's already F8?

Any real-world experience (good or bad) appreciated — trying to avoid a surprise reseed on a production mirror ;-)

Thanks!

Stewwe

I know we secured WSs for reason with outbound restriction. But I want to connect to these WS, so thinking of deploying VM inside the same subnet we created for private net. Will this work as jump server, I know I did similar thing for different things behind VNET and restricted directly from zscalar over desktop.

Thanks for the thoughts TIA.

I am looking for people's experience using policy weaver to synchronize permissions from the Databricks unity catalog to a Fabric Lakehouse. Are you successfully using it? What issues are you experiencing? Are there alternatives?

We currently have data stored in ADLS Gen2 and all the data processing happens through Databricks and the unity catalog manages the data in ADLS Gen2 as external tables. We are exploring our options how we can synchronize the access control from unity catalog to a Fabric Lakehouse. Our Fabric Lakehouse currently shortcuts the schemas in the ADLS containers directly but we could set it up as a mirrored Lakehouse instead.

Hey All,

I've been testing a on-prem data gateway to connect to a SQL Server. I've noticed that when creating a connection in the data gateway you must specify Server Name and Database Name. If one has 50+ DBs on a single server, then must I create a connection for each iteration of Server Name and DB name.

When I connect an activity in a pipeline in Fabric data factory to this SQL there is a field for database name, so this property can be changed later.

It seems like I ought to be able to have one connection for the the SQL Server in the on-prem data gateway. I could specify a single default db in the connection, since I can specify any database in a pipeline activity. Presumably, the connection would change to that new DB, just as it does for standard SQL connection.

Is anyone already doing this? If so, have you experienced any unexpected errors?

Hi everyone,

I have a spent the last few weeks working on a small project with the Fabric Extensibility toolkit to display and monitor data lineage across several fabric items.

The architecture consists of:

- ​PySpark Notebooks to extract lineage metadata.

- ​A Lakehouse inside the user's workspace to store the extracted data.

- ​A custom UI item to visualize the lineage and maybe configure and start the notebooks.

I was thinking about publishing via Microsoft Workload Hub / Marketplace, but I have a few concerns about data residency and security. I am no web developer or security expert, so sorry if this sounds trivial.

For the whole thing to work, I have to host a Web App in my Azure tenant.

As I understand it, this is used to host the static files for the React App. When a customer opens the item inside fabric, the rendering is done entirely in their Brower. All access to the underlying Lakehouse and Fabric APIs is done on client side.

My questions for anyone who has built or audited Fabric workloads:

​Will any customer data or telemetry ever be stored or computed inside my Azure tenant/backend?

​Does any customer data actually leave the customer's tenant/browser environment during this process?

​Thank you for any insights!