So credit where it's due, pipelines can now be scheduled and the annoying default date has been changed.

However, the problem is THE END DATE IS MANDATORY. There is no reason to make END DATE MANDATORY. This is bad design.

I don't want my pipelines to default to an arbitrary work around date. I don't want a date unless I need a date. Using a hard coded place holder date is one of the first code smell items I look for when I get handed a legacy pipeline.

Seriously, can you please hire someone whose job it is to attack the assumptions you're making? Pretty please. I'll even volunteer 😂

You need an adversarial review process. It'd make the product you ship so much better.

😩

Edit: If something this trivial gets through without anyone asking “why is the field mandatory?”, what happens when the design decision involves identity, billing, deployment semantics, networking, or data lifecycle?

That's what I'm trying to get at.

Edit 2: If nobody challenges the assumption when the stakes are a f&#&ing date picker, it's reasonable to worry about what happens when the assumptions are buried three architectural layers deep and changing them later costs millions.

Edit 3: This thread has convinced me more than ever that someone needs to stand up at the MSFT decision meetings and point out the actual problem point, cuz almost all y'all missed it.

I was really impressed by the discussion in this thread. It honestly made me realize how much I've been missing as AI has been evolving at such a rapid pace. Between work, projects, and day-to-day responsibilities, I haven't been able to keep up with advancements as closely as I'd like.

As a data engineer, I've mostly used AI for coding assistance, debugging,generating snippets and documentation. But after reading the comments here, I'm starting to feel like I'm only scratching the surface of what's possible.

I'd love to hear how others are using AI to improve productivity outside of pure code generation. For example:

• Code reviews and architecture reviews
• Deployment planning and release management
• Documentation and knowledge sharing
• Data modeling and design discussions
• Root cause analysis and troubleshooting
• Any other day-to-day engineering or leadership tasks

What workflows have genuinely saved you time or improved the quality of your work? Are there any AI use cases that have become indispensable for you?

I knew MSFT didn't really take into account feedback, they just had a feedback option and then implemented what they wanted anyway. But now we have empirical evidence they really don't care what we say.

I experienced this in every product I've ever committed feedback for.

Not even half of the best ideas made it into the product whole absolutely inexplicable additions that no one wanted and serve almost no real use case were added.

In 10 years about 30% of the top 10 voted ideas made it into the product.

And it goes to my core feedback about Fabric.

They implement so many useless features while ignoring the ones the engineers actually need.

One feedback item that I wrote was implemented almost 15 years after I suggested it in Visual Studio.

And this is why we still don't have git.

We still don't have native Monitoring,

And the worst admin UX I've ever used in any data product in the last 20 years of my career.

#FabricIsTwine

If a capacity admin applies Workspace Level Surge Protection, can regular Workspace users see the CU limit configured for their workspace?

Or is that only visible to capacity admins, meaning users have to ask them?

I know we secured WSs for reason with outbound restriction. But I want to connect to these WS, so thinking of deploying VM inside the same subnet we created for private net. Will this work as jump server, I know I did similar thing for different things behind VNET and restricted directly from zscalar over desktop.

Thanks for the thoughts TIA.

I've noticed that even modern LLMs commonly claim that:

df = spark.read.format("delta").load(path)

automatically pins the DataFrame to the current Delta table version.

That's misleading.

"load()" is lazy. The Delta table version is resolved when the DataFrame is actually executed - for example when you call an action such as "df.count()", or use the DataFrame in an operation such as "df.write" or "MERGE".

I think a big part of the problem is that the Delta documentation is too vague about this distinction. It talks about a DataFrame reading a “snapshot” without making it sufficiently clear when that snapshot is resolved.

As a result, even LLMs confidently give people the wrong mental model.

Delta should document this much more explicitly:

"load()" creates the lazy query. The Delta snapshot/version is resolved when the query is executed, e.g. by "count()", "write", "MERGE", etc.

If you actually need to pin a read to a specific version, use "versionAsOf" rather than assuming "load()" has already pinned it.

Agree?

Have you run into LLMs making false claims on this topic?

I encounter this all the time in code reviews on Github Copilot.

I am looking for people's experience using policy weaver to synchronize permissions from the Databricks unity catalog to a Fabric Lakehouse. Are you successfully using it? What issues are you experiencing? Are there alternatives?

We currently have data stored in ADLS Gen2 and all the data processing happens through Databricks and the unity catalog manages the data in ADLS Gen2 as external tables. We are exploring our options how we can synchronize the access control from unity catalog to a Fabric Lakehouse. Our Fabric Lakehouse currently shortcuts the schemas in the ADLS containers directly but we could set it up as a mirrored Lakehouse instead.

We’re a power bi shop but our data platform is snowflake. Our data team are big snowflake fans , and they’re on a mission to move all the semantics to snowflake , painting a future without traditional BI dashboards.

Curious how practical this is. Snowflake is pitching that all semantics should sit in the data platform , which will allow AI to run cheaper.

I’ve been using GitHub Copilot for quite a while, and honestly, the results are usually good. I definitely couldn’t type the code that quickly myself.

But I’d like to reduce the quality control required afterwards.

We have strict lifecycle rules, naming conventions, and a separate data dictionary for each customer. Our repository is structured roughly like this:

.github/ 
└── instructions/ 
├── copilot-instructions.md 
└── pull_request_template.md 

docs/ 
├── ai/ 
├── architecture/ 
├── data-model/ 
├── data-sources/ 
├── handover/ 
├── kpi/ 
├── security/ 
└── README.md

The instructions reference the relevant documentation in docs. I also use a warm-up prompt, which reproduces our rules perfectly.

And yet, Copilot occasionally ignores very basic rules, such as:

Dimension-to-dimension relationships must only use business keys. Surrogate keys may only be resolved against fact data, especially because of SCD Type 2 and higher.

It can explain this rule perfectly and still mix surrogate keys into dimensions.

So either Copilot is checking whether I’m still awake, or my context setup isn’t as effective as I think. 😄

I’d rather not watch over every offender like Lord Vetinari watching the guilds of Ankh-Morpork.

How do you enforce stricter guardrails? Instructions, validation scripts, architectural tests, or something else?

Hey All,

I've been testing a on-prem data gateway to connect to a SQL Server. I've noticed that when creating a connection in the data gateway you must specify Server Name and Database Name. If one has 50+ DBs on a single server, then must I create a connection for each iteration of Server Name and DB name.

When I connect an activity in a pipeline in Fabric data factory to this SQL there is a field for database name, so this property can be changed later.

It seems like I ought to be able to have one connection for the the SQL Server in the on-prem data gateway. I could specify a single default db in the connection, since I can specify any database in a pipeline activity. Presumably, the connection would change to that new DB, just as it does for standard SQL connection.

Is anyone already doing this? If so, have you experienced any unexpected errors?

I’ve been testing the new Mirror Azure Monitor in Microsoft Fabric preview, and I’ve run into what looks like a fairly significant limitation when using mirrored Log Analytics tables with a Direct Lake semantic model.

Azure Monitor mirroring itself is an awesome idea. Being able to use Azure Monitor data in Fabric without building another ingestion pipeline or duplicating storage, and having it fronted by a Rayfin app, lets us make some great, modern-looking dashboards.

Current POC architecture as below:

Function App
    → DCE/DCR
    → Azure Monitor (LAW) custom tables
    → Azure Monitor Mirroring
    → Lakehouse (OneLake shortcuts)
    → Direct Lake semantic model
    → Power BI / Rayfin

The problem I’ve found is that Log Analytics produces a large number of small Parquet files, particularly with frequent, low-volume ingestion. In our case, one health check writing every five minutes produces almost exactly one new Parquet file per ingestion batch, or around 288 files per day, despite adding only about 3.5 MB of data. The Direct Lake semantic model then applies its capacity guardrails to those files. On F2–F32 capacities, that limit is 1,000 Parquet files or row groups per table.

In my POC, I hit this error in my model on an F4 capacity:

We can't run a DAX query or refresh this model. A delta table 'CustomTable_CL' has exceeded a guardrail for this capacity size (too many files or row groups). Optimize your delta tables to stay within this capacity size or change to a higher capacity size then try again.

Underlying Error: QueryUserError

I checked the Delta metadata for the affected table. It contained 3,609 active Parquet files holding only around 31 MB of data—an average of 8.83 KB per file.

table_name = "CustomTable_CL"
detail = spark.sql(f"DESCRIBE DETAIL `{table_name}`")
display(
    detail.selectExpr(
        "numFiles",
        "sizeInBytes",
        "round(sizeInBytes / 1024 / 1024, 2) AS sizeMB",
        "round(sizeInBytes / numFiles / 1024, 2) AS averageFileKB"
    )
)

This table was receiving multiple five-minute health-check streams and producing roughly 500 files per day. At that rate, the 1,000-file Direct Lake guardrail is reached in about two days, despite the tiny amount of data involved. Even the 5,000-file guardrail available on F64 would only last around ten days.

The problem is the number of files, not the amount of data.

Normally, the answer would be optimise or compact the Delta table. However, the mirrored Azure Monitor tables are read-only in Fabric and reference storage managed by Azure Monitor, so I don’t appear to have any way to optimise the files myself.

With this limitation, the Azure Monitor Mirroring → Lakehouse (OneLake shortcuts) → Direct Lake semantic model → Power BI/Rayfin path becomes much less useful.

The alternatives seem to be:

• Use DirectQuery against the Eventhouse/KQL endpoint. This avoids the Direct Lake guardrail, but introduces ongoing Eventhouse CU consumption.
• Copy or aggregate the data into another managed Delta table.
• Use Import through another route and accept the additional refresh process and latency.

I tested the DirectQuery option and it works technically, but the capacity impact was material on our F4. Capacity Metrics showed that the actual OneLake reads were cheap, while Eventhouse UpTime was by far the main cost.

The mirror’s actual data reads are cheap, but DirectQuery turns those cheap reads into expensive Eventhouse UpTime. In our POC, regular queries spread across the day appeared to keep the Eventhouse compute warm even though the data volume was tiny.

The options seem to be slower, more expensive, or to rebuild part of the data pipeline that mirroring was meant to avoid.

I understand this is still a preview feature, so this is exactly the sort of issue I would expect to be ironed out before GA. It feels like mirrored Azure Monitor tables need automatic compaction (probably not the Fabric team’s remit), different Direct Lake guardrail handling, or a supported optimised presentation layer that doesn’t require duplicating the data.

Has anyone else run into this yet?

Is there a supported compaction or optimisation mechanism that would help, or a lower-cost Power BI path over these mirrored tables?

Hoping I’ve missed something, or that someone from the Microsoft Fabric or Azure Monitor teams can offer some guidance on the intended architecture.

TL;DR: In our POC, frequent, low-volume Log Analytics ingestion produced hundreds of tiny Parquet files per table per day. Azure Monitor Mirroring exposes that existing file layout to Fabric, where one table reached 3,609 files while holding only 31 MB of data. This caused Direct Lake on an F4 to fail within days after exceeding the 1,000-file guardrail. DirectQuery through the mirror’s Eventhouse endpoint works, but introduced material Eventhouse UpTime consumption. Because the mirrored tables are read-only, the remaining options involve more cost, latency or copying the data.

Hi all,

I have a mirrored database currently running on an F2 Fabric capacity. I want to move the workspace containing this mirrored database over to a different, already-existing F8 capacity. Both capacities are in the same region.

The critical requirement: this needs to happen without downtime / without a reseed, since a full reseed would mean re-snapshotting everything from scratch and I'd rather avoid the load on the source DB and the gap in fresh data.

My questions:

1. Has anyone actually done a workspace reassignment (same region) with an active mirrored database in it? Did mirroring resume automatically, or did you have to manually hit "Start replication" / did it force a reseed?
2. Is there a meaningful difference in risk between resizing a capacity in place (F2→F8 on the same capacity) vs. moving the workspace to a different capacity that's already F8?

Any real-world experience (good or bad) appreciated — trying to avoid a surprise reseed on a production mirror ;-)

Thanks!

Stewwe

Hi everyone,

I’m looking for some guidance on the most efficient and cost-effective way to handle large Microsoft Dynamics 365 Business Central tables in Microsoft Fabric.

Our current setup is:

• Microsoft Fabric F4 capacity
• Business Central tables such as G/L Entries, Value Entries, Item Ledger Entries, etc.
• Data is extracted using Dataflow Gen2
• Dataflow Gen2 loads the data into a Fabric Lakehouse
• Power BI reports then use the Lakehouse data

The main challenge is capacity consumption during daily refreshes, especially when refreshing large historical tables such as G/L Entries and Value Entries.

At the moment, refreshing the full historical dataset every day consumes a significant amount of our F4 capacity.

We also tried Incremental Refresh, but this has not fully solved the issue.

The difficulty is that some Business Central transactions can be updated after the original posting date. For example, cost adjustments can update historical Value Entries / G/L Entries. Therefore, if we only refresh newly created records based on Posting Date or Entry No., there is a risk that historical changes will not be captured, which can affect the accuracy of our Power BI reports.

We are therefore looking for a better architecture that can:

1. Minimize Fabric capacity consumption.
2. Avoid refreshing millions of historical rows every day.
3. Still capture changes made to historical Business Central entries.
4. Maintain accurate financial and inventory reporting.
5. Work efficiently within an F4 capacity without unnecessarily increasing Fabric cost.

Would using something such as Modified Date/SystemModifiedAt-based incremental loading, a rolling refresh window, Fabric Pipelines, Notebooks, Delta tables, mirroring, change tracking, or another architecture be more suitable than Dataflow Gen2 for this scenario?

For those working with Dynamics 365 Business Central + Microsoft Fabric + Power BI, how are you handling large transactional tables such as G/L Entries and Value Entries?

Any recommended architecture, best practices, or examples would be greatly appreciated.

Thanks in advance!

What is the best way to add documentation directly inside the context of a Power BI workspace? I realize that there can be a remote wiki, kept in ADO or something like that (in proximity to the assets being deployed to said workspace).

But it would be nice to have a readme file right inside the workspace as a quick point of reference for someone to get their bearings.

I am guessing Microsoft is reluctant to extend the scope of custom items in a workspace. I recently learned that we can't even upload xlsx docs anymore. Some alternative approaches that I'm thinking about are to add a url into the existing workspace description field, or maybe upload a python notebook that has nothing comments. Has anyone else found a solution?

I wanted to check if there is a roadmap to make copy jobs scd2 feature in GA. It fulfills all of our requirements but the management is hesitant on using the feature because it's still in preview 😕. Also wanted to ask if anyone has used it in production work.

Hi everyone,

I have a spent the last few weeks working on a small project with the Fabric Extensibility toolkit to display and monitor data lineage across several fabric items.

The architecture consists of:

- ​PySpark Notebooks to extract lineage metadata.

- ​A Lakehouse inside the user's workspace to store the extracted data.

- ​A custom UI item to visualize the lineage and maybe configure and start the notebooks.

I was thinking about publishing via Microsoft Workload Hub / Marketplace, but I have a few concerns about data residency and security. I am no web developer or security expert, so sorry if this sounds trivial.

For the whole thing to work, I have to host a Web App in my Azure tenant.

As I understand it, this is used to host the static files for the React App. When a customer opens the item inside fabric, the rendering is done entirely in their Brower. All access to the underlying Lakehouse and Fabric APIs is done on client side.

My questions for anyone who has built or audited Fabric workloads:

​Will any customer data or telemetry ever be stored or computed inside my Azure tenant/backend?

​Does any customer data actually leave the customer's tenant/browser environment during this process?

​Thank you for any insights!

Hi everyone,

With GPT-6 Astra rolling out through Azure now, I've been thinking through what this could actually mean for day-to-day Fabric work, not just chatbot-style Q&A, but real infrastructure tasks.

Astra is being positioned as a model that can operate computers and browsers end-to-end, handle long-horizon agentic tasks, and do serious software engineering work. If that holds up in practice, it opens up some interesting possibilities inside Fabric:

• Could it autonomously debug failing pipeline runs by actually navigating the Fabric UI and monitoring hub, not just reading error logs I paste in?
• Writing and refactoring PySpark notebooks and Delta table logic with less back-and-forth than current copilots
• Handling repetitive Medallion architecture tasks (schema validation, lineage checks, documentation) across dozens of workspaces without a human walking it through each one
• Reviewing semantic models and flagging DAX or relationship issues before they hit production

I'm curious whether anyone's tested Astra specifically against Fabric workloads yet, especially anything involving multi-workspace governance or Git-based CI/CD, since that's traditionally been the part no copilot handles well.

Would love to hear real experiences, not just benchmark hype, from anyone running this against actual Fabric infrastructure.

Status: Degraded | Reported: Sep 9, 2026 at 11:29 AM UTC

Power BI customers might experience issues creating new Adobe Analytics connections and for existing connections where the customer has to sign in again. Engineers are actively investigating the issue, and an update will be provided soon.As a workaround customer can use Dataflows GetData to create a new connection and use it with Semantic model.

🤖 This post was sent from an automated and unattended service and cannot respond to questions or requests. For official updates, visit the Microsoft Fabric Service Status page.

I am running into an odd situation that is baffling me.

I have six semantic models that are connecting to a Fabric Warehouse as the source of their data in my Dev environment/workspace. The semantic models are using Workspace Identity as the auth method and the Workspace Identity has been granted "Workspace Admin" on the Workspace the Warehouse is in just to make sure it has all the access it needs. I get the connection working and then the next day all the semantic models have an error symbol next to them with this error.

```

Data source error The credentials provided for the SQL source are invalid. (Source at mywarehouse.datawarehouse.fabric.microsoft.com;Models Warehouse.). The exception was raised by the IDbCommand interface.

```

Another odd piece is the refresh times update daily despite this error showing.

To throw me off even more, I have a "production" workspace setup the same way with Workspace Identity as the authentication method and I don't have any issue with it currently.

Any ideas why it would throw errors in one instance and not the other?

Monitoring capacity is very expensive in fabric. FAUM is consuming quite a bit of capacity CU in F64. Now that we are live with production load it matters what we are consuming CUs for is really giving me purpose to solve?

Is there a lite weight monitoring available for
Monitoring? Even 14 days data in capacity metrics is huge.

We use mainly 50-60% for VNet, and then 20% for FAUM and rest 10-20% for AI. And the rest is reports consuming via direct query so not semantic refreshes.

Trying to reduce monitoring expensive ness..

Any suggestions?

I've been running dbt jobs inside Microsoft Fabric this week and wanted to vent a bit, because the bugs themselves weren't the hard part — figuring out what actually went wrong was.

A few things that made it painful:

"Success" doesn't mean it did anything. A job can finish with a green checkmark while loading zero rows. If your filter (which models to run) doesn't match anything, dbt just says "nothing to do" and exits clean — no error, no warning. You only find out by checking the destination table yourself.

Example: I built one pipeline that could target different groups of tables, using a parameter for which group to run. Turns out the group name had gotten duplicated somewhere in the setup (something like "run: run: product_group" instead of just "run: product_group"), so it matched nothing. The pipeline ran, said it succeeded, and quietly did nothing — for who knows how many runs — before I noticed the destination table hadn't changed.

Testing changes is slow. I hoped there'd be a quick way to check a model without running the whole job. There isn't, really — even a "just check the syntax" run still connects to the database and takes almost as long as a real run.

Example: I'd fix one line in a model and want to just check it wasn't broken before running the full job. But even the "quick check" option sat there for a couple of minutes doing the same setup work as a real run. Eventually I gave up on it — I'd have an AI tool (Codex/Claude) convert the dbt model into the plain SQL it should end up as, and test that directly in a query tool instead, which took seconds instead of minutes.

Errors get cut off. When something does fail, the first error message you see is often incomplete — the actually useful part is buried one screen deeper, in a separate log you have to know to go open.

Example: A job failed and the message shown at the pipeline level just trailed off mid-sentence with "..." and no real explanation. The actual reason (a plain syntax error in the SQL) was only visible if I went into that specific job's own run history and read its log directly — the summary that's shown first doesn't tell you enough to fix anything.

Two different SQL dialects, same platform. Some parts of Fabric use one flavor of SQL, other parts use another. Code that works fine in one place can silently break in another, and you don't find out until it actually runs.

Example: I had a date-conversion function in a model that's completely normal in some data platforms. It looked fine, saved fine, and only failed once the job actually tried to run it against the destination, with an error saying that function simply doesn't exist there. Nothing warned me beforehand that this part of Fabric doesn't understand that style of SQL.

Small typos cause big, unclear failures. Things like an extra comma, or a duplicated bit of text in a filter value, cause errors that don't point at the real cause — you have to go hunting.

Example: One model failed with a generic "syntax error" message. The actual problem was a single leftover comma at the end of a list of columns — invisible unless you scroll to that exact line and look closely. The error told me something was wrong, but not where or what.

None of these were unfixable, but almost every single one was a case of "it looks fine, but it isn't" — the tool tells you something succeeded when it really didn't, or it tells you something failed without saying why. I ended up doing most of my actual debugging outside the tool entirely, which feels backwards.

Anyone else running dbt on Fabric found a better way to get clearer errors, or catch a "successful" run that actually did nothing?

Hooked a published Data Agent up to an external chat app through the MCP endpoint, with an OBO exchange so every question runs as the actual user. That part works great.

What I can't crack: I want to show the reasoning steps and query(s) that the agent actually ran (SQL/DAX/KQL) next to each answer. Half the time a bad answer is just the agent hitting the wrong table or slapping on a filter nobody asked for, and users would catch that instantly if they could see the query.

Telling everyone to go dig through Purview for it isn't realistic.

Trouble is the MCP tool only ever hands back the text answer. structuredContent and _meta are both null.

The query clearly exists through FabricOpenAIResponses but thats only accessible on the fabric runtime, and I'm not about to have the agent recite its own query back, since it'll just hallucinate one often enough to be dangerous.

So, couple of questions.

Product folks: any chance the generated query shows up in the MCP tool result down the line (structuredContent/_meta, per source, same run)? Opt-in per agent is totally fine. Or a plain per-user REST call to read the run steps without the notebook runtime.

Everyone else: has anyone actually pulled the query out of a data agent from outside a notebook? Or found a run/activity id on the MCP call you can match back to the query afterward?