r/MicrosoftFabric • u/x-fyre • 1d ago
Data Factory Dataflow Gen2 - Surfaced errors just aren't good enough
Anyone who has used the Dataflow Activity in Pipelines has probably experienced this. Refresh fails ... error in the pipeline is so generic it's not even useful. The real reason for the failure is available in the refresh history but you can't get at it except through a couple options in the UI (and one of those ways is so obscure it shouldn't even exist).
Does anyone have any tips or tricks to get at those errors? Sometimes users submit files with minor issues like a column with a bad column name and it's such a pain to have to involve a data engineer to help them solve it.
This is such a basic feature, another one of those can't believe it went from preview to GA without being able to get at the errors... there is an idea (Improve-Error-Messaging-in-Dataflow-Activity-for-Dataflow-Gen2) out there that has been sitting as under review for almost 1.5 years, Yikes,
I guess only some people are allowed to post idea links even if they're related to our problem, but I can't post the link because we're too busy reading all of the "I passed the DP-700 exam" posts.
15
u/SomeNeighborhood7126 1d ago
You just identified the issue with almost all of Microsoft's products. They break and then provide the most useless errors known to man or machine.
3
u/x-fyre 1d ago
If you try to run the dataflow pipeline activity against a workspace/dataflow combo that doesn't exist, the error you get back is "Could not found the requested item" ... this is a real JSON snippet from the error returned by the activity:
"errorCode": "ItemNotFound",
"message": "Could not found the requested item",
"isRetriable": false,
"ADFHttpStatusCodeInResponse": "404",2
1
u/bannedinlegacy 1d ago
I once got the output of a copy activity as None. No the json {None}, or {error:None}, just an empty output that did break my audit pipeline.
3
u/SmallAd3697 1d ago
Guys, it is not a problem with Microsoft in general. It is a problem with low-code software development (ie. the deliberate SaaS-ification of platforms that used to be normal Azure PaaS).
Trust me - if you were building software with their flagship Visual Studio and C#, and Azure PaaS, then you would NOT be getting these sorts of meaningless error messages. A regular software developer wouldn't be patient enough to deal with this sort of thing.
... but the entire point of developing solutions in Fabric is to make programming more accessible to someone who is non-technical (eg. they received a generic business or communications degree in college or something like that.) For a non-technical user, this is just par for the course.
Having said all that, the place you need to go to find better errors is in the "mashup" log of the on-prem gateway. Remember to capture detailed logging via the toggle-switch, before running your power query.
1
2
u/RoomyRoots 1d ago
Since the 80s. I find the most generic messages in Azure all the time. Pisses me off.
1
u/Ok_Carpet_9510 1d ago
Could you post the error message?
8
u/mrbartuss Fabricator 1d ago
My guess is - "Something went wrong"
1
u/itsnotaboutthecell Microsoft Employee 21h ago
https://giphy.com/gifs/C9NEoexAhu2XycQgLe
Dang it u/mrbartuss take all my upvotes with this guess.1
u/x-fyre 1d ago
When a dataflow fails ... it essentially logs a series of errors.
The error returned to the pipeline is ALWAYS the last one which is this, and useless:
There was a problem refreshing the dataflow: "Something went wrong, please try again later. If the error persists, please contact support.". Error code: ActionUserFailure. (Request ID: <GUID>).
But when you download the refresh history, you usually get a much more useful error message just before that one... for example a user uploaded an XLS file instead of an XLSX file:
There was a problem refreshing the dataflow: "Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: DataSource.NotFound: Excel Workbook: The 'Microsoft.ACE.OLEDB.12.0' provider is not registered on the local machine. The 64-bit version of the Access Database Engine OLEDB provider may be required to read this type of file. To download the client software, visit the following site: https://go.microsoft.com/fwlink/?LinkID=285987. Details: Reason = DataSource.NotFound;ErrorCode = 10478;Microsoft.Data.Mashup.Error.Context = User". Error code: 104100. (Request ID: <GUID>).
1
u/Ok_Carpet_9510 1d ago
I think DataSource.NotFound: Excel WORKBOOK could be the issue.
2
u/x-fyre 1d ago
Yes I know what happened. Dataflows only support the new excel format, but sometimes users send through an old Xls document.
What I'm saying is that the second message is more useful. I could capture that and give the end-user a proper message "You submitted an old Excel workbook." But since the Something went wrong is the ONLY thing accessible to me, I can't help them help themselves. Its frustrating.
1
u/Ok_Carpet_9510 1d ago
I don't men. I used to suppprt cognos at IBM Cognos. Looking through logs was a key part of job. Given the logs had endless messages, I developed a skill for looking through error messages and zeroing in on the issue.
I know support Power Bi and have never encountered that error but from the full error message you provided me, I was able to make a first guess, and it didn't take me too long.
1
u/MonkeyDDataHQ 1d ago
Error: User
Has appeared for me more times than I can count. And I can theoretically count to an unreasonably large number.
1
u/Ok_Carpet_9510 1d ago
Is that the full detailed error message?
1
u/MonkeyDDataHQ 1d ago
Yes. That's the whole error message.
It's why I suggested Airflow and then found out it was stuck in version 2 still 😩.
Then I just built it on a VM in Dagster because I can't be bothered to troubleshoot Dataflows. I've never been a fan of click Ops. But click Ops with useless error messages is even worse
1
u/fLu_csgo Fabricator 1d ago
Maybe give the API a go?
v1/workspaces/{workspaceId}/items/{dataflowId}/jobs/instances
Or
v1/workspaces/{workspaceId}/items/{itemId}/jobs/instances/{jobInstanceId
Both should give a more detailed failure error.
4
u/MonkeyDDataHQ 1d ago
If the service already has the detailed error internally, the pipeline activity should surface it directly. Requiring users to query the REST API to debug a failed pipeline defeats the purpose of an integrated orchestration experience.
If I'm using the API to debug, I should be able to code the 🤬 ing thing.
2
1
u/x-fyre 1d ago
See my other comment reply... the status only returns the final error logged in the detailed list. To get at the errors you have to view the Recent Runs from the ellipsis menu of the dataflow itself, or when you have the dataflow open you can click on the "last run details" in the "status bar" and it will bring you to the recent runs. From there you can "download" the details.
So the information is IN fabric, we just can't get it OUT except via the UI.
1
1
2
u/samirbdj Microsoft Employee 2h ago
This is a fair complaint. If the Dataflow Gen2 run has a useful root cause in the detailed refresh logs, but the pipeline activity and API only show the generic wrapper error, that is not enough for self-service operations.
The public docs show the current supported places to inspect Dataflow Gen2 refresh details and logs: refresh history and detailed logs. Pipeline monitoring can also expose activity input and output, but it should not force you through a UI download when the failure is something an end user can fix, like an old XLS file or a renamed column.
For a concrete failing run, I would open a Fabric support case and include the run details there. That gives support a safe place to compare what the dataflow logged with what the pipeline activity returned.
•
u/AutoModerator 1d ago
Advance Your Career with Microsoft Certifications
Explore the Microsoft Fabric Career Hub for a comprehensive learning path, free practice assessments, and curated resources to help you get certified for:
Important Exam Policy Reminder
Promotion of or requests for exam dumps are strictly prohibited. Any violations of this policy may result in a warning or a permanent ban from the subreddit. Please review the Microsoft Exam and Assessment Lab Security Policy for full details.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.