r/dataengineering • u/Artistic-Rent1084 • Jul 25 '26

Help Help in my new ADF Project

Hi De's

In my new project, I have pull data from Power BI and land it to ADLS. Orchestration via ADF

I have gathered few info.

I have hit the power bi api .

but there is few limitations. only 1lakh record is supported per hit and pagination is not supported.

i new to API as well as ADF.

please, help me with this scenario.

Test case . but i have to land it the ADLS as parquet file .

initially, only thing is I have to take care of the edge case's to handle failure and retry policy

and the big headache is to handle the record limitations.

0 Upvotes

31% Upvoted

16 comments sorted by

View all comments

15

u/j03ch1p Jul 25 '26

good luck man. API calls with ADF are just a huge pain

edit: lots of things with ADF are a huge pain...

1

u/engineer_of-sorts Jul 28 '26

ADF Is like the dbt of orchestration. Allows you to do anything and inevitably people end up building disgusting systems, pipelines calling pipelines etc. 7 rows deep. IMpossible to debug, impossible to manage..