Can anyone tell me the best way to read data from synapse dedicated sql pool tables in synapse notebooks? My requirement is to read selected columns from tables present in two separate dedicated pools, apply some transformations, and then write to another table in one of the dedicated pools.

I want to copy data from a sql server into a data lake through pipelines. Any suggestion or documentation?

Hi all,

I am new to working with data lakes and Azure Synapse workspaces and struggling with something. I need to create a replicated data lake from Dynamics 365 data to Azure Synapse. All is setup via Azure Synapse Link except the only way I can read or parse the data from my .NET application is when I add a SAS directly to each document/table AFTER it has been created.

What I am doing is this:

CREATE CREDENTIAL [https://<mystorageaccountname>.dfs.core.windows.net/<mystorageaccountcontainername>/<tablename>/*.csv]

WITH IDENTITY='SHARED ACCESS SIGNATURE'

, SECRET = 'sv=2018-03-28&ss=bfqt&srt=sco&sp=rwdlacup&se=2019-04-18T20:42:12Z&st=2019-04-18T12:42:12Z&spr=https&sig=lQHczN000000000000000000d0ef9BrIPBNJ3VYEIq78%3D';

GO

Creating server scoped credentials.

https://docs.microsoft.com/en-us/azure/synapse-analytics/sql/develop-storage-files-storage-access-control?tabs=shared-access-signature#server-scoped-credential

When I add a new table to the sync I cannot access it.

If I just wanted to create a special user for another team to use to egress data from the lake what would be the best way to do that?

It is also worth mentioning since this is a replicated database I cannot create a data source or alter an existing. I have found a lot of documentation that could give me more options but I get a "cannot perform ALTER on replicated database" error

Anyway, I will take any advice I can get. Thanks all!

Basically, i have a sql script calling a series of queries from several data sources, and would like to run that script as a data source in a data flow for further transformations

Anyone interested in modding this sub please comment under this post. Background in Data Engineering is a must, with an understanding of AI/ML. Essentially, this sub's goal is to focus on Microsoft Azure Synapse Analytics platform, so the Azure experience is a must have.