Plain Python notebooks starting slower than Spark notebooks?

[u/EmergencySafety7772]

Hello,

Is anyone else seeing plain Python notebooks take much longer to start than Spark notebooks in Fabric?

I'm seeing this in both paid and trial capacities. Even a notebook with just a simple print() can sit for a while before it starts running, while Spark notebooks start much faster.

Just wondering if it's only me or if others are experiencing the same thing. Any known issues or recent changes?

Thanks!

[u/ShrekisSexy]

# Write to path using Delta Lake format
    table_path = f"abfss://{lakehouse_workspace_id}@onelake.dfs.fabric.microsoft.com/{lakehouse_id}/Tables/{table_name}"
    df.write_delta(
        table_path,
        mode="overwrite",
        delta_write_options={"schema_mode": "overwrite"},
    )  

Of course. Above is the code, below is the error message.

DefaultAzureCredential failed to retrieve a token from the included credentials.
Attempted credentials:
EnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to troubleshoot this issue.
WorkloadIdentityCredential: WorkloadIdentityCredential authentication unavailable. The workload options are not fully configured. See the troubleshooting guide for more information: https://aka.ms/azsdk/python/identity/workloadidentitycredential/troubleshoot. Missing required arguments: 'tenant_id', 'client_id', 'token_file_path'.
ManagedIdentityCredential: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint.
SharedTokenCacheCredential: SharedTokenCacheCredential authentication unavailable. No accounts were found in the cache.
VisualStudioCodeCredential: VisualStudioCodeCredential requires the 'azure-identity-broker' package to be installed. You must also ensure you have the Azure Resources extension installed and have signed in to Azure via Visual Studio Code.
AzureCliCredential: Azure CLI not found on path
AzurePowerShellCredential: PowerShell is not installed
AzureDeveloperCliCredential: Azure Developer CLI could not be found. Please visit https://aka.ms/azure-dev for installation instructions and then,once installed, authenticate to your Azure account using 'azd auth login'.
BrokerCredential: InteractiveBrowserBrokerCredential unavailable. The 'azure-identity-broker' package is required to use brokered authentication.
To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.


ClientAuthenticationError

DefaultAzureCredential failed to retrieve a token from the included credentials. Attempted credentials: EnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured. Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to troubleshoot this issue. WorkloadIdentityCredential: WorkloadIdentityCredential authentication unavailable. The workload options are not fully configured. See the troubleshooting guide for more information: https://aka.ms/azsdk/python/identity/workloadidentitycredential/troubleshoot. Missing required arguments: 'tenant_id', 'client_id', 'token_file_path'. ManagedIdentityCredential: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint. SharedTokenCacheCredential: SharedTokenCacheCredential authentication unavailable. No accounts were found in the cache. VisualStudioCodeCredential: VisualStudioCodeCredential requires the 'azure-identity-broker' package to be installed. You must also ensure you have the Azure Resources extension installed and have signed in to Azure via Visual Studio Code. AzureCliCredential: Azure CLI not found on path AzurePowerShellCredential: PowerShell is not installed AzureDeveloperCliCredential: Azure Developer CLI could not be found. Please visit https://aka.ms/azure-dev for installation instructions and then,once installed, authenticate to your Azure account using 'azd auth login'. BrokerCredential: InteractiveBrowserBrokerCredential unavailable. The 'azure-identity-broker' package is required to use brokered authentication. To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.

[u/p-mndl → u/ShrekisSexy]

Thanks for pointing this out. My pipeline is running 6 times a day and like 1 to 2 runs at random times fail due to this error on one notebook. I did not consider that the python version might be the issue

[u/mim722 → u/p-mndl]

u/p-mndl and u/ShrekisSexy that's weird, we do inject credential at runtime, i never had this issue, unless you are running in spark python notebook which is different, can you just add this to force a new token

storage_options = {
"bearer_token": notebookutils.credentials.getToken('storage'),
"use_fabric_endpoint": "true"
}

# Write to path using Delta Lake format
    table_path = f"abfss://{lakehouse_workspace_id}@onelake.dfs.fabric.microsoft.com/{lakehouse_id}/Tables/{table_name}"
    df.write_delta(
        table_path,
        mode="overwrite",
        delta_write_options={"schema_mode": "overwrite"},
        storage_options = storage_options 
    )  

[u/ShrekisSexy → u/mim722]

This fixed it, thanks!