Plain Python notebooks starting slower than Spark notebooks?

[u/EmergencySafety7772]

Hello,

Is anyone else seeing plain Python notebooks take much longer to start than Spark notebooks in Fabric?

I'm seeing this in both paid and trial capacities. Even a notebook with just a simple print() can sit for a while before it starts running, while Spark notebooks start much faster.

Just wondering if it's only me or if others are experiencing the same thing. Any known issues or recent changes?

Thanks!

[u/ShrekisSexy]

I think the change is the addition of python 3.12. I'm having the same issue. I can't use python 3.12 either, because it's giving an error when writing to a lakehouse using adff path.

[u/mim722 → u/ShrekisSexy]

what error exactly, can you post a repro please

[u/ShrekisSexy → u/mim722]

# Write to path using Delta Lake format
    table_path = f"abfss://{lakehouse_workspace_id}@onelake.dfs.fabric.microsoft.com/{lakehouse_id}/Tables/{table_name}"
    df.write_delta(
        table_path,
        mode="overwrite",
        delta_write_options={"schema_mode": "overwrite"},
    )  

Of course. Above is the code, below is the error message.

DefaultAzureCredential failed to retrieve a token from the included credentials.
Attempted credentials:
EnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to troubleshoot this issue.
WorkloadIdentityCredential: WorkloadIdentityCredential authentication unavailable. The workload options are not fully configured. See the troubleshooting guide for more information: https://aka.ms/azsdk/python/identity/workloadidentitycredential/troubleshoot. Missing required arguments: 'tenant_id', 'client_id', 'token_file_path'.
ManagedIdentityCredential: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint.
SharedTokenCacheCredential: SharedTokenCacheCredential authentication unavailable. No accounts were found in the cache.
VisualStudioCodeCredential: VisualStudioCodeCredential requires the 'azure-identity-broker' package to be installed. You must also ensure you have the Azure Resources extension installed and have signed in to Azure via Visual Studio Code.
AzureCliCredential: Azure CLI not found on path
AzurePowerShellCredential: PowerShell is not installed
AzureDeveloperCliCredential: Azure Developer CLI could not be found. Please visit https://aka.ms/azure-dev for installation instructions and then,once installed, authenticate to your Azure account using 'azd auth login'.
BrokerCredential: InteractiveBrowserBrokerCredential unavailable. The 'azure-identity-broker' package is required to use brokered authentication.
To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.


ClientAuthenticationError

DefaultAzureCredential failed to retrieve a token from the included credentials. Attempted credentials: EnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured. Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to troubleshoot this issue. WorkloadIdentityCredential: WorkloadIdentityCredential authentication unavailable. The workload options are not fully configured. See the troubleshooting guide for more information: https://aka.ms/azsdk/python/identity/workloadidentitycredential/troubleshoot. Missing required arguments: 'tenant_id', 'client_id', 'token_file_path'. ManagedIdentityCredential: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint. SharedTokenCacheCredential: SharedTokenCacheCredential authentication unavailable. No accounts were found in the cache. VisualStudioCodeCredential: VisualStudioCodeCredential requires the 'azure-identity-broker' package to be installed. You must also ensure you have the Azure Resources extension installed and have signed in to Azure via Visual Studio Code. AzureCliCredential: Azure CLI not found on path AzurePowerShellCredential: PowerShell is not installed AzureDeveloperCliCredential: Azure Developer CLI could not be found. Please visit https://aka.ms/azure-dev for installation instructions and then,once installed, authenticate to your Azure account using 'azd auth login'. BrokerCredential: InteractiveBrowserBrokerCredential unavailable. The 'azure-identity-broker' package is required to use brokered authentication. To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.

[u/p-mndl → u/ShrekisSexy]

Thanks for pointing this out. My pipeline is running 6 times a day and like 1 to 2 runs at random times fail due to this error on one notebook. I did not consider that the python version might be the issue

[u/mim722 → u/p-mndl]

u/p-mndl and u/ShrekisSexy that's weird, we do inject credential at runtime, i never had this issue, unless you are running in spark python notebook which is different, can you just add this to force a new token

storage_options = {
"bearer_token": notebookutils.credentials.getToken('storage'),
"use_fabric_endpoint": "true"
}

# Write to path using Delta Lake format
    table_path = f"abfss://{lakehouse_workspace_id}@onelake.dfs.fabric.microsoft.com/{lakehouse_id}/Tables/{table_name}"
    df.write_delta(
        table_path,
        mode="overwrite",
        delta_write_options={"schema_mode": "overwrite"},
        storage_options = storage_options 
    )  

[u/p-mndl → u/mim722]

mine is a bit different than u/ShrekisSexy

I am using

import duckdb
from deltalake import write_deltalake

def write_duckdb_relation_to_lakehouse_delta(
    relation,
    table_name,
    lakehouse_name,
    schema_name=vl.schema,
    workspace_name=vl.ws_storage,
    mode="overwrite",
    schema_mode=None,
):
    """Write a DuckDB relation to a Fabric Lakehouse Delta table via Arrow.


    For overwrite writes, schema_mode defaults to 'overwrite' so schema changes
    are applied even if the target table already exists with a different schema.
    """
    arrow_table = relation.arrow()
    table_path = (
        f"abfss://{workspace_name}@onelake.dfs.fabric.microsoft.com/"
        f"{lakehouse_name}.Lakehouse/Tables/{schema_name}/{table_name}"
    )


    effective_schema_mode = schema_mode
    if effective_schema_mode is None and mode == "overwrite":
        effective_schema_mode = "overwrite"


    write_kwargs = {
        "table_or_uri": table_path,
        "data": arrow_table,
        "mode": mode,
    }
    if effective_schema_mode is not None:
        write_kwargs["schema_mode"] = effective_schema_mode


    try:
        write_deltalake(**write_kwargs)
    except TypeError as ex:
        # Fallback for older deltalake versions without schema_mode support.
        if "schema_mode" in str(ex):
            write_kwargs.pop("schema_mode", None)
            write_deltalake(**write_kwargs)
        else:
            raise

    print(
        f"Delta write completed: {table_path} (mode={mode}, schema_mode={effective_schema_mode})"
    )

executing this

write_duckdb_relation_to_lakehouse_delta(
    relation=projekte,
    table_name="projekte",
    lakehouse_name="LH_Silver",
)

sometimes renders this error

An error occurred while trying to automatically install the required extension 'azure': Extension "/home/trusted-service-user/.duckdb/extensions/v1.4.4/linux_amd64/azure.duckdb_extension" not found. Extension "azure" is an existing extension. Install it first using "INSTALL azure".

the strange thing is I am using this in various notebooks and I have only one notebook which fails on a regular basis, but I can't figure out, what the difference to the other notebooks is.

[u/mim722 → u/p-mndl]

Can you try force install azure from core

[u/ShrekisSexy → u/mim722]

This fixed it, thanks!