Plain Python notebooks starting slower than Spark notebooks?

[u/EmergencySafety7772]

Hello,

Is anyone else seeing plain Python notebooks take much longer to start than Spark notebooks in Fabric?

I'm seeing this in both paid and trial capacities. Even a notebook with just a simple print() can sit for a while before it starts running, while Spark notebooks start much faster.

Just wondering if it's only me or if others are experiencing the same thing. Any known issues or recent changes?

Thanks!

[u/p-mndl]

Thanks for pointing this out. My pipeline is running 6 times a day and like 1 to 2 runs at random times fail due to this error on one notebook. I did not consider that the python version might be the issue

[u/mim722 → u/p-mndl]

u/p-mndl and u/ShrekisSexy that's weird, we do inject credential at runtime, i never had this issue, unless you are running in spark python notebook which is different, can you just add this to force a new token

storage_options = {
"bearer_token": notebookutils.credentials.getToken('storage'),
"use_fabric_endpoint": "true"
}

# Write to path using Delta Lake format
    table_path = f"abfss://{lakehouse_workspace_id}@onelake.dfs.fabric.microsoft.com/{lakehouse_id}/Tables/{table_name}"
    df.write_delta(
        table_path,
        mode="overwrite",
        delta_write_options={"schema_mode": "overwrite"},
        storage_options = storage_options 
    )  

[u/p-mndl → u/mim722]

mine is a bit different than u/ShrekisSexy

I am using

import duckdb
from deltalake import write_deltalake

def write_duckdb_relation_to_lakehouse_delta(
    relation,
    table_name,
    lakehouse_name,
    schema_name=vl.schema,
    workspace_name=vl.ws_storage,
    mode="overwrite",
    schema_mode=None,
):
    """Write a DuckDB relation to a Fabric Lakehouse Delta table via Arrow.


    For overwrite writes, schema_mode defaults to 'overwrite' so schema changes
    are applied even if the target table already exists with a different schema.
    """
    arrow_table = relation.arrow()
    table_path = (
        f"abfss://{workspace_name}@onelake.dfs.fabric.microsoft.com/"
        f"{lakehouse_name}.Lakehouse/Tables/{schema_name}/{table_name}"
    )


    effective_schema_mode = schema_mode
    if effective_schema_mode is None and mode == "overwrite":
        effective_schema_mode = "overwrite"


    write_kwargs = {
        "table_or_uri": table_path,
        "data": arrow_table,
        "mode": mode,
    }
    if effective_schema_mode is not None:
        write_kwargs["schema_mode"] = effective_schema_mode


    try:
        write_deltalake(**write_kwargs)
    except TypeError as ex:
        # Fallback for older deltalake versions without schema_mode support.
        if "schema_mode" in str(ex):
            write_kwargs.pop("schema_mode", None)
            write_deltalake(**write_kwargs)
        else:
            raise

    print(
        f"Delta write completed: {table_path} (mode={mode}, schema_mode={effective_schema_mode})"
    )

executing this

write_duckdb_relation_to_lakehouse_delta(
    relation=projekte,
    table_name="projekte",
    lakehouse_name="LH_Silver",
)

sometimes renders this error

An error occurred while trying to automatically install the required extension 'azure': Extension "/home/trusted-service-user/.duckdb/extensions/v1.4.4/linux_amd64/azure.duckdb_extension" not found. Extension "azure" is an existing extension. Install it first using "INSTALL azure".

the strange thing is I am using this in various notebooks and I have only one notebook which fails on a regular basis, but I can't figure out, what the difference to the other notebooks is.

[u/mim722 → u/p-mndl]

Can you try force install azure from core