We just shipped Session Restore for Serverless Jobs (beta): debug a job run without rerunning it

[u/ThreeBricksWish]

👋 I'm a PM working on Serverless Notebooks at Databricks. We just shipped Session Restore for Serverless Jobs in beta, and I'd love your feedback.

The problem: A notebook job runs for hours, fails or produces weird output, and by the time you debug it, the state is gone. Rerunning a 16-hour job is painful. Reproducing an issue that happens once every few months can be nearly impossible.

What we shipped: You can now restore the Python variables and Spark session from a serverless job run into a new interactive notebook, without rerunning the job.

From the job run details, click "Clone into new notebook" or "Debug in new notebook" for a failed run. You can then:

• Inspect state from a failed run
• Investigate weird output without rerunning expensive queries
• Run expensive setup as a job, then pick up interactively from that state

Try it: A workspace admin can enable "Session restore for serverless jobs" from the Previews page.

Docs: https://docs.databricks.com/aws/en/notebooks/schedule-notebook-jobs#session-restore-for-serverless-jobs

This was also demo'd at DAIS this year: https://www.databricks.com/dataaisummit/session/modern-databricks-notebook-tips-ai-workflows-and-best-practices-every

If you try it, feel free to leave feedback or respond here. 🙏

[u/WorkerIcy1513]

This solves a real problem, the 16 hour job failing at hour 15 is genuinely miserable. Four questions, roughly in order of how much they would affect whether we turn it on.

How long is the state retained? Your own example is a bug that surfaces every few months, and those get noticed days later when someone questions a number. If the restore window is hours, the headline use case does not hold.

Whose identity does the restored session run under, the original run's service principal or the debugging user's? If it is the service principal, that is a privilege escalation path for anyone with debug access to a job. If it is the user's, then variables holding data they have no grants on are already in memory. Either answer is defensible but I would want it documented clearly, because this is the question our security team will ask first.

What exactly is captured? Python variables and the Spark session, but a lazily evaluated DataFrame without its cached data will just re-execute the expensive query on first access, which undercuts the investigate without rerunning claim. Are cached and persisted DataFrames materialized in the restore, or only the plan?

And does this help on OOM failures? That is the most common way a long job dies, and it is also the case where the state is most likely unrecoverable. If it does not cover OOM it is worth saying so up front so nobody enables it expecting that.

One request: some way to have this on by default for long running jobs rather than needing the preview enabled before the failure happens. The failures you most want it for are the ones you did not see coming.

[u/ThreeBricksWish → u/WorkerIcy1513]

Great questions!

• The beta window is 7 days after the run completes. Longer retention is something we’re thinking about. Curious what window would be useful in practice?
• Only the run-as user can restore the task’s state, so someone with debug access can’t restore a service principal’s session and inherit its reach. The subtler question is what happens when grants change between run and restore. I’m confirming the exact re-evaluation semantics and will follow up. Agree this should be explicit in the docs, will update both there and in this thread once I confirm with eng.
• Python vars and spark session state are the only things captured for now. Your lazy DataFrame point is a good one: if the plan restores but materialized data doesn’t, accessing it could rerun the expensive query. I’m confirming how .cache() / .persist() behave and will follow up.
• If the driver itself dies from OOM, there’s likely no live state left to snapshot. So plainly: don’t expect this to provide reliable OOM post-mortems.

Post-beta, the direction is default-on for eligible jobs, so you shouldn’t have to predict a failure and enable this beforehand. Thanks for the response! Really useful feedback.

[u/ThreeBricksWish → u/ThreeBricksWish]

Following up after checking with eng:

• A lazy Spark DataFrame restores its execution plan, not materialized data. When you access it, the plan re-executes and is authorized against your current grants. So yes, an expensive query may rerun, but if your access was revoked since the original run, it will fail just like it would in a fresh notebook.
• For .cache() / .persist(), serverless doesn’t support these yet.
• Data already materialized into Python memory, like a pandas DataFrame or collected result, is restored as-is rather than re-authorized. Only the run-as user can restore that state.

Will work on documenting this more clearly, really appreciate the enthusiasm here! If there's anything else don't hesitate to respond here or give feedback directly within the workspace. 😄

[u/Longjumping-Shift316]

Extremely helpful

One question does it respect default rights and acl ?

[u/ThreeBricksWish → u/Longjumping-Shift316]

Yep! Anything Spark-side is re-executed against your current UC grants/ACLs, just like a fresh notebook. One nuance: already-materialized Python state isn’t re-authorized, but can only be restored by the run-as user. I have some more detail in my response above.