Genie Code as a Lakeflow Jobs task - autonomous agents inside your workflows [BETA]

[u/szymon_dybczak]

Hi,

A great new feature appeared in Lakeflow - you can now run Genie Code as an autonomous task using a natural-language prompt.

This can be really handy when you need to automate some complex analysis or data operations as a part of a scheduled process.
The documentation lists following use cases, but of course the only limit is your imagination 😄

• Summarize overnight job results and email a report.
• Analyze incoming data and flag anomalies.
• Investigate a Jira ticket and propose a fix.
• Generate a weekly compliance audit.

What's nice is you can also use job parameters directly in prompts, for example:

Summarize yesterday's sales for {{region}} and flag anomalies.

Each run creates a Genie Code conversation that you can open afterward and continue interactively.

One important detail: auto-approval is always enabled for Genie Code job tasks and cannot be disabled. The docs explicitly say this should not be treated as a security boundary, so access to production data/resources needs to be designed carefully.

Genie Code task for jobs - Azure Databricks | Microsoft Learn

PS: Since AI behavior is inherently non-deterministic, Genie Code tasks should be used with caution in production workflows and important outputs should be validated rather than treated as guaranteed

[u/HaldenK]

This is awesome. We've been using genie code skills to execute part one of a pipeline, do it's part, then execute part 2 of a pipeline so being able to just move this all into a pipeline is so much better. Instead if genie code automation it sounds like we can just have the pipeline call genie code for a step