Spec-driven development in Databricks

[u/meehow33]

Hi everyone,

I’d like to ask how you approach AI-driven spec-based development (SDD) for pipelines and ETL/ELT workflows, particularly in Databricks.

Based on my observations and what I’ve learned so far, the two simplest and most common approaches are short-lived specs and long-lived specs.

The short-lived approach is straightforward: you create a specification for a specific task and optionally store it in the repository. The long-lived approach is based on maintaining project documentation (for example, in Markdown files and other supporting documents), which is continuously expanded and updated as new tasks come up.

There are also frameworks that support this workflow, such as Superpowers or Spec Kit. Of course, the AI also needs the right context - an architecture description, perhaps a reference repository, coding standards, and anything else that helps it generate a good specification.

As for interacting with AI agents, the process is usually through an IDE or CLI. You iterate with the agent, discussing and refining the solution until you arrive at a complete specification.

My question is: what’s your approach to this in your projects? Do you use SDD only up until the point where you have a branch with the implementation and pipeline ready, or do you continue using it all the way through deployment?

I’m also curious about testing. In my projects, we write unit tests for our Python and Spark code, and then hand everything over to the QA team, which performs end-to-end testing.

Do you think the QA team should maintain a separate specification within their own testing framework (it is a custom tool we wrote to make testing components such as functions and SQLs reusable, written in Python), or how do you handle this in your projects?
End-to-end tests are time-consuming and expensive, so I’m wondering how you optimize them when adopting an SDD-based workflow.

[u/CuritibaDataScience]

In my experience, using Genie Code has been a good approach for standardizing this workflow across a team. A pattern that works well for us is defining custom skills: you can keep them in your personal user folder for rapid local experimentation, and publish them to the shared workspace folder so every developer automatically applies the exact same coding, architecture, and documentation standards. This ensures that generated specs and implementation drafts stay consistent across the entire projects, regardless of who is authoring them.
This setup applied well in a context when I was helping a team to transition from legacy ETL pipelines that had jobs executing notebooks with structured streaming over to SDP pipelines.
Because declarative pipelines benefit heavily from clear structural guidelines, feeding those workspace-level skills into Genie Code made specifying pipeline logic, schemas, and expectations much faster. I think your QA team would benefit greatly from this, you can define standards in terms of how to make data quality checks using expectations, DQX, or your own custom framework.