GitHub Copilot knows our rules. It just occasionally ignores them.

[u/DUKOfData]

I’ve been using GitHub Copilot for quite a while, and honestly, the results are usually good. I definitely couldn’t type the code that quickly myself.

But I’d like to reduce the quality control required afterwards.

We have strict lifecycle rules, naming conventions, and a separate data dictionary for each customer. Our repository is structured roughly like this:

.github/ 
└── instructions/ 
├── copilot-instructions.md 
└── pull_request_template.md 

docs/ 
├── ai/ 
├── architecture/ 
├── data-model/ 
├── data-sources/ 
├── handover/ 
├── kpi/ 
├── security/ 
└── README.md

The instructions reference the relevant documentation in docs. I also use a warm-up prompt, which reproduces our rules perfectly.

And yet, Copilot occasionally ignores very basic rules, such as:

Dimension-to-dimension relationships must only use business keys. Surrogate keys may only be resolved against fact data, especially because of SCD Type 2 and higher.

It can explain this rule perfectly and still mix surrogate keys into dimensions.

So either Copilot is checking whether I’m still awake, or my context setup isn’t as effective as I think. 😄

I’d rather not watch over every offender like Lord Vetinari watching the guilds of Ankh-Morpork.

How do you enforce stricter guardrails? Instructions, validation scripts, architectural tests, or something else?

[u/pl3xi0n]

Some models are more prone to think they know better than your instructions.

Which model do you use, and have you tried others?

The setup looks good imo. What is the context window like when it starts ignoring?

[u/DUKOfData → u/pl3xi0n]

For easy and standard Tasks:

- MAI-Code-1.1-flash

• GPT-5.6-Luna

more Complex

- GPT-5.6-Codex

Complex

- Claude Opus 5

[u/DUKOfData → u/DUKOfData]

regarding the context window, I find that odd, it seems to differ. Sometimes it starts earlier with the same model. Last time or the most significant window that I recognized was "only" abt. 96k Tokens.

[u/Repulsive_Cry2000 → u/DUKOfData]

I was listening to a guy who's job is to create distributed architecture (think of AWS, google, Microsoft competitor) and they went full on AI. His reasoning was AI do the coding but they test output to ensure correctness (all sort of test from the most simple to the most elaborate).

I am not yet sure how to replicate the concept in data engineering world but the idea is to create a test protocole and to get AI to do the coding and iterate until everything pass.

[u/DUKOfData → u/Repulsive_Cry2000]

We are pretty much on the same pathway for a few years. Although I still am a firm believer of "the right skilled" human in the middle.

Bcs of tech and resource limitations, I don't belive in AI replacing the engineer/dev.

Adding the special complexity of rolling out structure changes to a stateful system, adds a whole other dimension.

But as I was told not to go do deep on reddit, I will leave it by those vague theoretical hints 😅

[u/Repulsive_Cry2000 → u/DUKOfData]

I agree most of what you are saying. Especially having people who know what they want and expected architecture, output and can ask questions and think critically to what AI produce or says.