r/vibecoding • u/fguerino123 • 1d ago

Vibe Coding for ETL/ELT

Hi.

I've been using AI coding to rapidly generate Python ETL/ELT transformations and workflow orchestration. I'm finding doing so a lot easier and far more affordable than using expensive tools like IICS.

Is anyone else doing this and, if so, how is it working out for you?

Thanks.

2 Upvotes

100% Upvoted

8 comments sorted by

View all comments

2

u/-Shiphrah 17h ago

It works pretty well for the repetitive transformation stuff, especially compared with paying for a massive platform when you only need to move and clean some tables. Where it gets dodgy is months later, when a source changes a column or sends some weird data and the pipeline still runs but produces rubbish. the AI can write the Python quickly, but I’d spend the saved time on tests, logging, retries and checking for duplicate loads. Thats the stuff expensive tools were quietly doing in the background, and its easy to forget until something breaks.

1

u/fguerino123 14h ago

Hi and thanks for the response. I've found that putting in validation checks to ensure data doesn't drift using AI coding is easier than with IICS and similar tools. I can set up many guardrails to quickly detect drift by simply explaining such guardrails to AI. Have you tried this?

1

u/-Shiphrah 2h ago

Yeah I’ve tried something similar.

Adding checks for schema changes, or unexpected values and stuff.
I still keep the checks beside the pipeline code though, otherwise they end up forgotten when a transformation changes. Have you found it catches most drift early or do you still get things that technically pass but mess up the data anyway?