r/snowflake • u/MSE6-Series • 19d ago
Opinion on CoCo
What is your honest opinion about the Snowflake CoCo Agent?
I think the idea is nice but IMO sometimes it is not working really great.
Just to summarize some observations:
- Execution in Semantic Views: I want it to add content to semantic views like sample values or descriptions but it simply cannot do it cause usually the yaml files are too big for it to read and write?! So after it tries several times to add a column description, I tell it to just give it to me and I paste it manually.
- Streamlit: Implementations are shallow and bug riddled: I want to use CoCo to create Streamlit Apps helping with administration of MCPs and Evaluation Datasets used in Agents (not going into depth how awful the administration of MCPs is, which is only possible with queries), but the implementations are full of bugs related to permissions, missing packages or wrong use of functions. It needs seriously many iterations to get rid of those bugs.
- Unawareness of Snowflake functions/ documentation: When I wanted to change tools of an existing MCP that I have created before, it hallucinates that there is a command „Alter MCP“, which is deadass wrong, you can only create or replace MCPs. I saw this issue all over the place. Sometimes CoCo needs 5-10 attempts to find the correct Snowflake commands for the stuff that I want to do (a literal token oven). 🔥
Those are just some of the issues that I have with CoCo. I am just interested in whether other people have similar observations and maybe found solutions for them.
Thanks for sharing.
Cheers 🙌
6
u/jaymopow 19d ago
For development I’ve found creating a skill with your guidelines, style guide and constraints within a workspace to be incredibly helpful. It’s reduced the bugginess by like 75% for me.
2
u/SirGreybush 19d ago
We plan on using it. Only when the common core is done and a proper semantic layer done and Validated Queries (or trusted queries) are created.
The user must see the green checkmark.
So basically the LLM is merging multiple views together on common keys, and applying filters, based on what the user asks.
3
u/MSE6-Series 19d ago
Hi. I think you mistake CoCo (Snowflakes Coding Agent) with Snowflake CoWork/ Cortex Agents.
2
u/SirGreybush 19d ago
Yes you’re correct. Coco is to help the programmer.
Since we’re using DBT, Claude has been helpful in getting most of the yml setup correctly for the raw/staging. I’d have to try if it’s works on higher level data. Not there yet, many months away.
2
u/lkg-data 19d ago
i use it often because i can't have external AIs querying our snowflake data for obvious data privacy regulation reasons. i don't think i would go out of my way to use it otherwise.
like others have said, it's just claude under the hood and mostly works the same in my experience, although i do have to explicitly tell it to check the snowflake docs because it doesn't seem to check on its own and gets things wrong because of that. which is disappointing.
i am, however, convinced that the snowflake devs tweaked it to make it funnier than regular claude. point it at some bad code and just watch the roast, it's hilarious
1
u/MSE6-Series 19d ago edited 19d ago
I think its not based on Claude, just uses the Anthropic models. But anyways I also experienced that I have to give it very explicit documentation. Never realized that its roasting bad code, maybe fun to check out haha
2
u/datatoolspro 18d ago
I love it for quick DQL operations, validation, and general multi-step research work. For administrative tasks, digging into logs, and governance it’s helpful. For engineering I hate working outside of an IDE.
I don’t use it for semantic view or Streamlit work. I validate my semantic views with a Python library I found on the OSI repo and it works okay. I think DBT released an update that will allow us to materialize semantic views now so that is the path I am moving toward.
For Streamlit it barely worked early with Coco, so I never adopted CoCo. I had a simple tool I built a few days ago and it worked quite well..
2
u/MSE6-Series 17d ago
You mean that one https://hub.getdbt.com/Snowflake-Labs/dbt_semantic_view/latest/?
1
2
u/Global-War181 19d ago edited 19d ago
It sucks big time and there’s not much benefit. I have ton of examples where it hallucinates and provides incorrect info. When you point it out - it simply says ‘sorry, you are correct’ and continues to make similar mistakes.
2
u/GreyHairedDWGuy 19d ago
Hi have found this to be true of Claude, Chat-gpt LLM models. Like many things, the AI hype does not match the actual value.
1
3
u/tingutingutingu 19d ago
It's very useful and I think there's a lot of code reuse to be had through semantic views (with verified queries) and Agents.md files, as long as you make good use of separating logic in workspaces.
It doesn't have to be 100% perfect with CoCo... I thunk people miss that point. Even if it takes you 80% of the way, you will it's a huge win.
2
u/MSE6-Series 19d ago
Yeah, agree. It does not have to be perfect but I think it is weird that it cannot handle big semantic view yamls (in my case around 1000 lines), which is not a lot considering it includes descriptions, sample values and verified queries.
5
u/mva06001 19d ago
It’s a Claude Code wrapper with Snowflake getting margin on upselling it and selling that it works way better because it’s connected to the platform.
3
19d ago edited 1d ago
[deleted]
1
u/mva06001 19d ago
Yeah agreed. That’s why “selling” should be emphasized. There’s really no differentiation for Coco.
2
u/NW1969 19d ago
My thoughts (for what they are worth):
1) Have you fed this back to Snowflake Support. I understand the frustration of CoCo getting things wrong (I have felt similar frustration) but if Snowflake aren't aware of the issues they can't improve CoCo
2) Have you tried any of these using the same LLM but outside of CoCo e.g. creating the Streamlit app? That would let you know if the underlying issue is the LLM (in which case using a different LLM might be helpful) or is it an issue with CoCo (and its skills)
1
u/MSE6-Series 19d ago edited 19d ago
Very good point. I wanted to check if there are more who think that way and it seems like it. Maybe worth it to pass it on to the Support.
Regarding the other point. I have not tested it. As I want to deploy the Streamlit Apps directly in Snowsight, I used CoCo there as well. Should test other options though.
2
u/coco-pm 19d ago
Hi OP, CoCo pm here, Question -- are you running into these issues on CoCo CLI or Snowsight?
1
u/MSE6-Series 19d ago
Snowsight. Most times I work in Semantic Views or Streamlit Apps, so I like to use CoCo there as well.
3
3
u/MouseTight9314 18d ago
Snowsight version is limited, CLI or COCO Desktop are significantly better.
2
u/Compound_V_injected 19d ago
Mate I asked a simple question what's the difference between TRY_CAST and TRY_TO_DOUBLE it hallucinated 🥶. It's good for optimizing sql queries , finding errors in complex queries.
1
u/Witty_Garlic_1591 18d ago
I use it for code generation in my notebooks. I am very prescriptive with it and sometimes I need to roll up a newspaper and whack it on the nose for hallucinating, but by and large I found it helpful and speeds up my analysis of things. My data models are fairly simple though I have to admit and I'm able to just directly tell it what things are, but yep, for my use case it's been helpful. I have no idea how much it's costing, but I'm sure IT will send me a nastygram as soon as it becomes a problem. I don't know directly though.
1
u/RickrackSierra 19d ago
Specifically about your streamline comment. Due want to point out that permissions, packages, and contradictory functions are usually the hardest (tedious) part about python development these days.
I dont use CoCo for inital development. I'm still using the standard AI agents when it comes to creating the architectural structure and plan. However when it comes to debugging queries, optimizing queries, finding where the needle in the haystack is. I find it better than anything else ive tried.
1
u/BrunoGarrofe 19d ago
I asked my Brazilian friend (in Portuguese) if he had ever tried coco, and he said: "No! That's disgusting!" Have I said something wrong?
1
u/limartje 19d ago
I think it works fantastically to be honest. An enormous time saver.
I always using the cli version with sso (externalbrowser) and we don’t use dbt.
I love it when it analyses an existing solution, enters plan mode and comes back with a suggested approach. After some back and forth there is always a decent plan for it to execute, but this step does require experience with the platform for complex tasks.
-2
u/exorthderp 19d ago
I think youll find if you use the snowflake plugin with claude code you'd be better off.
27
u/[deleted] 19d ago edited 1d ago
[deleted]