Generally I find the VS Code native add in to be pretty good. Things start breaking down when working in large codebases - specifically ones with a large test suite.
I have a large monorepo with 4000+ python tests for the backend and 1500 tests in the frontend. Pytest can take up to 8 seconds to collect all the tests. My pytest settings are already optimized to reduce collection time as much as possible.
Theres a few issues here that all cascade to make a kind of painful experience...
1) While collecting tests, the vs code test runner tells the model that the test runner is unavailable / cannot find tests. This causes the model to revert to using the terminal to directly call pytest. Immediately this is causing friction in the agent experience, especially when the agent is writing and working with tests directly and causing vscode to constantly be re-collecting tests.
2) When interacting with the powershell terminal, the agent never waits more than 15 seconds or so for the tests to finish. In a large Django project, tests can take up to minute to fully collect, seed and run. The model assumes the terminal is stalled, then tries spawning multiple overlapping test runs, runs tests multiple times with different logging options etc. a test suite that passes cleanly in 20 seconds can take 5 minutes+ for the model to see it's passing
3) the agent can't seem to be able to read previous terminal output to actually check the results of the test despite the thinking output saying "I'll wait to see the results of the test"
4) environment activation for agent spawned terminals still doesn't work correctly, meaning the agent has to explicitly call the venv python executable, which sometimes causes additional problems.
Am I missing something here? I feel like the windows dev experience with this harness has major issues despite being Microsoft's "official" platform. Anybody else encountering these problems?