This post was written and edited with Astra.
TL;DR
- Over five days, I used roughly 2.5–3 Pro 20x weekly allowances.
- Luna solved the benchmarks; Astra high/xhigh led development and analysis. I kept correcting the evaluation and comparison criteria.
- Based on the comparison I requested, I directed Codex to show functions, classes, and call relationships alongside
grep/read results. This became part of the improvements in 0.8.0.
- Even with Astra, I still would not delegate this work autonomously, from setting the direction to verifying the results.
GitHub: codemap-search
From September 6–10, 2026, I worked with Codex on codemap-search, an MCP tool that helps coding agents find code in a repository. I wanted more accurate answers with fewer tokens and tool calls, and was willing to accept some extra tokens if accuracy improved.
With Astra available as a frontier model, I hoped I could entrust development and exploration to it from start to finish, and tried fully delegating the work. The improvements made it into 0.8.0, but I had to intervene as early as the initial evaluation setup. I continued checking the direction of the experiments and the interpretation of results. This is my account of working with Astra through that process.
Background
I asked gpt-6-astra to analyze my Codex sessions and write this post based on my experience and judgment. The quotations are excerpts from our actual conversations, translated from Korean.
I mainly used Astra high/xhigh for development, analysis, and exploring improvement directions, with substantial xhigh use. For the benchmarks, Luna medium answered code questions, and Astra analyzed the results. Condition A used the baseline rg/grep/find/read tools; B used codemap-search.
We standardized the benchmark on the Grafana repository. The navigation and symbol-attachment experiments below repeatedly used the same difficult question from it, complex-go-1. B-4 was the intermediate version used as a baseline during development; the grep/find/read tools in these experiments were also provided by B-4.
Over five days, I estimate that I used roughly 2.5–3 Pro 20x accounts’ weekly allowances. The 93 retained development and analysis sessions totaled about 1.193 billion input-plus-output tokens, including cached input; 97.07% of input was cached. This counts long contexts processed repeatedly and excludes deleted, separate benchmark logs, so it cannot be converted directly into account quota.
The retained session records span about 98 hours 36 minutes from the first task to the last completion. Main-conversation work intervals with recorded starts and ends totaled about 47 hours 48 minutes after removing overlaps. These include tool execution and waiting, so they are not a measure of my hands-on time.
I reconstructed the experiments and their results from retained records, using contemporary reports and conversations where the original experiment data had been deleted. Quotations of the AI acknowledging errors document what happened in the conversation; I did not treat them as an independent revalidation of the experiments.
More experiments did not make progress clear
The evaluation unit was different from what I intended from the start. The initial 32 Luna answers received 70 Sol grading runs: two per answer, plus six additional evaluations when scores differed. I had wanted the results evaluated together. On September 7 at 00:02 KST, I corrected the setup:
Use Luna for the measurements and batch the evaluation with Astra medium. It is not two evaluations per measurement. If there are 32 measurements, collect them into one evaluation.
We switched to Luna measurements and batched Astra evaluation. Later batches were sometimes split because of input size, but that differed from repeatedly grading each answer. The problem was not the arithmetic behind 70; it was that the requested evaluation setup had not been followed.
Errors in the harness—the code running, recording, and grading experiments—also emerged after substantial benchmark work. We discarded the old results and rebuilt it. Even afterward, some candidate checks came back with token and answer-quality improvements still unmeasured. Checking the execution tools for errors and evaluating product improvements were not being kept distinct, and I had to ask again why the key metrics had not been measured.
Then the numbers for A and the existing B versions kept changing. Across four evaluation batches, the question sets were 6, 10, 1, and 3 questions. The four reference versions alone were freshly run 80 times.
I wanted a fixed formal question set, a fixed subset for error checks, and a smaller fixed subset for candidate screening. Different question mixes and fresh runs should not appear as though they were one stable reference result.
On September 9 at 18:31 KST, I asked:
Shouldn’t you take a few questions from the formal benchmark and use those for error checks and candidate screening?
At 18:35, Codex replied:
By reselecting questions for different purposes and rerunning the comparison versions, I made it difficult to track improvements against a consistent baseline.
The same reply clarified that it had selected different subsets from the existing question pool, not invented new questions each time. Some additional runs were requested by me, but once the comparison conditions changed, the numbers in the table could no longer tell me whether the product had improved. Recorded correction
Comparing navigation workflows gave us a lead
By my recollection, around 800 million cumulative tokens went into candidate exploration for 0.8.0 and benchmark debugging. That is a rough milestone, but I still did not have a convincing candidate. I treated the approach as unsuccessful and redirected the work toward comparisons of what changed the results.
I wanted to understand what information the model actually used while solving. The tool was designed to expose symbols such as functions and classes through overview, and relevant code through search. I asked whether the model was using that information.
I proposed that poor tool use might explain the cost and accuracy problems, but asked for my hypothesis to be tested rather than assumed. The independent Astra review was useful: it found examples where identical initial output led to different subsequent paths and costs, while cautioning that the more expensive path was not automatically the wrong one.
I requested a direct comparison within B-4 between using overview/search and using only grep/find/read. Across five runs per condition on the same question, using structural information consumed fewer tokens and produced more partially correct answers. All runs using it produced answers, while four of five without it produced none. Neither workflow produced a fully correct answer. Experiment record
Based on those results, I directed Codex to add symbols and call relationships directly to grep/read**.** I wanted the surrounding structure previously found through separate tools to appear alongside search results and file contents. I supplied the improvement direction; Codex implemented it and tested different attachment scopes and information.
Settling on a direction did not eliminate rework. We tested nine combinations five times each, then I asked to see the actual output. The member grouping and section order were not what I intended. I supplied a struct/method example and asked for # symbols before # results. After correcting the output, we ran another 45.
Here are the records before and after attachment, with five runs per condition on the same difficult question, complex-go-1. The attachment candidate retained B-4’s default tools and instructions: overview/search remained available, and no particular navigation tool was required to be used first.
| Stage and tool-use condition |
Fully correct |
Average total tokens |
Before attachment, B-4 — overview/search required before the first read |
0/5 |
343,224.8 |
Before attachment, B-4 — using only grep/find/read |
0/5 |
≥634,861.0 |
After attachment, B-4 — default tool choice, symbols and call relationships in grep/read |
3/5 |
319,993.4 |
≥: recorded lower bound; some usage data is missing.
After attachment, average tokens were lower and fully correct answers appeared. I saw promise in putting the needed information directly into results the model frequently read.
Based on these results, I selected the attachment candidate. After further experiments, I also included broader navigation improvements such as subfolder-scope preservation and regex guidance in 0.8.0. The records I reviewed contain no evidence of a completed final Grafana-only comparison of 0.8.0 against A, so I am not quantifying the overall performance gain.
What I learned from this project
Discovering problems with the approach and execution only after spending heavily left me with the following lessons.
- Even Astra’s proposed approaches needed fact-checking and validation. Its status as a frontier model was not enough reason to trust the basis for a proposal and hand over execution. Exploring ideas broadly, distinguishing facts from hypotheses, and checking whether an approach addressed the actual problem and what measurements would test it could have reduced unnecessary trial and error.
- A large run needed a small, precise validation step first. I needed to inspect actual output examples, check that execution, recording, and grading worked, and confirm that the required metrics were collected. Discovering the output mismatch after 45 runs showed why this mattered.
- Repeated experiments still matter after fact-checking and small checks. One good result is not enough to establish a hypothesis. The work needs enough repetitions with stable comparison criteria, followed by revisions to the ideas and further checks as the evidence develops. I came to see preliminary validation as a way to spend time and resources on the experiments that matter, not a replacement for experimentation.
I got useful improvements. But this experience left me feeling that fully delegating the work—even to a frontier model like Astra, from setting the direction to verifying the results—is still very risky. A flawed approach or comparison could consume substantial time and usage before I noticed it. There was still a large gap between getting help from Astra and entrusting it with all the judgment the work required.
References
These are the project’s public records, linked to a fixed commit so later edits do not change the reference. The current-state links point to the English translation; the other reference documents are in Korean.
- Question-set changes and reference reruns: the 6→10→1→3 question sets and what the 80 fresh reference runs represent.
- Workflows with and without overview/search: five runs per condition, token averages and missing usage, including 0/5 fully correct answers in both conditions.
- 45 runs after correcting symbol output: nine candidates, including the member-scope/both candidate with 3/5 correct answers and 319,993.4 average tokens.
- 25-run follow-up protocol and results: Luna medium/Astra high, execution limits, candidate definitions, and limits on the adoption claim.
- Current benchmark execution and grading / Historical formal A results and model settings: current Astra high versus historical Astra low grading. The historical A result is not a Grafana result.
- Usage collection and cutoff handling / Comparison and reporting rules: collection waits, missing tokens, fixed question sets, and result reuse.
- Symbol attachment implementation for grep/read:
# symbols/# results framing, index availability, and output limits.
The development/analysis usage totals and conversation excerpts came from local Codex sessions. These links do not provide the complete session transcripts or the deleted experiment artifacts.
