Write-up · Never4gA
Why I removed the vector search lane from Never4gA.
Never4gA gives an AI coding session the context it needs from notes and project history, using search instead of model calls. For a while it also had a semantic search lane. I measured it against plain keyword search and it didn't earn its place, so I took it out.
What the lane was
Retrieval in Never4gA started with two lanes: full-text keyword search ranked with BM25, and traversal of the links between notes. I added a third, semantic search. An embedding model turns text into vectors, and notes are found by meaning instead of by matching words. The case for it is that it should find relevant notes that keyword search misses.
It came with a real cost. The embedding model was about ten gigabytes and ran on a different machine on my network, because it would not fit in my laptop's graphics memory.
How I measured it
I keep an evaluation corpus of real questions about my own notes, each with the notes that should come back and the ones that must not. It scores four things: recall in the top ten, MRR (how high the first right note ranks), latency, and forbidden results, meaning notes that should never appear for a question. I ran it on my real notes with the lane off, then on.
| Measure | Keyword only | With the lane |
|---|---|---|
| Recall@10 | 0.929 | 0.929 |
| MRR | 0.822 | 0.844 |
| Median latency | 56 ms | 516 ms |
| Slowest search | 123 ms | 3.4 s |
| Forbidden results | 2 | 0 |
What the numbers said
Recall did not move. The lane found no note that keyword search had missed, and the questions that failed, failed the same way in both runs.
MRR moved by 0.022, which the corpus cannot resolve. The evaluation harness prints its own resolution, and for this corpus it is 0.029. A difference smaller than that is not one it can report.
The lane was about nine times slower at the median. Most of the slowest search was the model loading cold, because a large model on a bursty access pattern gets evicted from memory between uses.
The only measurable thing the lane did was remove two forbidden results, and both were the same note. It ranked for common words like "the" and "why" because my keyword search did not filter stopwords. The lane was masking a bug in the lane next to it.
What I decided
I removed semantic retrieval. Keyword search and link traversal, fused, are the whole of retrieval now. I kept the interfaces for an embedding provider and a vector index as empty seams, with a test that fails if anything outside them imports one, so bringing it back would not be a rewrite.
Median search is 56 ms instead of 516. Nothing outside the laptop is needed to answer a query, no large model sits in memory, and there is no cold-start delay on the first search of a session.
What I did not show
The fair comparison is against keyword search with the stopword bug fixed, and I have not run it. I was offered the chance to fix the bug first and re-measure. I declined, because a lane whose only proven job was hiding one bug did not justify nine times the latency and a ten-gigabyte model while the question stayed open.
That is a judgment call and I want it read as one. It is also why the removal comes with conditions. I would revisit it only if all three are true: the corpus shows keyword search failing a class of question that can't be fixed, the stopword bug is fixed so the comparison is honest, and a model small enough for my laptop's graphics memory has been measured on this corpus instead of assumed. Any one alone is how the lane got in the first time.
What I took from it
The evaluation corpus is what made this a decision instead of a feeling. It is also what retired the lane, and it is what would notice keyword search getting worse. An optional capability needs a measurement that can say no to it.