Keep an index fresh by storing a content hash per file, re-chunking and re-embedding only files whose hash changed, and removing rows for deleted files. Record the embedding model version with each vector. A full rebuild is then needed only when you change the model or chunking rules.
Why freshness is the real work
Building the first index is a weekend. Keeping it correct is the ongoing job. A stale index answers confidently about code that no longer exists. A model trained on the code goes stale the same way, only slower to fix. See myths about training on your codebase.
The recipe
1. Hash files. Compute a hash of each file's content. Store it with the path.
2. Diff on each run. List files now, compare to the stored table.
- New path: chunk and embed.
- Same path, different hash: delete old chunks, then re-chunk and embed.
- Path missing: delete its chunks.
- Same hash: skip.
3. Trigger on events. Run after merges to the main branch, or on a schedule, such as hourly. A webhook from your source host is cleaner than polling.
4. Handle renames. A moved file has the same hash and a new path. Update the path in place and skip re-embedding.
5. Keep both indexes in step. If you use keyword and vector search together, update both in the same job so they never disagree. See hybrid retrieval.
Chunk level hashing
If files are large, hash each chunk. After parsing, only chunks whose text changed need new vectors. Tree-sitter, a parsing library, is built to update syntax trees efficiently as files change, and it returns useful trees even with syntax errors, so it suits half edited code. Chunking by syntax, as in chunking source code, keeps chunk boundaries stable when you edit one function.
Version everything that affects vectors
Store, per row: the embedding model name and version, the chunking rule version, and the time indexed. Ollama's documentation warns that mixing embedding models yields incompatible vectors. A version column lets your job detect the mismatch and trigger a rebuild instead of mixing.
Branches
Decide which branch you index. The default branch is the simplest. If developers need answers about their feature branch, index per branch with a branch label, and delete branch chunks when it merges.
Deletions and secrets
If a secret is committed and then removed, make sure the old chunks are deleted from the index, not just excluded from new runs. Rotate the secret regardless. See what to scan before you index.
Monitor it
Alert when the index is more than a set number of commits behind, when a run fails, or when row counts drop sharply. A silent failure in the freshness job is the common way indexes rot.
Quick check
How often should I rebuild everything?
Only when the model or chunking rules change. Otherwise incremental updates are enough.
Is a file modification time enough instead of a hash?
It is cheaper but unreliable, since checkouts and builds touch times. A content hash is safer.
Sources
Last reviewed: 19 October 2026.
