Evolutionary Coding Agents: How FunSearch and AlphaEvolve Work

An evolutionary coding agent is a loop: a language model proposes changes to a program, an automatic evaluator runs and scores each version, and the best are kept as examples for the next round. Google DeepMind's FunSearch (Nature, December 2023) evolved single Python functions this way; AlphaEvolve (May 2025) evolves whole files with several models and scores.

Both systems found results their authors could check by machine: new mathematical constructions, a faster scheduling heuristic, faster kernels. Both also depend completely on the evaluator. Where nobody can write a cheap, trustworthy scoring function, the approach has little to say. This post explains how the loop works, what Google DeepMind reported in its own words, what the open source reimplementations claim, and what a defender should demand before evolved code reaches production. It ends with a small Python loop you can run yourself.

Key takeaways

• The evaluator is the system. The model only proposes; code that runs and scores each candidate does the selecting, so results are never better than the scoring function.
• FunSearch evolved one function inside a fixed skeleton, using a code model built on PaLM 2, an island based program database and millions of samples.
• AlphaEvolve evolves whole files in any language with an ensemble of Gemini 2.0 Flash and Pro, diff based edits, evaluation cascades and several scores at once.
• The engineering results are self-reported by Google: 0.7% of fleet compute recovered, a 23% kernel speedup worth 1% of Gemini training time, and 32% (paper) or 32.5% (blog) on a FlashAttention kernel.
• The mathematical results are checkable, with published verification code, including 593 spheres for the kissing number problem in 11 dimensions.
• Without a cheap, honest evaluator the method has little to offer, as the AlphaEvolve white paper itself says.
• Open source frameworks such as OpenEvolve and ShinkaEvolve run model written code on your machine by default. Isolate them and reproduce their claims.
• Evolved code is still code: it needs independent tests, review, run provenance and normal dependency patching.

What is evolutionary program search, and why does it need an automatic evaluator?

Evolutionary program search treats programs as a population. You start with one or more working programs, make variations of them, measure each variation, and keep the better ones so they can be varied again. Repeat for thousands or millions of rounds and the population drifts towards programs that score well on whatever you measured.

Classical genetic programming did this with hand written mutation operators: swap a subtree, change a constant. The AlphaEvolve white paper names the weakness: such operators are hard to design and may miss important properties of the domain.

Language models changed the mutation step, because a model trained on code proposes changes that look like a programmer's. Two papers made this explicit before FunSearch:

• Evolution through Large Models (Lehman and colleagues, arXiv 2206.08896, June 2022) used a code model as the mutation operator inside MAP-Elites, a quality diversity algorithm, and reported generating hundreds of thousands of working Python programs that build walking robots in a simulated domain.
• Language Model Crossover (Meyerson and colleagues, arXiv 2302.12170, February 2023) showed that few-shot prompting a model with several parent "genotypes" (bit strings, sentences, equations, Python code) and parsing its output works as a crossover operator.

The model gives you plausible variation, not truth. The Google DeepMind FunSearch blog post states the design goal: pair the model with an automated "evaluator" that guards against hallucinations and incorrect ideas.

So the evaluator has three jobs:

1. Reject what is broken. Candidates that crash, time out, exceed memory or return invalid output are discarded before they can influence anything.
2. Rank what works. Every surviving candidate gets a number, ideally a graded one, so the search can tell "slightly better" from "much better".
3. Represent what you actually want. If the score rewards something other than your real goal, the search will find that something. Evolutionary search is very good at exploiting loopholes in a scoring function.

The third job is the hardest and the one most often skipped in demonstrations.

The basic loop

Every system in this post runs the same loop; the AlphaEvolve white paper prints nearly this pseudocode:

# Conceptual loop, after Figure 2 of the AlphaEvolve white paper
while budget_remains():
    parent, inspirations = database.sample()
    prompt = build_prompt(parent, inspirations)
    change = llm.generate(prompt)            # a diff or a whole new function
    child = apply(parent, change)
    scores = evaluator.run(child)            # in a sandbox, with limits
    if scores is not None:
        database.add(child, scores)

Systems differ in what goes into the prompt, what counts as a program, how parents are chosen and how the evaluator is built.

The papers and systems covered, in one table

None of these systems changes model weights during the search; what evolves is code or prompts. The AlphaEvolve authors mention distilling the gains into future base models only as a possible next step. For AlphaEvolve, this post uses the blog date (14 May 2025) for the announcement and the arXiv white paper (v1, 16 June 2025) for technical claims.

How did FunSearch pair a language model with an evaluator and a program database?

FunSearch was published in Nature on 14 December 2023 by Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov and colleagues at Google DeepMind, with the mathematician Jordan Ellenberg among the collaborators. Its core idea: search for a program that produces the answer, not for the answer itself.

The specification: evaluate, a skeleton, and one function to evolve

The user gives FunSearch three things:

• An evaluate function that scores a candidate solution. For the cap set problem it checks that a set of vectors really has no three points on a line and returns the size of the set.
• A skeleton: boilerplate code and known problem structure that will not change. For cap sets the skeleton is a greedy algorithm that starts with an empty set and repeatedly adds the highest priority vector that does not break the constraint.
• An initial version of the function to evolve, which can be trivial. In the cap set case it was a priority function that returned a constant.

The paper reports that performance improves significantly when the skeleton carries prior knowledge. FunSearch does not invent algorithms from nothing: a person fixes the algorithm's shape and FunSearch searches over one decision inside it.

The program database and islands

Every candidate that passes evaluation goes into a programs database. Candidates that fail to run within the imposed time and memory limits, or that produce invalid outputs, are discarded.

To keep diversity, FunSearch uses an islands model: several subpopulations evolve independently. It samples an island, then programs within it, favouring higher scoring and shorter ones. Periodically it wipes the worst half of the islands and reseeds them from the best survivors.

Best-shot prompting

The prompt holds a few of the best sampled programs, ordered by score, and asks the model for the next version. The paper calls this best-shot prompting.

Which model, and how much compute

The paper used Codey, a code model built on the PaLM 2 family, through its API with no fine-tuning; the code availability section also lists the open StarCoder model. FunSearch runs as three kinds of asynchronous worker: the programs database, samplers that call the model, and evaluators. The authors typically used 15 samplers and 150 CPU evaluators, which they say fit on five CPU servers. They also used a sandbox for safely executing generated code on Google's internal systems; it is not in the public repository.

What FunSearch found

The headline results, as the paper states them:

• Cap sets. In dimension 8, FunSearch found a cap set of 512 vectors, larger than previously known, from scratch. Because the output was a program, the authors could read it, notice symmetries and build the 512-cap by hand.
• Cap set capacity. Via admissible sets, FunSearch's first program raised the lower bound on the cap set capacity from 2.2180 to 2.2184; reading it helped the authors reach 2.2202, still far from the upper bound of 2.756.
• Online bin packing. Starting from best fit, FunSearch evolved heuristics that used fewer bins than first fit and best fit on the OR-Library benchmarks OR1 to OR4 and on Weibull distributed instances, after training only on OR1 sized instances.

What FunSearch did not show

The paper is unusually clear about the conditions it needs. The authors say FunSearch currently works best for problems with three characteristics: an efficient evaluator, a "rich" scoring signal that grades improvement rather than pass or fail, and a skeleton with an isolated part to evolve. They give theorem proving as an example of a problem that does not fit yet.

Hard cases needed many runs: only four of 140 experiments found the 512 cap set in dimension 8. The public google-deepmind/funsearch repository has the discovered functions, a bin packing evaluation suite and a single threaded pipeline; its README says it does not contain language models, the sandbox for executing untrusted code, or the distributed infrastructure.

How does AlphaEvolve extend this to whole files, model ensembles and multiple metrics?

Google DeepMind announced AlphaEvolve on 14 May 2025 in a blog post describing "a Gemini-powered coding agent for designing advanced algorithms". The white paper, "AlphaEvolve: A coding agent for scientific and algorithmic discovery" by Alexander Novikov, Ngân Vũ, Marvin Eisenberger and colleagues, appeared on arXiv as 2506.13131 on 16 June 2025.

The white paper includes its own comparison with FunSearch. Restated:

Marking what may change

A user marks the parts AlphaEvolve may change with comments, # EVOLVE-BLOCK-START and # EVOLVE-BLOCK-END. Code inside is the starting solution; everything outside is a fixed skeleton tied to an evaluate function. A human decides the boundary, and the evaluation harness sits outside it. Keep that separation in any system you build: evolved code must not be able to edit the code that judges it.

Choosing the level of abstraction

The same problem can be attacked by evolving the solution directly, a function that constructs it (the FunSearch approach), or a search algorithm that finds it within a fixed budget. The authors hypothesise that constructors suit highly symmetric solutions and evolved search suits non-symmetric ones.

Prompt sampling

Prompts carry previous solutions from the database plus instructions. Users can add:

• Explicit context: fixed human written instructions, equations, code snippets or relevant literature, including PDF files.
• Stochastic formatting: template placeholders with human provided alternatives, sampled from probability distributions in a config file, to increase diversity.
• Rendered evaluation results: the program, its output and its scores.
• Meta prompt evolution: instructions suggested by the model and co-evolved in their own database, the self-referential idea Promptbreeder applied to prompts in 2023.

Diffs, not whole rewrites

For larger codebases, AlphaEvolve asks for diff blocks in a search and replace format:

<<<<<<< SEARCH
    # exact lines to find in the current program
=======
    # replacement lines
>>>>>>> REPLACE

For short code it can ask for a full rewrite instead. Diffs spare the model from reproducing hundreds of unchanged lines.

An ensemble of models

The white paper says AlphaEvolve used Gemini 2.0 Flash for throughput and Gemini 2.0 Pro for occasional higher quality suggestions, and that it performs better as the underlying model improves, unlike FunSearch.

Evaluation: cascades, model feedback, parallelism and several scores

The evaluator is a user supplied Python function returning a dictionary of metrics, with optional extras:

• Evaluation cascades: harder test stages unlocked only by doing well on easier ones, to prune faulty programs early.
• Model generated feedback: separate model calls grade hard to code properties such as simplicity.
• Parallel evaluation: the paper says spending on the order of 100 compute-hours on one candidate becomes feasible.
• Multiple scores: optimising several metrics often improved the one the authors cared about.

The evolutionary database and the pipeline

The database is inspired by MAP-Elites, which keeps the best program in each cell of a grid of features, and by island models. The system runs as an asyncio pipeline of controller, samplers and evaluators, optimised for throughput.

Ablations: which parts matter

On tensor decomposition and kissing numbers, the authors compared the full method with versions without evolution, without prompt context, without meta prompts, without full-file evolution, or with only a small model, and state that each component gives a significant improvement. Curves average three seeds and are plotted against fractions of a compute budget, not absolute hours or cost.

What results did Google DeepMind report?

Every number here is Google DeepMind's own claim, from the white paper unless marked as the blog. The engineering results were measured by Google on internal systems that outsiders cannot reproduce.

Reading the self-referential claim carefully

The white paper calls the kernel deployment an instance where Gemini, through AlphaEvolve, "optimizes its own training process". The measured effect is a 1% cut in training time, and the authors write that the gains are moderate and the feedback loops are on the order of months. It is a useful engineering result, not evidence of rapid self-improvement, and the authors do not claim otherwise.

The follow-up mathematics paper

On 3 November 2025, Bogdan Georgiev, Javier Gómez-Serrano, Terence Tao and Adam Zsolt Wagner posted arXiv 2511.02864, applying AlphaEvolve to 67 problems. Their abstract says it rediscovered the best known solutions in most cases and improved several. Again, these are constructions a verifier can check.

What the reported results do not show

• No outside measurement of the engineering numbers. The 0.7%, 23%, 1% and 32% figures are Google measuring Google systems.
• No full compute or cost accounting for any headline result.
• No public code to run AlphaEvolve. The alphaevolve_results README says it does not contain that code; it holds a notebook with the improved mathematical results and code to verify them.
• No results on problems without automated evaluation. The paper scopes these out on purpose.

Why do these results not generalise to problems without a cheap, trustworthy evaluator?

The AlphaEvolve white paper names its main limitation: it handles problems for which an automated evaluator can be devised, and in the natural sciences only some experiments can be simulated or automated. Five things go wrong as the evaluator weakens.

1. Cost per evaluation caps the number of generations. Even AlphaEvolve's thousands of samples must each be run. If one evaluation takes a week, the loop cannot iterate enough to matter.

2. Pass or fail gives the search nothing to climb. Without a graded score, selection cannot tell nearly right from hopeless.

3. The search optimises the score, not the goal. A timer that ignores correctness rewards code that skips work; weak assertions reward code that returns a constant. AlphaEvolve's engineering cases closed these gaps by design: the Borg and kernel heuristics could only change a ranking or tiling choice, and the FlashAttention changes were checked against the reference on randomised inputs and reviewed by experts. Those checks are part of the result.

4. Overfitting. Both papers held out data (OR-Library test sets for bin packing, half the kernel shapes, unseen recent workloads for scheduling). A demo that reports only the training score has not shown generalisation.

5. Correctness is often not a number. Security, maintainability and behaviour on untested inputs are hard to score quickly, and every proxy has blind spots. Coverage guided fuzzing is the same kind of loop with the same limit; see how coverage guided fuzzing works.

Where an evaluator is cheap and honest

The pattern fits exact checks (constructions, decompositions, packings), changes that cannot break correctness by design, optimisation against a strong reference on random inputs, and evaluations that parallelise on cheap hardware. It fits poorly where success depends on human judgement, rare events or long delays.

What open source reimplementations exist, and how should their claims be checked?

Neither system is available as runnable software from Google DeepMind. Two open frameworks follow the same design and are actively released; each was checked on its repository and package index on 23 October 2026.

Repository star counts and download figures change daily and say nothing about correctness, so they are not listed here.

How to check a reimplementation's claims

Treat every README result as self-reported until you reproduce it:

1. Read the evaluator. Is the score what the claim describes, or a proxy?
2. Look for a held-out set.
3. Check the baseline. "Matches published benchmarks" (the OpenEvolve README's wording for circle packing) is weaker than "beats them".
4. Count the budget: model calls, models, evaluator hours.
5. Rerun with several seeds. FunSearch found its best cap set in 4 of 140 runs.
6. Verify the output with code the search never touched, and benchmark performance claims on your own hardware.
7. Check what the framework executes by default (next section).

What these frameworks execute on your machine

Both frameworks run model written code. How they run it matters more than any benchmark claim.

OpenEvolve's evaluator module, read on its main branch on 23 October 2026, imports your evaluation file, writes each candidate to a temporary file and hands it to your evaluate function in the same Python environment, with a default timeout of 300 seconds and cascade thresholds of 0.5, 0.75 and 0.9. The README offers a Docker image as an alternative. ShinkaEvolve's README lists a default job_type of "local", with Slurm with Conda or Docker as alternatives.

That is reasonable for research tools, but running the example executes model written code with your permissions, environment variables and network access. Treat it like running an untrusted pull request.

How evaluator driven evolution differs from agents that learn skills

An evolutionary coding agent improves an artefact against a fixed external evaluator, and its own code does not change. Agents that learn from experience, such as Nous Research's open source Hermes Agent, change their own behaviour by writing memory and reusable skills; its README, read on 23 October 2026, describes "autonomous skill creation after complex tasks". There is no single external score in that loop. Both produce model written code that gets reused, but an evolutionary loop with a good evaluator measures every candidate, while a skill learning agent may save a script after one success. The controls below apply to both.

A minimal evolutionary loop in Python

This example evolves an online bin packing priority function, the kind of problem FunSearch tackled, starting from best fit. A random mutation operator stands in for the model; the rest has the shape of the real systems: fixed skeleton, isolated evaluator, population, selection and a held-out test set. It needs only the standard library and runs on Linux or macOS. Save it as evo_toy.py and run python3 -I evo_toy.py.

"""Toy evolutionary program search: evolve a bin packing priority function.

mutate() stands in for the language model. The evaluator runs every candidate
in a separate, isolated Python process with a timeout, CPU limit and empty
environment. That is not a real sandbox: run real model output only inside a
disposable container or VM with no network and no credentials.
"""
import json, math, os, random, resource, subprocess, sys, tempfile

CAPACITY = 100
TERMS = ["(cap - item)", "(cap - item) ** 2", "item / cap", "cap",
         "1.0 / (cap - item + 1)", "((cap - item) % 7)",
         "item / (cap - item)"]
TEMPLATE = "def priority(item, cap):\n    return {expr}\n"

HARNESS = r'''
import json, sys
ns = {}
exec(open(sys.argv[2]).read(), ns)
priority = ns["priority"]
def pack(items, capacity):
    bins = []
    for item in items:
        fits = [i for i, rem in enumerate(bins) if rem >= item]
        if not fits:
            bins.append(capacity - item)
            continue
        best = max(fits, key=lambda i: float(priority(item, bins[i])))
        bins[best] -= item
    return len(bins)
data = json.load(open(sys.argv[1]))
print(json.dumps({"bins": [pack(x, data["capacity"]) for x in data["instances"]]}))
'''

def make_instances(seed, n_instances=20, n_items=120):
    rng = random.Random(seed)
    return [[rng.randint(5, 70) for _ in range(n_items)] for _ in range(n_instances)]

def limit_child():
    resource.setrlimit(resource.RLIMIT_CPU, (5, 5))          # 5 s of CPU
    try:
        resource.setrlimit(resource.RLIMIT_AS, (512 * 2**20,) * 2)  # 512 MB
    except (ValueError, OSError):
        pass                                                # not enforced on every OS

def evaluate(expr, instances):
    """Return fitness (higher is better) or None if the candidate is rejected."""
    with tempfile.TemporaryDirectory() as tmp:
        cand, data, harness = (os.path.join(tmp, n) for n in ("cand.py", "data.json", "h.py"))
        open(cand, "w").write(TEMPLATE.format(expr=expr))
        json.dump({"capacity": CAPACITY, "instances": instances}, open(data, "w"))
        open(harness, "w").write(HARNESS)
        try:
            out = subprocess.run([sys.executable, "-I", harness, data, cand],
                                 capture_output=True, text=True, timeout=10,
                                 env={}, cwd=tmp, preexec_fn=limit_child)
            bins = json.loads(out.stdout)["bins"]
        except Exception:
            return None                                     # crash, timeout, bad output
    if len(bins) != len(instances) or not all(isinstance(b, int) for b in bins):
        return None
    excess = [(b - math.ceil(sum(x) / CAPACITY)) / math.ceil(sum(x) / CAPACITY)
              for b, x in zip(bins, instances)]
    return -sum(excess) / len(excess)                       # 0.0 would be optimal

def render(genome):
    """Turn a list of (coefficient, term) pairs into Python source."""
    return " + ".join(f"{c} * {t}" for c, t in genome)

def mutate(genome, rng):
    child = list(genome)
    r = rng.random()
    if r < 0.3 and len(child) > 1:
        child.pop(rng.randrange(len(child)))
    elif r < 0.6:
        child.append((round(rng.uniform(-2, 2), 2), rng.choice(TERMS)))
    else:
        i = rng.randrange(len(child))
        c, t = child[i]
        child[i] = (round(c + rng.gauss(0, 0.5), 2), t)
    return tuple(child)

def main(iterations=400, seed=1):
    rng = random.Random(seed)
    train, test = make_instances(1), make_instances(2)      # held-out test set
    baseline = ((-1.0, "(cap - item)"),)                    # best fit
    population = [(evaluate(render(baseline), train), baseline)]
    rejected = 0
    for _ in range(iterations):
        parent = max(rng.sample(population, min(3, len(population))))[1]
        child = mutate(parent, rng)
        score = evaluate(render(child), train)
        if score is None:
            rejected += 1
            continue
        population = sorted(population + [(score, child)], reverse=True)[:20]
    best_score, best = population[0]
    print("best fit  train", round(evaluate(render(baseline), train), 4),
          "test", round(evaluate(render(baseline), test), 4))
    print("evolved   train", round(best_score, 4),
          "test", round(evaluate(render(best), test), 4))
    print("rejected candidates:", rejected)
    print(TEMPLATE.format(expr=render(best)))

if __name__ == "__main__":
    main()

What the pieces correspond to

• HARNESS is the skeleton. The greedy packer never changes and only asks priority to rank bins that already have room, so a bad candidate packs badly but never illegally: the "correct by construction" property AlphaEvolve relied on for Borg.
• render fills the evolve block; mutate stands in for the model, which in a real system would receive the parent and top programs and return a diff.
• evaluate is the evaluator and sandbox boundary: a fresh -I isolated process, empty environment, throwaway directory, CPU limit and timeout. Crashes, hangs and malformed output score None.
• The score is mean excess bins over a simple lower bound, negated, so 0.0 would be perfect. population keeps the top 20; test is never seen during the search.

What it printed when we ran it

On 23 October 2026, with Python 3.9.6 on macOS and the defaults above (400 iterations, seed 1), the run took under a minute and printed:

best fit  train -0.0456 test -0.0535
evolved   train -0.0443 test -0.0513
rejected candidates: 13
def priority(item, cap):
    return -0.88 * (cap - item) + -0.26 * ((cap - item) % 7) + 2.3 * item / cap + -0.93 * (cap - item) + 1.7 * cap + -0.29 * item / cap + -2.03 * item / cap + 1.18 * 1.0 / (cap - item + 1) + -0.74 * item / cap

Best fit wasted about 4.6% of bins over the lower bound on training data and 5.4% on held-out data; the evolved function cut that to about 4.4% and 5.1%. With seed 2, and with seed 0 at 150 iterations, the search only tied best fit. Thirteen candidates were rejected because item / (cap - item) divides by zero when an item fills a bin exactly. The run shows the real papers' lessons in miniature:

• Most runs rediscover the baseline. AlphaEvolve matched the best known answer on about 75% of problems and improved about 20%; the toy improved on one seed of three.
• Gains are small and need held-out confirmation: about 0.2 percentage points here.
• Evolved code bloats. Four item / cap terms nearly cancel. Simplify before shipping and confirm the simpler version scores the same.
• The evaluator catches crashes, not intent. A valid but meaningless ranking would have been scored normally.

To use a real model, replace mutate with a call that returns a new function body, and move evaluate into a container first. Process isolation stops a buggy candidate hanging the loop; it does not stop a hostile one reading your files.

What does a defender need when evolved code ships to production?

Evolved code arrives as a diff like any generated code, but it was selected for one score over thousands of attempts by a process that executed every attempt. Risk sits in two places: while the search runs, and after its output ships.

Threats and controls

Step by step: a controlled evolution run and merge

The steps use OpenEvolve as the example because its README documents installation and configuration; the pattern applies to any framework that executes candidate code. Commands were checked against each tool's documentation on 23 October 2026.

Step 1. Install the framework pinned, with hashes.

Pin the release you reviewed and require hashes.

python3 -m venv .venv-evolve && . .venv-evolve/bin/activate
pip download openevolve==0.4.0 -d wheels/
# Build requirements.txt with exact versions and hashes for openevolve and every dependency,
# for example with pip-compile --generate-hashes, then install only what is listed:
pip install --require-hashes --no-deps -r requirements.txt
pip hash wheels/openevolve-0.4.0-py3-none-any.whl   # compare with the hash you recorded

If the wheel filename differs on your platform, use the name pip download printed.

Step 2. Separate what evolves from what judges.

Keep the evolved program and the evaluator in different directories, and hash the evaluator and data before the run.

mkdir -p run/evolve run/judge run/out
cp initial_program.py run/evolve/
cp evaluator.py config.yaml eval_data.json run/judge/
sha256sum run/judge/* | tee run/out/judge.sha256

On macOS use shasum -a 256 in place of sha256sum.

Step 3. Run evaluations in a locked down container.

The controller needs the model API; candidate code needs no network. Where evaluation runs in a separate process, give it a container with no network. Where it shares the controller's process, as in OpenEvolve's default, run everything in a container whose only route out is an egress proxy to the model endpoint, with a short lived, low limit API key. An evaluation container with no network:

docker run --rm \
  --network none \
  --read-only \
  --cap-drop ALL \
  --security-opt no-new-privileges \
  --pids-limit 256 \
  --memory 1g --cpus 1 \
  --tmpfs /tmp:rw,size=256m \
  -v "$PWD/run/judge:/judge:ro" \
  -v "$PWD/run/evolve:/evolve:ro" \
  -v "$PWD/run/out:/out:rw" \
  python:3.12-slim \
  python -I /judge/evaluator.py /evolve/candidate.py

Pin the image by digest, and never mount your home directory, SSH agent, cloud credentials or Docker socket. See sandboxing an AI agent that runs shell commands for the wider isolation choices.

Step 4. Fix the seed and record the run.

OpenEvolve's documented configuration includes random_seed and max_iterations. Set both and write a run manifest:

cat > run/out/run-manifest.json <<'EOF'
{
  "framework": "openevolve==0.4.0",
  "config_sha256": "<sha256 of config.yaml>",
  "evaluator_sha256": "<sha256 of evaluator.py>",
  "eval_data_sha256": "<sha256 of eval_data.json>",
  "models": ["<model ids exactly as configured>"],
  "random_seed": 42,
  "max_iterations": 1000,
  "best_program_sha256": "<sha256 of the program you intend to merge>",
  "train_score": "<from the run log>",
  "heldout_score": "<from your independent test>",
  "started_utc": "<timestamp>",
  "operator": "<who ran it>"
}
EOF

Step 5. Verify the judge was not changed.

sha256sum -c run/out/judge.sha256

If any hash fails, discard the run.

Step 6. Test the winner with tests the search never saw.

The evaluator chose the winner, so its score is the least informative number you have. Run a held-out set of production like inputs, differential tests against the current implementation on randomised inputs (as AlphaEvolve did for FlashAttention), property based tests for invariants, and your normal checks. Our list of tests to demand before merging AI written code applies directly.

Step 7. Simplify, then review like any other change.

Remove dead terms, retest, and put the manifest hash in the commit so provenance travels with the code:

git checkout -b evolved/packing-heuristic-2026-10
cp run/evolve/best_program.py src/scheduler/priority.py
git add src/scheduler/priority.py
git commit -m "Replace packing priority with evolved heuristic

Evolution-Run-Manifest: sha256:<hash of run-manifest.json>
Heldout-Excess: 0.0513 (baseline 0.0535)"

Require review from the module's code owner; nobody on the team wrote this code.

Step 8. Sign the run manifest as an attestation.

cosign can sign an in-toto attestation over any file, with the manifest as predicate:

cosign attest-blob \
  --predicate run/out/run-manifest.json \
  --type https://example.com/evolution-run/v1 \
  --key cosign.key \
  --bundle run/out/priority.py.attestation.json \
  src/scheduler/priority.py

cosign verify-blob-attestation \
  --bundle run/out/priority.py.attestation.json \
  --key cosign.pub \
  src/scheduler/priority.py

Use a type URI your organisation controls. Keep emitting SLSA build provenance (predicate type https://slsa.dev/provenance/v1) for releases: the manifest records how the source was produced, the build provenance how the release was built from it.

Step 9. Roll out behind a flag and measure.

Keep the old implementation behind a flag and compare the production metric the evaluator was meant to predict, as the AlphaEvolve scheduling case did: simulator, unseen test set, then fleet measurement.

Step 10. Keep the dependencies patched.

The framework, the evaluator's libraries and any imports the evolved code added are open source components that receive vulnerability reports. Generate SBOMs for the repository and the evaluation image, scan them, and patch on your normal schedule.

Where Safeguard fits, honestly

Safeguard does not run evolutionary searches, does not evaluate algorithms, and its models are single purpose security models that do not do mathematics or scientific discovery. The fit is with the code these systems emit and the software they run on. Safeguard generates and analyses SBOMs in SPDX and CycloneDX for repositories and container images, matches components against vulnerability databases, ranks findings with reachability analysis, and evaluates policies and gates that can block a merge or a release. It supports attestation and signing with SLSA, Sigstore and in-toto, and its Griffin models can open tested fix pull requests through autonomous remediation, which ships switched off by default. None of that judges whether an evolved heuristic is correct; that remains your evaluator's and your reviewers' job.

Common mistakes

• Treating the evaluator's score as the result. The winner was selected to maximise it, so it is biased upward. Report held-out numbers.
• Letting evolved code reach the evaluator, its data or its scoring file.
• Running the quickstart on a workstation with credentials. Candidate code runs as you.
• Reporting one run instead of the spread across seeds.
• Beating a trivial baseline and calling it state of the art.
• Shipping bloated output without simplifying and retesting.
• Quoting "23% faster" without its conditions: one kernel's shapes, Google hardware, Google's measurement.
• Treating "the model improved its own training" as a trend. It was a 1% cut with months long feedback loops.
• Skipping provenance, so nobody can later say which models, data and evaluator produced a heuristic.

FAQ

What is the difference between FunSearch and AlphaEvolve?

FunSearch (Nature, December 2023) evolves one Python function of 10 to 20 lines with a small code model, millions of samples and one score. AlphaEvolve (May 2025) evolves whole files in any language with an ensemble of Gemini models, thousands of samples, richer prompts and several scores.

Is AlphaEvolve open source?

No. Google DeepMind published a repository with verification code for AlphaEvolve's mathematical results, and its README states that it does not contain the code to run AlphaEvolve. The May 2025 blog post said Google DeepMind was planning an early access programme for selected academic users. Open source frameworks such as OpenEvolve and ShinkaEvolve follow a similar design but are separate projects.

How does AlphaEvolve know if its code is correct?

Through a user supplied evaluation function. Mathematical constructions are checked exactly; engineering changes were either limited to parts that cannot break correctness or checked against a reference on randomised inputs and confirmed by experts. AlphaEvolve itself does not prove correctness.

Did AlphaEvolve really improve on Strassen's algorithm?

The white paper reports a procedure that multiplies two 4 by 4 complex valued matrices with 48 scalar multiplications, which it describes as the first improvement in this setting since Strassen's 1969 algorithm. The claim is specific to complex valued 4 by 4 matrices. The Google DeepMind blog notes that an earlier specialised system had improved 4 by 4 multiplication only for binary arithmetic.

Can an evolutionary coding agent improve itself?

Not in a compounding loop. AlphaEvolve's kernel heuristic cut Gemini's training time by 1%, which the authors describe as a moderate gain with feedback loops of months. None of the systems in this post evolves its own search code or model weights.

What problems are evolutionary coding agents good at?

Problems with a fast, honest, graded evaluator: verifiable constructions, heuristics inside systems that already enforce correctness, and performance tuning against a reference. Natural science experiments that cannot be simulated are out of scope, per the AlphaEvolve white paper.

Is it safe to run OpenEvolve or ShinkaEvolve on my laptop?

They execute model written code; OpenEvolve evaluates in your Python environment with a timeout and ShinkaEvolve defaults to local jobs. Use a disposable container or VM with no credentials and no network for evaluation, and pin the versions you reviewed.

What should I record when evolved code goes to production?

Framework version, models, seed, hashes of the evaluator, configuration, data and merged program, training and held-out scores, and the operator. Sign it as an attestation over the merged file and reference it from the commit.

Fix it without waiting for upstream

An evolutionary search framework is itself an open source Python package with its own dependency tree, and the code it emits can import new libraries, so a single run adds open source components to your codebase in two places. Those components receive vulnerability reports like any others, and an evolved heuristic is only as safe as the packages it runs on.

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation covers eight categories, including dependencies, licenses, security posture and findings. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds. The safeguard-sh/gold-scan-action GitHub Action can fail a build on a known exploited vulnerability, on any known vulnerability, or never, so a version you have just removed cannot quietly come back in next week's merge.

3. Gold Registry: start from components that are already fixed

registry.safeguard.sh provides hardened, drop in replacements for upstream packages and container images. Every Gold artifact is held continuously to zero exploitable critical or high CVEs, zero malware and a completed license check, with SLSA L3 or higher provenance, a Sigstore signature and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. This fork and patch flow is in early access today, so ask which packages it covers before you plan around it.

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

None of these demos run an evolutionary search. They show how Safeguard handles the code and dependencies that a search framework and its output add to a repository.

The first demo is a real Safeguard CLI session: asked to fix a lodash CVE, the assistant reads the finding and advisory, asks before each command, edits package.json to lodash 4.17.21, runs npm install and npm test with your approval, prints a summary and pushes nothing. The same approval driven flow applies when a dependency of your evolution framework, or a library an evolved function now imports, has a fixed version available.

Click through it at your own pace, or open it full screen.

The second demo connects Claude to the Safeguard MCP server with OAuth and chooses which tools Claude can use, installed from Claude's connector directory rather than through a hand written configuration. Once your repositories and evaluation images are in Safeguard, the same connection lets you ask which projects contain a given package version when an advisory lands for something your search pipeline depends on.

Click through it at your own pace, or open it full screen.

The third demo sets up the local runner on your own machine, approves it with a device code, then scans from the terminal. For an evolution workflow, that means scanning the repository and the evaluation container image from the same isolated build host where you run searches, before the winning program is merged.

Click through it at your own pace, or open it full screen.

To try the CLI, download Safeguard and follow the CLI documentation.

Quick check

Sources

• Google DeepMind: AlphaEvolve, a Gemini-powered coding agent for designing advanced algorithms (14 May 2025)
• Novikov et al.: AlphaEvolve, a coding agent for scientific and algorithmic discovery (arXiv 2506.13131)
• Romera-Paredes et al.: Mathematical discoveries from program search with large language models (Nature, 2023)
• Google DeepMind: FunSearch, making new discoveries in mathematical sciences using large language models
• google-deepmind/funsearch repository README
• google-deepmind/alphaevolve_results repository README
• Georgiev, Gómez-Serrano, Tao, Wagner: Mathematical exploration and discovery at scale (arXiv 2511.02864)
• Lehman et al.: Evolution through Large Models (arXiv 2206.08896)
• Meyerson et al.: Language Model Crossover, variation through few-shot prompting (arXiv 2302.12170)
• Fernando et al.: Promptbreeder, self-referential self-improvement via prompt evolution (arXiv 2309.16797)
• Lange, Imajuku, Cetin: ShinkaEvolve, towards open-ended and sample-efficient program evolution (arXiv 2509.19349)
• ShinkaEvolve repository README (Sakana AI)
• OpenEvolve repository README
• OpenEvolve evaluator source, main branch
• PyPI: openevolve
• PyPI: shinka-evolve
• Hermes Agent README (Nous Research)
• SLSA v1.0 provenance specification
• Sigstore: verifying in-toto attestations with cosign
• cosign attest-blob command reference
• Python documentation: resource module
• CycloneDX specification overview

Last reviewed: 23 October 2026. The open source frameworks named here release often; check each repository for the current version and defaults before running any command above.