Automated Prompt Optimisation: DSPy, OPRO, Promptbreeder, TextGrad

Automated prompt and program optimisation means letting an algorithm, usually another language model, rewrite the instructions and examples in an LLM pipeline and keep whichever version scores best on a metric you define. DSPy compiles declarative modules into prompts and few-shot demos, OPRO asks a model to propose better instructions from a scored history, Promptbreeder evolves a population of prompts and the prompts that mutate them, and TextGrad passes written critiques backwards through a pipeline like gradients. None of them changes model weights unless you ask for fine-tuning, and every one of them produces an artifact that should be versioned, reviewed and signed like code.

This post explains each method from the authors' own papers and docs, puts the reported results next to the compute and benchmarks behind them, runs a small DSPy optimisation you can reproduce without an API key, and then covers the part most write-ups skip: dev set overfitting, evaluation leakage, and the change control a compiled prompt needs before production. It is a research explainer, not a ranking. Safeguard does not build or sell prompt optimisers; where we fit is the software those pipelines run on, and we say so plainly near the end.

Key takeaways

• The thing being optimised is text, not weights. DSPy (by default), OPRO, Promptbreeder and TextGrad edit instructions, demonstrations or other text. DSPy's BootstrapFinetune changes weights, and only when you choose it.
• The evaluator is the real specification. Exact match, a test suite or an LLM judge: whatever the metric rewards is what you get, blind spots included.
• Reported gains are large but conditional. Each headline number in the papers belongs to one model, one benchmark and one budget.
• Selection on a small dev set inflates scores. In our reproducible run, the program that won on a 40-example dev set (60.0 percent) scored 46.0 percent on 400 held-out examples, below a simpler program.
• The output is a deployable artifact. A saved DSPy program contains rewritten instructions and verbatim training examples; pickled programs can execute code on load. Diff, review, pin, sign and gate it like source code.
• This is bounded search, not recursive self-improvement. "Self-referential" in Promptbreeder's title means the mutation prompts are also mutated, not that the system rewrites its own model.

What is automated prompt and program optimisation?

A modern LLM application is rarely one prompt. It is a pipeline: retrieve documents, reason over them, call a tool, check the answer. Each step carries an instruction, sometimes worked examples, and an output format, and changing those strings can move accuracy by tens of points. Prompt optimisation treats the strings as parameters and searches for values that maximise a score.

Four research lines defined the field between September 2023 and June 2024, and two newer optimisers built on them. First-version dates are from the arXiv API, checked on 23 October 2026.

Underneath sits Automatic Prompt Engineer (APE, arXiv 2211.01910, November 2022), which showed that an LLM could propose instructions and a score function could choose among them, matching or beating human-written instructions on 19 of 24 tasks.

Why optimise prompts and pipelines automatically instead of by hand?

Prompts do not transfer between models. The DSPy paper describes hand-built templates as "lengthy strings discovered via trial and error". If you change model or provider every quarter, re-running an optimiser against a fixed metric is cheaper than re-tuning every string by intuition.

Multi-step pipelines have a credit assignment problem. When a retrieval and answer pipeline gets a question wrong, which prompt was at fault? The MIPRO paper names this directly: you usually have labels only for the final output. Optimisers that score the end result can still improve intermediate steps by keeping the demos and instructions that lead to good answers.

Small, cheap models benefit most. TextGrad improves gpt-3.5-turbo with feedback from gpt-4o, reasoning that "by paying a fixed cost to optimize a Prompt" the cheaper model can then serve traffic. DSPy reports programs compiled for T5-Large and llama2-13b-chat competing with expert prompt chains written for GPT-3.5.

Hand tuning leaves no record. An optimiser run logs every candidate and its score, which is only useful if you keep it.

The catch in every case is the metric. An optimiser is a very persistent intern that will find whatever your metric rewards, including its mistakes.

How does DSPy compile declarative modules into prompts and few-shot examples?

DSPy is an open source Python framework from Stanford NLP (MIT licence; repository created January 2023 per the GitHub API). The current PyPI release is dspy 3.4.1, uploaded 12 October 2026, for Python 3.10 up to but not including 3.16. Its README calls it a framework for "programming, not prompting" language models.

Signatures, modules and adapters

You declare what a step does, not how to prompt for it. A signature names inputs and outputs, as a string such as "question -> answer" or a class with typed fields and a docstring. A module decides how it runs: dspy.Predict asks directly, dspy.ChainOfThought adds a reasoning field, dspy.ReAct runs a tool loop. Modules compose into a program in plain Python. At run time an adapter renders the signature, instruction and stored demos into chat messages and parses the reply. That rendered prompt is what optimisers change.

What "compile" means

The paper calls optimisers "teleprompters" (the docs now say optimizers) and defines compilation as taking a program, a training set and a metric and returning a new program. The code does not change; each predictor's demos and, for some optimisers, its instruction text do. The docs say the training inputs "may be very small (i.e., only 5 or 10 examples)".

What the optimisers do

The DSPy optimizers page, read on the main branch on 23 October 2026, groups them into five families.

The docs suggest BootstrapFewShot for about 10 examples, random search for 50 or more, and MIPROv2 for longer runs where you have "200 examples or more to prevent overfitting". They estimate that "A typical simple optimization run costs on the order of $2 USD and takes around ten minutes", ranging from cents to tens of dollars; those are the project's own figures.

MIPROv2, per its docs, bootstraps traces and keeps those from high-scoring runs, drafts candidate instructions from your code, data and traces, then runs a Bayesian search over combinations of instructions and demos on mini-batches, returning the program that scores best on the full validation set. GEPA, available as dspy.GEPA, lets the metric return written feedback with the score, and a reflection model reads execution traces to propose new instruction text.

What the DSPy paper reported, and what it did not

On GSM8K the authors sampled 200 training and 300 development examples and tested on the 1.3k official test set; on HotPotQA they used a ColBERTv2 retriever over the 2017 Wikipedia abstracts. They report that "within minutes to tens of minutes of compiling", simple programs rose from 33 to 82 percent on GSM8K and from 32 to 46 percent on HotPotQA with GPT-3.5, and from 9 to 47 and 22 to 41 percent with llama2-13b-chat, at a cost of "a few thousand" program runs in the expensive settings.

What it did not show: the original optimisers mostly selected demonstrations, two task families were tested, and the baselines were chains the authors implemented with 2023 models. The numbers show the abstraction works, not what you will get on your task.

How does OPRO use a model as the optimiser over a scored history of attempts?

OPRO, Optimization by PROmpting (Yang and colleagues, Google DeepMind), describes the problem in natural language, shows a model its previous attempts with their scores, and asks for a better one.

The loop

1. Build a meta-prompt with a task description, a few training examples, and past instructions sorted by score.
2. Ask the optimiser LLM for new instructions: by default 8 calls per step at temperature 1.0.
3. Score each candidate with a scorer LLM by inserting it into each training question and measuring accuracy.
4. Add candidates and scores to the history. The meta-prompt keeps "the best 20 instructions so far and 3 randomly picked exemplars".
5. Repeat until scores plateau or the budget runs out.

The paper first demonstrates the loop on linear regression and small travelling salesman instances, and is frank that context length and sensitivity to the starting point limit it as a mathematical optimiser.

What was found

Instructions can go before the question (Q_begin), after it (Q_end) or at the start of the answer (A_begin). On GSM8K with PaLM 2-L as scorer, instruction-tuned PaLM 2-L found "Take a deep breath and work on this problem step-by-step" at 80.2 percent test accuracy, against 71.8 percent for "Let's think step by step" and 34.0 percent for an empty string. Other optimiser models found different wording with similar scores: 79.9 percent for "Break this down", 78.5 percent for gpt-3.5-turbo's suggestion.

The famous phrase obscures the lesson. The winning instructions were not insightful; they were what this scorer responded to on this benchmark. Transfer to two other maths datasets, MultiArith and AQuA, was positive but modest.

Compute and data

For GSM8K the authors sampled 3.5 percent of the 7,473 training examples and kept that subset fixed, so intermediate scores approximate training accuracy. For Big-Bench Hard they used 20 percent of each task. Each step costs 8 optimiser calls plus 8 passes over the training subset. The repository README carries a "Precaution on API costs" warning.

What OPRO did not show

Section 5.4 says the default setup does not hold out a validation set "for simplicity". In a separate check with data split into thirds, training and validation accuracy moved together on two tasks, which the authors argue makes training accuracy acceptable when candidates overfit "to a similar extent". That holds for those tasks, not necessarily yours. The authors also say the optimiser "does not effectively utilize error cases", a gap GEPA later targets.

How does Promptbreeder evolve both task prompts and the mutation prompts that change them?

Promptbreeder (Fernando and colleagues, Google DeepMind) applies a genetic algorithm to prompts, with an LLM doing the mutation.

Units, selection and budget

Each unit in the population typically holds two task prompts (one produces working-out, the second turns it into an answer), one mutation prompt (an instruction telling the LLM how to rewrite task prompts), and in the few-shot case two or three correct workings-out. The initial population mixes a problem description with author-written lists of "thinking styles" and mutation prompts.

Selection is a binary tournament: pick two units, overwrite the loser with a mutated copy of the winner. Fitness is accuracy on "a batch of 100 Q&A pairs" from the training set. Runs used 50 units for "typically 20-30 generations", which the appendix puts at 1,000 to 2,000 fitness evaluations. The fittest unit of the whole run is then tested once.

Nine mutation operators in five classes

Hypermutation is the "self-referential" part: the system improves the prompts that improve prompts. The underlying PaLM 2-L is never changed.

Results and limits

On zero-shot GSM8K the paper reports 83.9 percent against OPRO's 80.2 percent, noting that the winning evolved prompt was the single word "SOLUTION". Across the arithmetic and commonsense benchmarks borrowed from the Plan-and-Solve paper, Promptbreeder reports the top zero-shot and few-shot numbers in its own table, and it also evolves prompts for hate speech classification on ETHOS.

Several baseline rows in Table 1 are copied from the Plan-and-Solve paper, which used text-davinci-003; the authors mark them in brackets. There is one model family, no dollar or token cost comparison, and a one-word winner on GSM8K suggests a model quirk more than a portable reasoning strategy. We did not find an official code release.

How does TextGrad treat model feedback as a gradient through a computation graph?

TextGrad (Yuksekgonul and colleagues at Stanford, published in Nature in March 2025 as "Optimizing generative AI by backpropagating language model feedback") borrows PyTorch's shape. The textgrad package on PyPI is MIT licensed; its latest upload, 0.1.8, is from 26 March 2025.

The backward system prompt tells the feedback model it is "the gradient (feedback) engine", there to criticise rather than solve. The "gradient" is a paragraph of advice; the update is another LLM call that rewrites the variable.

Instance optimisation versus prompt optimisation

In instance optimisation, the variable is one answer, code snippet or molecule, refined at test time. In prompt optimisation, it is a system prompt meant to work across a benchmark. The distinction matters when reading results:

• GPQA, 51 to 55 percent is instance optimisation: each GPT-4o answer is refined at test time with model-written critique. It is not a reusable prompt.
• LeetCode Hard, 20 percent relative gain is instance optimisation of code with test feedback.
• Prompt optimisation used gpt-3.5-turbo-0125 forward and gpt-4o backward, batch size 3, 12 iterations, on 50/100/100 train, validation and test splits for Big-Bench Hard tasks. Object Counting rose from 77.8 percent (zero-shot chain of thought) to 91.9 percent, against 84.9 percent for DSPy's random search with 8 demos. On Word Sorting and GSM8K, TextGrad tied the DSPy baseline at 79.8 and 81.1 percent, and the authors note the two make "complementary adjustments".

The molecule and radiotherapy planning results are proof-of-concept demonstrations scored by in-silico tools and clinical metrics; we report them as the authors' claims only. The method's benefit is entangled with access to a stronger critic model, and test-time refinement multiplies per-query cost, which the paper does not price.

How do the four approaches compare?

Maturity differs. DSPy is a maintained framework (111 releases of the dspy package on PyPI as of 23 October 2026). OPRO's research code is published by Google DeepMind. TextGrad is a library with a slower cadence. Choose by what your team can operate and evaluate, not by the largest number in an abstract.

A worked example: a small DSPy program before and after optimisation

This example needs no API key and no network. Instead of a hosted model it plugs in a stand-in model built on DSPy's own DummyLM test class. The stand-in reads the few-shot demos in the prompt DSPy builds, finds the demo whose input shares the most words with the query, and copies its label; with no close demo it falls back to a weak rule ("bugfix" if the message contains "fix", otherwise "feature"). It ignores instructions, so it isolates one thing: demo selection, which is what DSPy's few-shot optimisers change.

The task: label a commit message as security, bugfix, feature or docs. The data is synthetic, generated from templates with a fixed seed, with 20 percent of messages borrowing a verb from another class so the task is noisy: 40 training, 40 dev and 400 test examples.

Step 1: install a pinned DSPy

python3.14 -m venv .venv
. .venv/bin/activate
pip install "dspy==3.4.1"
python -c "import dspy; print(dspy.__version__)"   # 3.4.1

Step 2: program, metric and optimisers

import random, re, hashlib
import dspy
from dspy.utils import DummyLM

LABELS = ["security", "bugfix", "feature", "docs"]
random.seed(7)

PARTS = {
    "security": (["sanitize", "escape", "validate", "restrict", "harden", "bound-check"],
                 ["user input in", "path handling in", "token checks in", "upload size in", "redirect target in"]),
    "bugfix":   (["fix", "correct", "handle", "repair", "avoid"],
                 ["off-by-one in", "null value in", "timeout in", "race in", "wrong total in"]),
    "feature":  (["add", "support", "introduce", "enable", "expose"],
                 ["dark mode for", "CSV export in", "retry option in", "bulk edit in", "webhooks for"]),
    "docs":     (["document", "clarify", "update readme for", "explain", "reword"],
                 ["setup steps for", "config keys in", "examples in", "changelog for", "API notes for"]),
}
MODULES = ["auth", "parser", "billing", "api", "cli", "uploader", "scheduler", "search"]

def make(label):
    verbs, objs = PARTS[label]
    if random.random() < 0.2:  # 20% noise: borrow a verb from another class
        verbs = PARTS[random.choice([l for l in LABELS if l != label])][0]
    return f"{random.choice(verbs)} {random.choice(objs)} {random.choice(MODULES)}"

def dataset(n):
    return [dspy.Example(commit=make(l), label=l).with_inputs("commit")
            for l in (random.choice(LABELS) for _ in range(n))]

trainset, devset, testset = dataset(40), dataset(40), dataset(400)

FIELD = re.compile(r"\[\[ ## (\w+) ## \]\]\n(.*?)(?=\n\n|\Z)", re.S)
fields = lambda text: {k: v.strip() for k, v in FIELD.findall(text)}
words = lambda s: set(s.lower().split())

class NearestDemoLM(DummyLM):
    """Offline stand-in: copy the label of the most similar demo in the prompt."""
    def __init__(self):
        super().__init__([], follow_examples=True)

    def _use_example(self, messages):
        query = fields(messages[-1]["content"]).get("commit", "")
        best, best_score = None, 0.0
        for user, asst in zip(messages[:-1], messages[1:]):
            if user["role"] == "user" and asst["role"] == "assistant":
                d_in = fields(user["content"]).get("commit")
                d_out = fields(asst["content"]).get("label")
                if d_in and d_out:
                    s = len(words(query) & words(d_in)) / len(words(query) | words(d_in))
                    if s > best_score:
                        best, best_score = d_out, s
        if best is None or best_score < 0.25:
            best = "bugfix" if "fix" in query else "feature"
        return self._format_answer_fields({"label": best})

dspy.configure(lm=NearestDemoLM())

class Classify(dspy.Signature):
    """Classify a commit message as security, bugfix, feature or docs."""
    commit: str = dspy.InputField()
    label: str = dspy.OutputField()

def exact(example, pred, trace=None):
    return example.label == pred.label.strip().lower()

def score(program, data):
    return round(100 * sum(exact(x, program(commit=x.commit)) for x in data) / len(data), 1)

base = dspy.Predict(Classify)
results = {
    "baseline (no demos)": base,
    "LabeledFewShot k=8": dspy.LabeledFewShot(k=8).compile(base, trainset=trainset),
    "BootstrapFewShot 4+4": dspy.BootstrapFewShot(
        metric=exact, max_bootstrapped_demos=4, max_labeled_demos=4).compile(base, trainset=trainset),
    "RandomSearch 8 candidates": dspy.BootstrapFewShotWithRandomSearch(
        metric=exact, max_bootstrapped_demos=4, max_labeled_demos=4,
        num_candidate_programs=8, num_threads=1).compile(base, trainset=trainset, valset=devset),
}
for name, prog in results.items():
    print(f"{name:28} dev {score(prog, devset):5}  test {score(prog, testset):5}")

results["RandomSearch 8 candidates"].save("classify_v1.json")
print("sha256", hashlib.sha256(open("classify_v1.json", "rb").read()).hexdigest())

Step 3: the numbers

We ran this on 23 October 2026 with Python 3.14 and dspy 3.4.1 on macOS. Two consecutive runs produced identical output:

program                      dev%  test%
baseline (no demos)           40.0  28.5
LabeledFewShot k=8            47.5  54.5
BootstrapFewShot 4+4          50.0  40.2
RandomSearch 8 candidates     60.0  46.0
sha256 524db391f79b71550c7bf7c1c2cae9d42b4252f9a42046bfea73b3e3378851d3

Random search scored 11 candidate programs on the dev set (the 8 requested plus the zero-shot, labelled-only and unshuffled bootstrap baselines its docs describe) and returned the best. That program scored 60.0 percent on dev and 46.0 percent on 400 unseen examples. Plain LabeledFewShot, which never saw the dev set, scored lower on dev (47.5) and higher on test (54.5).

Step 4: read the numbers honestly

1. Selection on a small dev set is optimistic. With 40 examples, a 95 percent confidence interval around 60 percent is roughly plus or minus 15 points, so picking the maximum of 11 scores partly picks noise. With 400 test examples the interval is about plus or minus 5 points.
2. Bootstrapping inherits the teacher's blind spots. BootstrapFewShot keeps only traces the current program already gets right. Our stand-in's fallback can only say "bugfix" or "feature", so a weak teacher bootstraps examples of what it can already do.
3. The optimised program contains training data verbatim. The saved JSON is 902 bytes. Its demos list holds three bootstrapped traces (marked "augmented": true) and one labelled example, "add path handling in parser", labelled security. That is one of the deliberately noisy examples, and it now ships inside every prompt this program sends.

To use a real model, replace the dspy.configure line with dspy.configure(lm=dspy.LM("provider/model-name")) and supply credentials through environment variables. The numbers will change, and instruction optimisers such as MIPROv2 or GEPA only make sense with a model that reads instructions.

What overfitting and evaluation leakage risks come with optimising against a dev set?

Optimisers are search procedures, and search finds the weaknesses of whatever it is scored against. Six patterns matter.

1. The winner's curse on small dev sets

Keep the best of many candidates scored on the same small set and the best score is biased upwards; our run shows a 14 point dev to test gap. DSPy's advice to have "200 examples or more to prevent overfitting" for long MIPROv2 runs acknowledges this. OPRO's choice to skip validation by default was justified on two tasks; do not copy it as a default.

2. Overlapping splits

Bootstrapped demos are training examples. If one also appears in the test set, the program is tested on a question whose answer sits in its prompt. This happens when data is deduplicated after splitting, or when the same ticket appears twice with different IDs. Hash normalised inputs and check overlap across all three splits.

3. Benchmark contamination

Public benchmarks such as GSM8K and HotPotQA may sit in a model's pre-training data, and a prompt that unlocks memorised answers looks like a better prompt. Optimisers will find and exploit that. Our post on benchmark contamination concerns covers detection; for your own tasks, build part of the test set from data created after the model's training cutoff.

4. An LLM judge as the metric

When the metric is a model's opinion, the optimiser learns to please that model, and GEPA's written feedback and TextGrad's critique loss both rely on model judgements. Spot check judged outputs against human labels; see our notes on LLM-as-judge pitfalls.

5. Test-time search reported as a better prompt

TextGrad's GPQA gain refines each answer at test time, and GEPA's docs describe an inference-time search mode that sets the valset to the evaluation batch itself. Both are legitimate, but the result is per-query search, priced per query, not a reusable prompt.

6. Silent metric drift

If the metric changes next month to accept synonyms, every historical score becomes incomparable. Version the metric with the program and record which version produced each score.

NIST's Generative AI Profile (NIST AI 600-1, July 2024) frames this in suggested action MS-2.3-002: "Evaluate claims of model capabilities using empirically validated methods." It also warns that common benchmark test datasets "can contain label errors". An optimised prompt is a capability claim with a number attached. Our post on using an eval harness as a release gate for AI features shows how to wire a held-out score into deployment.

What changes in the artifact need versioning, review and provenance like code?

An optimisation run produces a file that changes how a production system behaves. Whatever the extension, that is code.

What a saved DSPy program contains

DSPy's saving and loading design notes are explicit. A state-only .json save writes, per predictor, traces, train examples, demos, the signature (instructions plus each field's prefix and description) and the LM configuration, plus a metadata block with Python, DSPy and cloudpickle versions. Several design decisions are security decisions:

• API keys are never serialised. LM.dump_state excludes api_key, with no flag to include it.
• Endpoints are stripped on load by default. api_base, base_url and model_list are dropped unless the loader passes allow_unsafe_lm_state=True, because a saved program "might point at an internal endpoint".
• Pickle requires explicit trust. .pkl saves and whole-program saves (save_program=True) use cloudpickle, and loading either requires allow_pickle=True, because "loading a pickle can execute arbitrary code".
• Version mismatches warn but do not block.

We read these notes on the DSPy main branch on 23 October 2026 and confirmed the allow_pickle and allow_unsafe_lm_state parameters, both defaulting to False, in the installed dspy 3.4.1 source. Check again for the version you pin.

A control map for optimised artifacts

Why demos deserve the most scrutiny

Demos look like data but are injected into every prompt the program sends. If someone can add examples to your training set, for instance through a feedback button that feeds a "good answers" collection, an optimiser may select their example as a demo and ship it to every user. That is a persistent prompt injection path through the build rather than through user input. We describe it conceptually only; the defence is the control map above and giving training data contributions the same permission model as code. The same principle applies to agents that write their own skills: Nous Research's Hermes Agent documents a skills.write_approval setting that defaults to false, so agent-written skills are saved without review unless you turn the gate on (Hermes skills docs, checked 23 October 2026).

Step by step: put an optimised program under change control

This assumes the program lives in a git repository with its metric, dataset manifests and a held-out test set.

1. Pin the optimiser and its dependencies

pip install pip-tools
echo "dspy==3.4.1" > requirements.in
pip-compile --generate-hashes requirements.in -o requirements.txt
pip install --require-hashes -r requirements.txt

2. Store the run's inputs alongside its output

mkdir -p optimised/classify
python run_optimiser.py --out optimised/classify/classify_v1.json   # your script from the example
sha256sum data/train.jsonl data/dev.jsonl data/test.jsonl > optimised/classify/datasets.sha256
git add optimised/classify requirements.txt metric.py run_optimiser.py
git commit -m "classify: RandomSearch 8 candidates, seed 7, dev 60.0, test 46.0"

Keep the optimiser's full log as a CI artifact.

3. Review the diff, not just the score

git diff HEAD~1 -- optimised/classify/classify_v1.json
python - <<'EOF'
import json
state = json.load(open("optimised/classify/classify_v1.json"))
for d in state["demos"]:
    print(d.get("augmented", False), "|", d["commit"], "->", d["label"])
print("instructions:", state["signature"]["instructions"])
EOF

A reviewer reads every demo and the full instruction and asks: is there data that should not be in a prompt, is any label wrong, and would the product owner sign off on the instruction?

4. Gate merges on a held-out score

# .github/workflows/prompt-eval.yml
name: prompt-eval
on: [pull_request]
permissions:
  contents: read
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install --require-hashes -r requirements.txt
      - run: sha256sum -c optimised/classify/datasets.sha256
      - run: python eval_heldout.py --program optimised/classify/classify_v1.json --min-test 0.50

eval_heldout.py loads the JSON into a freshly constructed program with program.load(path), scores the test set and exits non-zero below the threshold. The threshold applies to the held-out set, never to the dev set the optimiser saw. In our toy run this gate would have rejected the random search winner (46.0 percent) and accepted LabeledFewShot (54.5 percent).

5. Sign the artifact and attest where it came from

Sigstore's cosign signs arbitrary files and writes a bundle with the signature, certificate and transparency log proof:

cosign sign-blob optimised/classify/classify_v1.json --bundle classify_v1.sigstore.json

In GitHub Actions, actions/attest (which attest-build-provenance now wraps, per its README) binds the file's digest to SLSA build provenance. The job needs id-token: write and attestations: write:

      - uses: actions/attest@v4
        with:
          subject-path: optimised/classify/classify_v1.json

At deploy time, verify with gh attestation verify or cosign verify-blob before loading. For pickled programs, this check is the only thing between allow_pickle=True and code execution from a tampered file.

6. Roll back with a backup first

Reverting changes what production loads, so copy the deployed artifact before you revert:

cp optimised/classify/classify_v1.json /tmp/classify_v1.deployed.json   # backup first
git revert <commit-of-the-bad-optimisation>

git revert adds a commit rather than rewriting history, which keeps the record of what shipped.

Are these systems "self-improving"?

DSPy's paper title says "self-improving pipelines" and Promptbreeder's says "self-referential self-improvement". In both, the system improves its own prompts or demos by running itself and keeping what scores well. Weights stay frozen (unless you choose BootstrapFinetune), a human supplies the metric, the budget is fixed and the search stops. None of these papers claims open-ended capability growth, and none shows it. The honest summary: LLMs are decent proposal generators for discrete search over text, and a good metric plus a modest budget often beats an afternoon of manual prompt tweaking.

Common mistakes

• Reporting the dev score. The optimiser chose the program because of it. Report the held-out score with its sample size.
• Comparing optimisers on 20 to 40 examples. The confidence intervals are too wide to rank anything.
• Letting test examples leak into demos. Check overlap after normalisation, not just by ID.
• Treating program JSON as configuration that needs no review. Every request carries its instructions and examples.
• Loading pickles from shared storage. Use JSON, or verify a signature first.
• Forgetting the metric is code. Version it with the program.
• Assuming gains transfer across models. Re-run on the model you will deploy.
• Ignoring cost. Population-based search and test-time refinement multiply model calls; estimate first, as the OPRO README warns.

FAQ

What is DSPy used for?

DSPy is a Python framework for building LLM pipelines from declarative modules and optimising the prompts, few-shot examples or weights behind them against a metric. Teams use it for retrieval and answer systems, classifiers, extraction and agent loops, especially when they expect to change models.

Does DSPy fine-tune the model?

Only if you ask. Most optimisers change demos and instructions. BootstrapFinetune distils a prompted program into fine-tuned weights, and BetterTogether alternates prompt and weight optimisation.

What is the difference between OPRO and Promptbreeder?

OPRO asks one optimiser model for a better instruction given past instructions and their scores. Promptbreeder evolves a population of prompts with a genetic algorithm and nine mutation operators, including one that rewrites the mutation instructions themselves. OPRO is simpler per step; Promptbreeder spends more evaluations to keep diversity.

Is TextGrad the same as backpropagation?

No. It borrows the structure and API style. Backpropagation computes exact derivatives; TextGrad asks an LLM to write feedback for each upstream variable and another call to edit it. The "gradient" is advice, with no guarantee it points in an improving direction.

Which prompt optimiser should I use?

Whatever your team can operate and evaluate properly. If you use DSPy, its docs suggest BootstrapFewShot for about 10 examples, random search for 50 or more, and MIPROv2 for longer runs with around 200. The paper results here used different models and benchmarks, so none of them ranks the methods for your task.

Can prompt optimisation overfit?

Yes. Any search that keeps the best of many candidates on a fixed set overfits to it. Our run shows a dev winner at 60.0 percent scoring 46.0 percent on test. Hold out a test set the optimiser never sees.

Is it safe to load a saved DSPy program?

A JSON state file loads without executing code, never contains API keys, and by default drops endpoint settings, per the DSPy docs. Pickled programs can execute code on load, which is why DSPy requires allow_pickle=True. Only load pickles you produced or whose signature you have verified.

Does Safeguard optimise prompts?

No. Safeguard does not build prompt optimisers. Its role is the software supply chain around them: the packages a pipeline installs, the repositories holding compiled artifacts, and the provenance and policy checks that decide what ships.

Where Safeguard fits

The fit is indirect, and we would rather say so than force it. Safeguard does not evaluate prompts, score optimisers or inspect demo content. A prompt optimisation pipeline does produce ordinary supply chain material: a Python environment with dspy and its transitive dependencies, a repository of JSON or pickle artifacts, CI jobs that build and sign them, and a service that loads them. Safeguard generates and analyses SBOMs for those environments and repositories in SPDX and CycloneDX, matches components against vulnerability data, applies policies and gates before a build ships, and supports attestation and signing with SLSA, Sigstore and in-toto. Its MCP server, with hundreds of tools, can be added to an assistant such as Claude from the assistant's connector directory. AI-BOM discovery, which inventories AI components, is in early access.

An optimisation pipeline is a Python project like any other, pulling in a framework, a model client, a serialisation library and dozens of transitive packages, any of which can receive a CVE after you pin it. The compiled program only runs inside that environment, so patching it is part of keeping the optimised system safe; the fork and patch option below is in early access.

Fix it without waiting for upstream

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation covers eight categories, including dependencies, licenses, security posture and findings. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds. The safeguard-sh/gold-scan-action GitHub Action can fail a build on a known exploited vulnerability, on any known vulnerability, or never, so a version you have just removed cannot quietly come back in next week's merge.

3. Gold Registry: start from components that are already fixed

registry.safeguard.sh provides hardened, drop in replacements for upstream packages and container images. Every Gold artifact is held continuously to zero exploitable critical or high CVEs, zero malware and a completed license check, with SLSA L3 or higher provenance, a Sigstore signature and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. This fork and patch flow is in early access today, so ask which packages it covers before you plan around it.

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

None of these demos optimise a prompt or score an LLM. They show how Safeguard handles the dependencies and repositories an optimisation pipeline lives in.

The first demo connects Claude to the Safeguard MCP server with OAuth and lets you choose which tools Claude can use. In practice you add Safeguard from Claude's connector directory rather than writing an MCP configuration by hand, then ask which projects carry a given version of a Python package your DSPy environment depends on.

Click through it at your own pace, or open it full screen.

The second demo is a real Safeguard CLI session: asked to fix a lodash CVE, it reads the finding and advisory, asks before each command, edits package.json to lodash 4.17.21, runs npm install and npm test with approval, prints a summary and pushes nothing. The same approve-each-step flow applies when a pinned package in an optimisation pipeline's requirements needs an upgrade before you rerun the held-out evaluation.

Click through it at your own pace, or open it full screen.

The third demo sets up Safeguard's local runner on your own machine, approves it with its device code, and runs a scan from the terminal. Scanning the repository that holds your compiled programs gives you a dependency inventory to keep next to each optimisation run.

Click through it at your own pace, or open it full screen.

To try it on your own projects, download Safeguard or read the CLI documentation.

Quick check

Sources

• Khattab et al., "DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines," arXiv 2310.03714
• Yang et al., "Large Language Models as Optimizers," arXiv 2309.03409
• Fernando et al., "Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution," arXiv 2309.16797
• Promptbreeder in the Proceedings of ICML 2024 (PMLR 235)
• Yuksekgonul et al., "TextGrad: Automatic 'Differentiation' via Text," arXiv 2406.07496
• Yuksekgonul et al., "Optimizing generative AI by backpropagating language model feedback," Nature 639, 2025
• Opsahl-Ong et al., "Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs" (MIPRO), arXiv 2406.11695
• Agrawal et al., "GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning," arXiv 2507.19457
• Zhou et al., "Large Language Models Are Human-Level Prompt Engineers" (APE), arXiv 2211.01910
• DSPy repository (stanfordnlp/dspy)
• DSPy docs: Optimizers
• DSPy docs source: Saving and loading design notes
• DSPy docs: MIPROv2
• DSPy docs: GEPA overview
• PyPI: dspy
• OPRO repository (google-deepmind/opro)
• TextGrad repository (zou-group/textgrad)
• PyPI: textgrad
• NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
• Sigstore docs: Signing blobs with cosign
• actions/attest-build-provenance README
• SLSA v1.0 provenance specification
• Hermes Agent docs: Skills

Last reviewed: 24 October 2026.