Self-Training Language Models: STaR, Self-Rewarding and Self-Play

A self-training language model fine-tunes on outputs it generated itself, after a filter decides which ones to keep. STaR keeps reasoning that reached a known correct answer, Self-Rewarding models grade their own answers, and self-play trains a model against earlier copies of itself. The filter, not the model, decides whether the loop learns or decays.

Every method below has three parts: a generator (the model), a selector (an answer key, a test, a game result or a judge) and an update (fine-tuning or reinforcement learning). When the selector is an external, checkable signal, published results are strong. When it is the model's own opinion, the authors themselves flag length inflation, reward hacking and a ceiling set by the human data. We go paper by paper, then cover model collapse, reward hacking, Nous Research's Atropos and Hermes Agent trajectory tooling, and a checklist for training on agent-generated data.

Key takeaways

• Self-training changes weights. That differs from an agent editing its own memory or skill files, which is what Hermes Agent's learning loop does. Its batch runner and Nous's Atropos framework produce data someone may later train on.
• STaR (arXiv 2203.14465, March 2022) fine-tuned a 6B model on its own rationales that reached the correct answer, and reported 72.5% on CommonsenseQA against 60.0% for direct fine-tuning. Its filter is an answer key, so a right answer reached by bad reasoning still passes.
• Self-Rewarding Language Models (arXiv 2401.10020, January 2024) made one Llama 2 70B model its own judge for three rounds of DPO. AlpacaEval 2.0 win rate rose from 9.94% to 20.44%, while responses more than doubled in length, which the authors say "may be a factor".
• Self-play works best with a rule that cannot be argued with. AlphaGo Zero learned Go from game outcomes alone. SPIN (ICML 2024) adapts the idea to language, and its own theory caps it at the quality of its human data.
• Verifiable rewards beat self-judgement when available. ReST-EM, Tulu 3's RLVR and DeepSeek-R1 use checkers or tests. DeepSeek-R1's authors avoided neural reward models because of reward hacking.
• Model collapse is a data management problem, not a law of nature. The Nature 2024 paper showed distribution tails vanishing when each generation trains on the last one's output; follow-up work avoided it by accumulating synthetic data alongside real data.
• Agent-generated data is a supply chain input. Record each sample's model, verifier, licence and source, check overlap with your evaluation sets, and strip untrusted tool content.

What "self-training" means, and what it does not

"Self-improving" covers at least five things, best separated by what changes:

This post is about the last row. The other rows change what a deployed agent sees or runs; the last changes the model, so a filter mistake is copied into every use of the weights.

Hermes Agent, the open source agent from Nous Research, does not fine-tune its own model during normal use. Its skills docs say that by default "the agent writes skills freely", and skills.write_approval: true stages every skill write for human approval. That is learning in context. Its batch runner, covered below, is where it touches self-training. For the agent side of self-improvement (reflection, skill libraries and memory), see our post on AI agent memory security risks.

The papers covered

Dates are the first arXiv version unless a journal is named. "What changes" names the thing the method updates. "Evaluator" is what decides which samples count as good during training. Results are given in the authors' own terms and on the authors' own benchmarks.

The strongest headline numbers (AlphaGo Zero, DeepSeek-R1, RLVR) come with evaluators the model cannot talk its way past. Almost every paper uses few benchmarks, one or a few base models and few iterations. None shows open-ended improvement, and none claims to.

How STaR bootstraps reasoning from its own rationales

STaR, the Self-Taught Reasoner, comes from Eric Zelikman, Yuhuai Wu, Jesse Mu and Noah Goodman (Stanford, with Google Research). The loop in the paper's own description is: generate rationales for many questions using a few worked examples as a prompt; if an answer is wrong, try again with the correct answer given as a hint; fine-tune on every rationale that "ultimately yielded correct answers"; repeat.

The filter that keeps it honest

A rationale enters the training set only if its final answer matches the dataset's label, a cheap external check the model cannot influence.

The second ingredient is what the authors call rationalization. For missed problems the model sees the correct answer as a hint and writes a rationale reaching it; the hint is removed before training. The authors say rationalization "accelerates and improves the bootstrapping process".

What STaR reported

The base model was GPT-J, a 6B parameter open model, fine-tuned with the GPT-J repository's own script. The acknowledgements thank Google's TPU Research Cloud for compute. On CommonsenseQA, a five-way multiple choice benchmark:

The authors compare 72.5% to 73.0% for a directly fine-tuned model about 30 times larger. On GSM8K grade school maths, STaR with rationalization reached 10.7% against 5.8% for direct fine-tuning, training on 28.7% of the dataset.

What STaR did not show, in the authors' words

The limitations section is unusually direct. First iteration success needs "few-shot performance must be above chance", and the authors report that GPT-2 "was not able to bootstrap from few-shot reasoning in even the arithmetic domain." Second, "settings with a high level of chance performance (e.g. binary decisions) yield many poor rationales", and how to filter bad reasoning there is called "an open problem." CommonsenseQA itself has five options, so about one answer in five is right by chance. A right answer reached through a wrong argument passes the filter.

An appendix on faithfulness adds that the written rationale need not be the computation that produced the answer. An answer key checks answers, not reasoning; if you depend on the reasoning, add a second check.

What Self-Rewarding Language Models are, and what iterating the judge showed

Self-Rewarding Language Models, by Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu and Jason Weston (Meta and NYU), removes the separate reward model. The model being trained also scores its own candidate answers.

How the loop works

1. Seed data. Start from Llama 2 70B. Fine-tune on 3,200 human-written instruction examples from the Open Assistant dataset (instruction fine-tuning, IFT) plus 1,630 examples of how to evaluate a response (evaluation fine-tuning, EFT), built from Open Assistant's ranked human responses. This gives model M1.
2. Self-instruction creation. Generate new prompts, sample several responses from the current model, and have the same model score each on an additive 0 to 5 LLM-as-a-Judge scale.
3. Preference pairs. Take the highest and lowest scored responses for each prompt as a winner and loser pair, discarding ties.
4. Update. Train the next model with Direct Preference Optimization (DPO) on those pairs. Repeat.

What it reported

The judge column is the interesting claim: agreement with human rankings rose with no new evaluation data after M1.

What it did not show

The authors say they "only ran three iterations in a single setting", observed longer generations with "a known correlation between length and estimated quality", and want to understand "if so-called 'reward-hacking' can happen within our framework." Model-judged evaluation and safety are listed as needing more work.

The length column is the warning sign: when a model judges itself and the benchmark is also model-judged, longer answers can win both without being better.

What self-play is, from AlphaGo Zero to SPIN

AlphaGo Zero: a perfect referee

AlphaGo Zero (Silver and colleagues, DeepMind, Nature 550, October 2017) learned Go "tabula rasa", from games it played against itself, with no human game records. A network guided a tree search and was trained to imitate the search and predict the winner. The paper reports that it won 100 to 0 against the previously published AlphaGo that had beaten a world champion. An extended data figure describes a larger 40 block network trained for 40 days.

The reason this works is that Go has a referee nobody can fool. The rules decide who won. Self-play supplies opponents at the right difficulty, and the outcome signal is always correct.

SPIN: self-play without a referee

Self-Play Fine-Tuning (SPIN), by Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji and Quanquan Gu (UCLA), published at ICML 2024, borrows the shape of self-play for language. The "opponent" is the previous iteration of the model. At each round the model learns to prefer the human-written response in a supervised dataset over its own previous generation for the same prompt.

Starting from zephyr-7b-sft-full (Mistral-7B fine-tuned on UltraChat200k) and 50k UltraChat200k prompts, the authors report the average score on the HuggingFace Open LLM Leaderboard rising from 58.14 to 63.16 across iterations, and say SPIN outperformed models trained with DPO plus extra preference data from a stronger model.

Why self-play works best with a verifiable signal

SPIN's own limitation section answers this sub-question. The authors prove the objective is optimised only when the model's distribution matches the human data, so "our study focuses on a fixed target data distribution generated by humans, which inherently imposes a ceiling on the performance." In other words, SPIN is a very data-efficient way to absorb an existing human dataset. It is not a way to go beyond it.

Self-play improves beyond its starting data only with a signal that is correct and hard to game. Games have one, code with tests and checkable maths have a weaker one, open-ended chat has none.

Verifiable rewards versus learning from a model's own preferences

The last three years of results split cleanly into two families.

Family one: verifiable rewards

ReST-EM ("Beyond Human Data", arXiv 2312.06585) generates samples, filters them with "binary feedback", fine-tunes on the survivors and repeats a few times. On MATH and APPS with PaLM 2 models, the authors report it "scales favorably with model size and significantly surpasses fine-tuning only on human data."

Tulu 3 (arXiv 2411.15124, from Ai2) named the approach Reinforcement Learning with Verifiable Rewards (RLVR): "it obtains a constant reward value if a completion is successful." The paper sets that value at α = 10 "based on pilot experiments", gives 0 otherwise, and trains with PPO. The authors report RLVR improved GSM8K, MATH and IFEval, the last being instruction following with constraints a program can check. The paper also stresses "substantial decontamination" of open datasets against its benchmarks.

DeepSeek-R1 (arXiv 2501.12948, later Nature 645) trained its R1-Zero variant with a "rule-based reward system" of two parts: accuracy rewards (a final answer in a fixed format that a rule can check, or code compiled and run against test cases) and format rewards (reasoning placed between think tags). The authors write that they did not use an outcome or process neural reward model "because we find that the neural reward model may suffer from reward hacking in the large-scale reinforcement learning process." Version 1 of the paper reports R1-Zero's pass@1 on AIME 2024 rising from 15.6% to 71.0% during RL. The paper places its gains on "verifiable tasks."

Family two: the model's own preferences

Constitutional AI (Anthropic, arXiv 2212.08073) is the clearest early example. In its supervised phase the model critiques and revises its own answers against a written list of principles, and the original model is fine-tuned on the revisions. In its RL phase a model compares pairs of samples, a preference model is trained on those AI preferences, and the policy is trained with RL against it. Anthropic calls this "RL from AI Feedback" (RLAIF). The only human oversight is the list of principles.

Self-Rewarding Language Models, covered above, removes the separate preference model entirely and lets the policy judge itself.

How the two families differ in practice

Use a verifiable signal wherever one exists and treat a model judge as a weaker signal that needs an independent check. The verifier then becomes the most security-sensitive file in your pipeline: whoever can edit the tests or answer key decides what the model learns.

What model collapse is, and the conditions that avoid it

Our longer piece is model collapse myth versus fact.

The Nature paper ("AI models collapse when trained on recursively generated data", Shumailov, Shumaylov, Zhao, Papernot, Anderson and Gal, Nature 631, published 24 July 2024, first posted as arXiv 2305.17493 in May 2023) argues that "indiscriminate use of model-generated content in training causes irreversible defects", in which "tails of the original content distribution disappear." It shows the effect in variational autoencoders, Gaussian mixture models and language models.

The language model experiment is small: OPT-125m fine-tuned on wikitext2, each generation trained on text from the previous one and scored on the original wikitext2 test set. Two regimes:

• Five epochs, no original data retained. Performance degraded, "from 20 to 28 perplexity points" in their description, and low perplexity samples accumulated over generations, the signature of the tails being lost.
• Ten epochs, 10% of original data preserved each generation. The authors report that "preservation of the original data allows for better model fine-tuning and leads to only minor degradation of performance."

Gerstgrasser and colleagues ("Is Model Collapse Inevitable?", arXiv 2404.01413) then asked what happens when data accumulates rather than being replaced. They confirmed that replacing real data with each generation's synthetic data "does indeed tend towards model collapse", and showed that "accumulating the successive generations of synthetic data alongside the original real data avoids model collapse" across their model sizes and architectures, with a bounded test error in an analytic linear model.

What this means for a team doing self-training:

• Never replace your human or ground-truth data with generated data. Add to it.
• Filter generated data with an external signal; collapse experiments train on unfiltered output.
• Track which samples are synthetic and from which generation, and cap the mix.
• Watch the tails, since averages can hold while rare cases vanish.

The Nature authors make the provenance point themselves: the need to tell LLM-generated data from other data "raises questions about the provenance of content."

How reward hacking shows up when the judge is the model itself

Reward hacking is the model getting a high score from the evaluator without doing what the evaluator was meant to measure. It is Goodhart's law applied to training.

The measured version. "Scaling Laws for Reward Model Overoptimization" (arXiv 2210.10760) let a fixed "gold-standard" reward model stand in for humans and trained a proxy from its labels. Optimising the proxy by RL or best-of-n first raises, then lowers, the gold score, with curves that scale smoothly with reward model size.

Capability makes it worse. "The Effects of Reward Misspecification" (arXiv 2201.03544) built four RL environments with misspecified rewards and found that "more capable agents often exploit reward misspecifications, achieving higher proxy reward and lower true reward", sometimes through sudden "phase transitions" in behaviour. That matters for self-rewarding loops because each iteration makes the policy more capable against a judge that is improving more slowly.

Generalisation from small gaming to larger gaming. "Sycophancy to Subterfuge" (arXiv 2406.10162) trained LLM assistants on a curriculum of increasingly gameable environments. Training on the early, simple forms of gaming increased gaming later in the curriculum, and "a small but non-negligible proportion of the time" models trained on the full curriculum rewrote their own reward function in a held-out environment. Retraining against early gaming reduced but did not remove it, and adding harmlessness training did not prevent it. The environments were built to be gameable, so this does not show production models doing it; it shows a training signal can teach a habit that generalises.

What it looks like with a self-judge. In Self-Rewarding Language Models the visible symptom was length, which the authors could not separate from the win rate gain. Look for:

• Response length, list count or hedging growing faster than accuracy on a held-out checked set.
• Judge scores rising while human spot checks stay flat.
• The model adopting phrasing that appears in the judge prompt.

Defences, strongest first: a verifiable reward where possible; a judge kept separate from the policy; a frozen held-out evaluation set; length-controlled comparisons; a KL penalty to a reference model; and human review of a random sample each iteration. Our post on LLM-as-a-judge pitfalls goes further on the judge side.

How RL environment frameworks and agent trajectories feed the loop

Agents add tools, multi-step trajectories and environments that score whole episodes. Nous Research has open sourced plumbing for both. All Hermes and Atropos facts below were checked on the projects' own repositories and docs on 23 October 2026.

Atropos: environments as services

Atropos (github.com/NousResearch/atropos, MIT licence, repository created 29 April 2025 per the GitHub API) describes itself as "an environment microservice framework for async RL with LLMs." Environments run as services, a trajectory API collects their scored rollouts, and a trainer pulls batches from that API. The trainer and inference engine are not included. Listed environment types include datasets (GSM8K, MMLU), games, RLAIF and RLHF judges, multi-turn tool calling and code execution (MBPP, HumanEval).

The README reports self-measured results, such as a tool calling environment taking a model from 10% to 46% on the Berkeley Function Calling Benchmark's parallel tasks. Those are Nous's own numbers.

Atropos also produces offline data. Its atropos-sft-gen and atropos-dpo-gen commands collect rollouts and convert them into supervised fine-tuning or DPO datasets, with rejection sampling controls (keep the top n per group, allow or drop negative scores, require a minimum score spread). The environment's scoring function is the filter, as in STaR and ReST-EM.

Hermes 4: rejection sampling against about a thousand verifiers

The Hermes 4 Technical Report (arXiv 2508.18255, August 2025) shows how Nous used Atropos on its own models. The authors "rejection sample against roughly a thousand task-specific verifiers using Atropos", which "creates a large corpus of verified reasoning trajectories." The final dataset is about 5 million samples and 19 billion tokens (3.5 million reasoning and 1.6 million non-reasoning), and training ran on 192 NVIDIA B200 GPUs.

Two environments illustrate verifier design:

• An answer format environment gives a binary reward "1.0 for a valid format and 0.0 for an invalid one", and the report says "the correctness of the responses does not factor in the reward." That is deliberate, and a reminder that a reward valid for one purpose is useless for another.
• An instruction following environment uses Tulu 3's program-checkable RLVR-IFEval tasks and keeps only successful trajectories.

Hermes Agent: batch trajectories

Hermes Agent's README lists "Batch trajectory generation, trajectory compression for training the next generation of tool-calling models" as a feature. Its batch_runner.py runs the full agent with tools over a JSONL file of prompts, gives each prompt "its own isolated environment" (optionally a named container image), and writes ShareGPT-format trajectories with tool statistics. Output lands under data/<run_name>/ as trajectories.jsonl, per-batch files, a checkpoint and statistics.json. Outside the batch runner, trajectory saving is a library-level switch (save_trajectories=True), and the docs say the hermes CLI does not expose a config key for it. The release we checked against was v0.22.5, dated 22 October 2026 in its release notes.

Read the quality filtering section closely. The batch runner discards samples with no reasoning and drops entries with hallucinated tool names. Neither checks whether the answer was right, and completed: true means the run finished, not that the task succeeded. Raw batch output is closer to the unfiltered regime of the collapse experiments than to STaR. Add a task-specific verifier, as Hermes 4 did with Atropos.

For provenance, note that the batch runner can use a system prompt "used during execution but NOT saved to trajectories", so record it yourself. Trajectories also include raw tool output (terminal, files, web pages), which is where secrets, personal data and untrusted web text enter your training set.

What to check before training on agent-generated data

Treat a trajectory dataset like any third-party component entering your build. OWASP's LLM04:2025 Data and Model Poisoning entry recommends that teams "track data origins and transformations using tools like OWASP CycloneDX or ML-BOM" and "verify data legitimacy during all model development stages." Concretely:

Licence

• The data's licence. Record the licence of every source prompt dataset with the derived data.
• The generating model's terms. Read the generating model's licence or terms for limits on training other models with its outputs, and record its exact identifier and version.
• The tooling. Atropos and Hermes Agent are both MIT licensed per their repositories.

Provenance

For every sample, record: the generating model and version, the prompt source and its hash, the environment or container image digest, the verifier name and version, the verifier's score, the iteration number, and a content hash of the sample. Without it you cannot say what iteration 3 learned from, or enforce the accumulate-not-replace rule.

Contamination

If evaluation prompts appear in generated training data, benchmark gains are not real. Like Tulu 3, hash or n-gram match generated samples against every evaluation set you report and drop overlaps.

Poisoning and untrusted content

Trajectories contain whatever the agent read, including planted instructions in a page or tool response if the agent still finished the task. Training on that can teach the model to follow such text. We describe the attack conceptually in data poisoning attacks against LLMs and the pipeline defences in data poisoning training pipeline defenses. For trajectories specifically: strip or quarantine tool outputs from untrusted sources, or train only on the assistant turns with tool content masked from the loss.

Secrets and personal data

Tool output can contain keys, tokens, hostnames and personal data. Redact every tool turn before data leaves the generation environment, and review a sample by hand.

The verifier is code

Verifiers, tests, answer keys and judge prompts decide what the model learns, so they need version control, review and signed releases, as do the training environment's dependencies. See our post on training data provenance regulation.

Step by step: a small, reproducible self-training loop with a verifier

This example runs on a laptop with no GPU or model download. A noisy stand-in "model" on toy arithmetic shows each part of STaR-style self-training: generate, verify, rationalize with a hint, keep verified samples, accumulate, decontaminate and record provenance. Swap generate() for your own model or an Atropos environment later.

Step 1: pin the environment

python3 -m venv .venv && . .venv/bin/activate
pip install pip-tools==7.6.2
cat > requirements.in <<'EOF'
cyclonedx-bom==7.5.0
EOF
pip-compile --generate-hashes requirements.in -o requirements.txt
pip install --require-hashes -r requirements.txt

Hash-pinned installs make the environment rebuildable. Pin Atropos the same way if you add it (PyPI showed atroposlib 0.4.0 on 23 October 2026).

Step 2: the self-training loop

# star_toy.py: STaR-style self-training on a toy task, with provenance.
import hashlib, json, random

random.seed(7)
MODEL_ID = "toy-noisy-adder-v1"
VERIFIER_ID = "exact-int-match-v1"

def make_problems(n, lo, hi):
    return [{"q": f"{a}+{b}", "answer": a + b}
            for a, b in ((random.randint(lo, hi), random.randint(lo, hi)) for _ in range(n))]

def generate(question, hint=None, skill=0.6):
    """Stand-in model: right with probability `skill`, always right when given a hint."""
    a, b = map(int, question.split("+"))
    if hint is not None or random.random() < skill:
        ans = a + b
    else:
        ans = a + b + random.choice([-10, -1, 1, 10])
    rationale = f"Add the ones, carry, then add the tens: {a} + {b} = {ans}."
    return rationale, ans

def verify(sample, answer):
    return sample["pred"] == answer          # the filter: an external answer key

def sha(obj):
    return hashlib.sha256(json.dumps(obj, sort_keys=True).encode()).hexdigest()

human_data = [{"q": p["q"], "rationale": f"{p['q']} = {p['answer']}.", "pred": p["answer"],
               "source": "human"} for p in make_problems(200, 10, 99)]
eval_set = make_problems(100, 10, 99)
eval_qs = {p["q"] for p in eval_set}
train_pool = [p for p in make_problems(1000, 10, 99) if p["q"] not in eval_qs]

dataset = list(human_data)                    # accumulate, never replace
for iteration in range(1, 4):
    kept, rationalized, rejected = [], 0, 0
    for p in train_pool:
        rationale, pred = generate(p["q"])
        s = {"q": p["q"], "rationale": rationale, "pred": pred}
        if not verify(s, p["answer"]):
            rationale, pred = generate(p["q"], hint=p["answer"])   # rationalization
            s = {"q": p["q"], "rationale": rationale, "pred": pred, "rationalized": True}
            rationalized += 1
            if not verify(s, p["answer"]):
                rejected += 1
                continue
        s.update({"source": "synthetic", "iteration": iteration, "model": MODEL_ID,
                  "verifier": VERIFIER_ID})
        s["sha256"] = sha(s)
        kept.append(s)
    dataset += kept
    synthetic_share = sum(d["source"] == "synthetic" for d in dataset) / len(dataset)
    print(f"iter {iteration}: kept {len(kept)}, rationalized {rationalized}, "
          f"rejected {rejected}, synthetic share {synthetic_share:.2f}")

leaks = [d for d in dataset if d["q"] in eval_qs]
assert not leaks, "evaluation questions leaked into training data"

with open("train.jsonl", "w") as f:
    for d in dataset:
        f.write(json.dumps(d) + "\n")
manifest = {"model": MODEL_ID, "verifier": VERIFIER_ID, "samples": len(dataset),
            "human": sum(d["source"] == "human" for d in dataset),
            "train_sha256": hashlib.sha256(open("train.jsonl", "rb").read()).hexdigest()}
json.dump(manifest, open("manifest.json", "w"), indent=2)
print(json.dumps(manifest, indent=2))

Run python3 star_toy.py. Each iteration keeps verified samples and rationalizes misses. The synthetic share climbs past 0.9 by the third round, which is exactly why you should cap it and sample human data back in. The script also refuses to write a dataset containing evaluation questions. A real pipeline fine-tunes between iterations; the toy keeps the model fixed so the bookkeeping is readable. Try setting skill near 0.2, where rationalization does most of the work (the regime STaR's authors warn about), or removing the hint path to get plain rejection sampling.

Step 3: gate agent trajectories before they reach training

For Hermes Agent batch output, add your own verifier and provenance on top of the built-in filters:

# gate_trajectories.py: keep only verified, provenance-tagged trajectories.
import hashlib, json, re, sys

src, dst, run_meta = sys.argv[1], sys.argv[2], json.load(open(sys.argv[3]))
REDACT = re.compile(r"(?i)(api[_-]?key|token|secret|password)\s*[:=]\s*\S+")

def task_verifier(entry):
    """Replace with a real check: run tests, compare to an answer key, diff an output file."""
    return entry.get("completed") is True and entry.get("partial") is False

kept = dropped = 0
with open(src) as fin, open(dst, "w") as fout:
    for line in fin:
        e = json.loads(line)
        if not task_verifier(e):
            dropped += 1
            continue
        for turn in e["conversations"]:
            if turn["from"] == "tool":
                turn["value"] = REDACT.sub(r"\1=[REDACTED]", turn["value"])
        e["provenance"] = {**run_meta, "sha256": hashlib.sha256(line.encode()).hexdigest()}
        fout.write(json.dumps(e) + "\n")
        kept += 1
print(f"kept {kept}, dropped {dropped}")

run_meta is a JSON file you write before the run: model identifier, batch runner commit, hidden system prompt hash, image digests and verifier version. The task_verifier shown only checks completion flags, the weakest acceptable check; replace it with one that knows whether the task was done right. The redaction regex is a floor; review samples by hand.

Step 4: record the environment and sign the dataset

cyclonedx-py environment > train-env.cdx.json      # CycloneDX SBOM of the training venv
sha256sum train.jsonl manifest.json train-env.cdx.json > SHA256SUMS
cosign sign-blob --bundle train.sigstore.json train.jsonl
cosign verify-blob train.jsonl --bundle train.sigstore.json \
  --certificate-identity you@example.com \
  --certificate-oidc-issuer https://accounts.google.com

The SBOM records which libraries built the dataset, the checksums tie manifest to data, and the Sigstore bundle lets anyone downstream check who produced the file. Commit manifest, SBOM, checksums and bundle next to the training config so the dataset version is reviewed with the code that consumes it.

Step 5: evaluate against a frozen, external check

After each iteration, score the new model on the frozen evaluation set with the external verifier, not its own judgement. Track checked accuracy, average output length and a small human-reviewed sample. Stop when checked accuracy stops rising, even if judge scores climb.

Choosing an approach: a decision table

Where Safeguard fits, honestly

Safeguard does not train language models or judge their outputs, and it is not connected to Hermes Agent or Atropos. The overlap is the software around the pipeline: the trainer, environments, verifiers and trajectory-generating agent all have open source dependency trees, and batch agents install packages and write scripts. Safeguard generates and analyses SBOMs in SPDX and CycloneDX for those codebases and container images, supports attestation and signing with SLSA, Sigstore and in-toto for the artifacts you build, evaluates policies and gates in CI so a vulnerable dependency does not reach the training environment, and can open fix pull requests through autonomous remediation, which ships switched off by default. AI-BOM discovery, which inventories AI assets across code and cloud, is in early access. None of this checks whether a training sample is correct; that is your verifier's job.

Common mistakes

• Treating "completed" trajectories as correct. Completion means the episode ended. Add a task verifier.
• Self-grading with no external check. Keep a frozen, checkable evaluation set neither judge nor policy trains on.
• Replacing real data with synthetic data. Accumulate and cap the synthetic share.
• Ignoring length. Longer outputs that only a model judge prefers suggest reward hacking.
• Unversioned verifiers. A silently changed test suite changes what the model learns.
• No decontamination. Check overlap against every evaluation you report.
• Keeping raw tool output. It carries secrets and untrusted instructions into training.
• Unpinned training environments. Pin with hashes and keep an SBOM.

FAQ

What is self-training in a language model?

Fine-tuning a model on data it generated itself, after a filter keeps the good outputs. The filter can be an answer key (STaR), tests (ReST-EM, RLVR), the model's own judgement (Self-Rewarding) or human data (SPIN).

How does STaR work?

It generates rationales, keeps those whose answers match the ground truth, rationalizes the misses using the correct answer as a hint, fine-tunes on everything kept and repeats. It reported 72.5% on CommonsenseQA with a 6B model against 60.0% for direct fine-tuning.

Can a language model improve by judging its own answers?

In the Self-Rewarding paper it did, for three iterations on one 70B model, by AlpacaEval 2.0. Responses also grew much longer, and the authors say reward hacking needs more study. A self-judge is weaker than a checker.

What is self-play fine-tuning (SPIN)?

SPIN trains a model to prefer human-written responses over its own previous generations. It lifted a 7B model's Open LLM Leaderboard average from 58.14 to 63.16 and converges at the quality of its human data.

What is RLVR, reinforcement learning with verifiable rewards?

A fixed reward given only when a program verifies the answer, such as a matching maths answer or a satisfied constraint. Tulu 3 named it; DeepSeek-R1 used rule-based rewards because a checker is harder to game than a learned reward model.

Does training on AI-generated data cause model collapse?

It can when generated data replaces real data across generations, as the Nature 2024 paper showed. Keeping original data reduced the damage, and later work found accumulating synthetic with real data avoided collapse.

Does Hermes Agent train its own model?

Not during normal use. It writes memory and skill files into its context. Its batch runner generates trajectories for training, and Nous Research's Atropos provides RL environments, but training on that data is a deliberate, separate step.

What is Atropos from Nous Research?

An MIT licensed framework that runs LLM reinforcement learning environments as services, collects scored trajectories and hands batches to a trainer, or writes rejection-sampled SFT and DPO datasets. Nous used it with about a thousand verifiers for Hermes 4.

What is reward hacking in LLM training?

A model raising its evaluator score without doing the task better, such as writing longer answers a judge prefers. Research shows more capable models exploit misspecified rewards more often.

What should I check before fine-tuning on agent trajectories?

That each trajectory actually succeeded, its provenance (model, prompt, environment, verifier, hashes), licences, overlap with evaluation sets, secrets and personal data in tool output, and untrusted tool content in the loss.

Fix it without waiting for upstream

A self-training pipeline is built from open source components: the trainer, the RL environments, the verifiers, and the packages and scripts that agents install and write while generating trajectories, all of which receive CVEs like any other dependency. Keeping that software patched and pinned matters as much as filtering the data, because a compromised or vulnerable dependency in the training environment sits upstream of every model you produce.

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation covers eight categories, including dependencies, licenses, security posture and findings. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds. The safeguard-sh/gold-scan-action GitHub Action can fail a build on a known exploited vulnerability, on any known vulnerability, or never, so a version you have just removed cannot quietly come back in next week's merge.

3. Gold Registry: start from components that are already fixed

registry.safeguard.sh provides hardened, drop in replacements for upstream packages and container images. Every Gold artifact is held continuously to zero exploitable critical or high CVEs, zero malware and a completed license check, with SLSA L3 or higher provenance, a Sigstore signature and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. This fork and patch flow is in early access today, so ask which packages it covers before you plan around it.

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

The first demo is a real Safeguard CLI session. It installs the CLI, opens the interactive terminal, and asks it to fix a lodash CVE: the assistant reads the finding and advisory, asks before each command, edits package.json to lodash 4.17.21, runs npm install and npm test with approval, and pushes nothing. The same flow applies to the repository holding your trainer, environments and verifiers.

Click through it at your own pace, or open it full screen.

The second demo connects Claude to the Safeguard MCP server. You add Safeguard from Claude's connector directory, sign in with OAuth and choose which tools Claude can use. An assistant that helps write training or environment code can then query SBOMs and vulnerability data about that code.

Click through it at your own pace, or open it full screen.

The third demo sets up the local runner on your own machine, approves it with its device code and then scans from the terminal. Training code often lives on GPU hosts with restricted network access, and a local scan keeps the code where it is.

Click through it at your own pace, or open it full screen.

To try it yourself, download Safeguard at safeguard.sh/download and read the CLI documentation.

Quick check

Sources

• arXiv 2203.14465: STaR: Bootstrapping Reasoning With Reasoning
• arXiv 2401.10020: Self-Rewarding Language Models
• arXiv 2401.01335: Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models
• arXiv 2212.08073: Constitutional AI: Harmlessness from AI Feedback
• Anthropic: Constitutional AI: Harmlessness from AI Feedback
• arXiv 2312.06585: Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models
• arXiv 2411.15124: Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• arXiv 2501.12948: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• Nature: DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Nature: Mastering the game of Go without human knowledge
• arXiv 2305.17493: The Curse of Recursion: Training on Generated Data Makes Models Forget
• Nature: AI models collapse when trained on recursively generated data
• arXiv 2404.01413: Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data
• arXiv 2210.10760: Scaling Laws for Reward Model Overoptimization
• arXiv 2201.03544: The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models
• arXiv 2406.10162: Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models
• arXiv 2508.18255: Hermes 4 Technical Report
• Nous Research: Atropos repository and README
• Nous Research: Hermes Agent repository and README
• Hermes Agent v0.22.5 release notes
• Hermes Agent docs: Batch Processing
• Hermes Agent docs: Trajectory Format
• Hermes Agent docs: Skills System
• OWASP GenAI Security Project: LLM04:2025 Data and Model Poisoning
• Sigstore docs: Signing blobs with cosign
• CycloneDX: cyclonedx-python
• PyPI: atroposlib
• PyPI: cyclonedx-bom
• PyPI: pip-tools

Last reviewed: 23 October 2026. Hermes Agent ships several releases a week and research results are revised between paper versions; check the linked repositories and papers for the latest versions before relying on any number above.