Self-Improving AI Agents Explained: Reflection, Skills and Memory

A self-improving AI agent is one that changes something it keeps between tasks, so it does better next time without anyone retraining the model. In almost every system you can run today, that something is text or code: a written reflection on a failure, a memory entry about you, or a saved skill. The model weights stay frozen.

That distinction matters for both capability and security. Research systems such as Reflexion, Self-Refine, Voyager, Generative Agents and MemGPT showed between 2023 and 2024 that an agent can get measurably better on a benchmark by writing things down and reading them back. Production agents, including Nous Research's open source Hermes Agent and the skill and memory features in Anthropic's and OpenAI's products, now ship the same ideas. Every one of those saved notes and skills is a new artifact that persists, loads into future prompts and sometimes executes. This guide explains how each technique works, what the papers actually measured, where the loops stall, and how to put review, provenance and rollback around them.

Key takeaways

• Most "self-improvement" is context engineering that persists. Reflexion stores verbal lessons, Voyager stores JavaScript functions, MemGPT pages text in and out of the prompt. None of the five core papers updates model weights.
• The evaluator decides whether the loop works. Reflexion's coding gains depend on self-written unit tests; when those tests were unreliable (MBPP Python, 16.3% false positive rate), it scored below its own baseline. Self-Refine barely moved on maths, because the model judged its own wrong answers as fine.
• Intrinsic self-correction is weak. An ICLR 2024 paper found models often get worse when asked to fix their own reasoning without outside feedback, and a TACL 2024 survey found no convincing success with feedback from prompted models alone, outside tasks unusually suited to it.
• Skills stored as code compound, and so do their mistakes. Voyager's skill library let it keep progressing where an ablation without it plateaued, but each saved skill is executable code with no reviewer by default.
• Production agents mostly write freely by default. Hermes Agent (checked 23 Oct 2026, v0.22.5) ships memory.write_approval: false and skills.write_approval: false; you turn on staging and review yourself.
• Each level of change brings its own risk. Stale or false memory, poisoned or drifting skills, unreviewed code and unpinned dependencies. OWASP's Agentic AI Threats and Mitigations guide lists memory poisoning as threat T1 and supply chain compromise as T17.
• The defence is ordinary engineering. Stage writes, diff them, keep them in version control, attach provenance, scan the code and dependencies they pull in, and keep a tested rollback path.

What counts as "self-improvement" in an agent?

An agent is a loop: a model reads a prompt, decides on an action, calls a tool, reads the result and repeats. "Self-improvement" means the agent changes some input to future runs of that loop based on its own experience. The useful question is not whether it improves but which layer it changes, because each layer has a different cost, a different evaluator, a different blast radius and a different way to undo it.

There are five layers worth separating.

1. Context within a task. The agent critiques and rewrites its own draft before answering. Nothing persists after the task. Self-Refine works here.
2. Episodic and semantic memory. The agent writes notes that survive the session: lessons from failures (Reflexion), observations and reflections about the world (Generative Agents), facts about the user (most production assistants), or a paged archive of everything (MemGPT).
3. Skills and tools. The agent saves a reusable procedure, as instructions, as code, or both, and loads it by name or by similarity later. Voyager's skill library and the SKILL.md format used by Hermes Agent, Anthropic and OpenAI live here.
4. The agent's own code or prompts. The agent edits its scaffolding, its system prompt or its tool implementations. This is the subject of a separate post in this series on self-modifying coding agents; it is rarer in production and much harder to review.
5. Weights. The model is fine-tuned on its own outputs or on feedback. This is self-training (STaR, self-rewarding models and similar), covered in its own post. It is the only level where the model itself changes.

The rest of this article covers levels one to three, because that is where the foundational research sits and where shipping products put their learning loops.

The research at a glance

The table lists the five core papers and two critical follow-ups, each checked on arXiv on 23 October 2026. "First version" is the arXiv v1 date. Results are reported in the authors' own terms and on their own benchmarks.

A note on compute and models. All five core systems call hosted OpenAI API models from the GPT-3.5 and GPT-4 generation through prompts; none fine-tunes a model. The Generative Agents authors report that simulating 25 agents for two game days cost "thousands of dollars in token credits" and took multiple days. Voyager's authors list API cost as the first limitation of their system. These are research budgets, and the benchmarks are narrow: a text household simulator, Wikipedia question answering, short coding problems, a sandbox town and Minecraft.

How does Reflexion's verbal reinforcement work?

Reflexion, from Noah Shinn and colleagues, describes itself as reinforcing language agents "not by updating weights, but instead through linguistic feedback." It has three parts.

• An actor, the model that acts in an environment, such as a ReAct-style agent in a text game or a code generator.
• An evaluator that scores the attempt. This can be the environment's success signal, a heuristic (for example, detecting that the agent repeated the same action several times), an exact-match check, or unit tests.
• A self-reflection step, where the model reads the failed trajectory and the score and writes a short natural language lesson: what went wrong and what to do differently.

The lesson goes into an episodic memory buffer and is prepended to the prompt on the next trial of the same task. The paper keeps the buffer deliberately small: the last three reflections for ALFWorld and HotPotQA, and just one for programming. That is the whole mechanism. There is no gradient, no reward model and no training run. It is "reinforcement" in the sense that a scalar or binary signal is turned into text that changes the next attempt.

What Reflexion showed

On ALFWorld, 134 text-based household tasks, the ReAct plus Reflexion agent completed 130 of 134 tasks over 12 trials. The plain ReAct agent stopped improving between trials 6 and 7 and, in the authors' analysis, converged at a 22% hallucination rate with no sign of recovery. The most common baseline failure was the agent believing it held an item it did not have; reflection let it spot the early mistake in a long trajectory.

On 100 HotPotQA questions, Reflexion lifted accuracy for every model the authors tried, for example from 0.39 to 0.51 for one ReAct configuration.

On code, the headline was 91.0% pass@1 on HumanEval Python against a previous best of 80.1%. For programming, the evaluator is a set of up to six unit tests the agent writes for itself, and that is where the interesting detail sits.

Where Reflexion fell short

The authors are candid about three limits.

• Self-written tests can lie. On MBPP Python, Reflexion scored 77.1% against a baseline of 80.1%, below its own starting point. The reason was a 16.3% false positive rate: generated tests passed on wrong solutions, so the agent stopped early and submitted them. On HumanEval Python the same rate was 1.4%. The loop is only as good as its judge.
• Reflection without tests can hurt. In an ablation on the 50 hardest HumanEval Rust problems, removing test generation dropped accuracy to 52% against a 60% baseline, because the agent kept making harmful edits with no way to tell whether the current code was already correct.
• It cannot explore its way out. On WebShop, an e-commerce navigation benchmark, the authors stopped after four trials because the agent showed no improvement and wrote unhelpful reflections. They conclude Reflexion struggles with tasks that need diverse, creative exploration. With a smaller open code model, Reflexion gave no gain at all (0.26 before and after).

The lesson for builders: verbal reinforcement converts a reliable outside signal into better behaviour. It does not create a reliable signal where none exists.

How does Self-Refine work, and when does self-critique fail?

Self-Refine, from Aman Madaan and colleagues, is the simplest loop in this list. One model plays three roles with three few-shot prompts:

1. Generate an initial output.
2. Feedback: critique that output with specific, actionable comments.
3. Refine: rewrite the output using the feedback.

Steps 2 and 3 repeat until a stop condition, which in the paper is either a fixed number of iterations (at most four) or a stop signal the model emits in its feedback. Nothing is stored afterwards; Self-Refine improves the current answer, not the agent.

The authors evaluated seven tasks: dialogue responses, code optimisation, code readability, maths reasoning, sentiment reversal, acronym generation and a hard constrained generation task with 20 to 30 required keywords. They report about 20% absolute improvement on average, measured with task metrics where they exist and blind human or model-judged preference elsewhere.

What the ablations say

• Feedback quality matters. Replacing specific feedback with generic feedback, such as a bare request to make code faster, cut sentiment reversal from 43.2 to 31.2, and removing feedback entirely made that task fail.
• Iteration beats resampling. Refined outputs were preferred over the best of four fresh samples, so the gain is not only from trying more times.
• Small models could not do it. A 13B open chat model could not produce feedback in the required format and, even when given correct feedback, often repeated its output instead of refining it.

When self-critique fails

Maths is the clearest failure in the paper itself. Gains were zero or 0.2 points, because "errors can be nuanced" and a consistent-looking chain of reasoning fools the critic. The authors report that for 94% of maths instances the model's feedback said everything looked good. When an external signal said whether the answer was wrong, the gain rose above 5%.

Two later papers generalise the point. Huang and colleagues (ICLR 2024) studied intrinsic self-correction, where the model revises its reasoning with no outside feedback, and found models struggle and sometimes get worse. Kamoi and colleagues (TACL 2024) surveyed the field and concluded that no prior work shows successful self-correction with feedback from prompted models alone, except in tasks exceptionally suited to it; it works well when reliable external feedback is available, and large-scale fine-tuning can enable it. They also found that many earlier studies used impractical frameworks or unfair evaluations that overstated the effect.

Put simply: self-critique is good at fixing things the model can check cheaply (did I include all 30 keywords, does this code compile, is the tone positive) and poor at catching errors it was unable to see the first time.

How did Voyager build an ever-growing skill library, and why are skills stored as code?

Voyager, from Guanzhi Wang and colleagues, is an agent that plays Minecraft without human intervention. It is the paper that made "skill library" a standard term. It has three components.

1. An automatic curriculum. The model proposes the next task based on the agent's inventory, surroundings and what it has already done, aiming to maximise exploration.
2. A skill library. Every program that successfully completes a task is saved as a JavaScript function that calls the Mineflayer bot API. Each skill is indexed by an embedding of a model-written description. When a new task arrives, the agent retrieves the five most relevant skills and puts them in the prompt so it can call or compose them.
3. Iterative prompting. The agent writes code, runs it, and feeds back three signals: environment feedback (for example, that a required material is missing), execution errors from the interpreter, and self-verification, where a second model call acts as a critic and judges whether the task succeeded. If the agent is still stuck after four rounds, the curriculum moves on and may come back later.

Why code rather than notes

The authors give three properties: skills as code are temporally extended (one function can run a long sequence of actions), interpretable, and compositional (a "craft iron pickaxe" skill calls "mine iron" and "craft furnace" skills). Code also runs deterministically once written, so the agent does not need to re-derive a procedure each time, and a stored function does not get "forgotten" the way a fine-tuned model can lose old behaviour. The paper argues this "alleviates catastrophic forgetting."

What Voyager showed

Against reimplemented ReAct, Reflexion and AutoGPT agents in the same environment, the authors report 3.3x more unique items within 160 prompting iterations, 2.3x longer travel distances, and tech tree milestones unlocked up to 15.3x faster. In a fresh world with an empty inventory, Voyager solved unseen tasks the baselines could not solve within 50 iterations, and giving AutoGPT Voyager's learned skill library improved AutoGPT too.

The ablations are the most useful part for anyone building a learning agent:

• Replacing the automatic curriculum with a random one cut discovered items by 93%.
• Without the skill library, the agent tended to plateau in later stages. Accumulated, reusable skills are what kept progress going.
• Removing self-verification cut discovered items by 73%, making it the most important feedback signal.

What Voyager's authors flagged

The limitations section lists cost, cases where the agent gets stuck and cannot write a correct skill, a self-verification module that sometimes misjudges success, a curriculum that proposes items that do not exist in the game, and generated code that calls functions not present in the API. Every one of those failure modes is a reason to treat a saved skill as unreviewed code, not as knowledge.

How do memory architectures work?

Reflexion's memory is a three-item list. Two other papers built much richer designs, and production systems borrow from both.

Generative Agents: the memory stream and reflection

Joon Sung Park and colleagues at Stanford and Google built a sandbox town of 25 agents and published the work at UIST 2023. Each agent keeps a memory stream: a complete, time-stamped log of observations in natural language.

To decide what to put in the prompt, the system scores every memory on three factors, normalises each to the range 0 to 1, and adds them:

• Recency: exponential decay over game hours since the memory was last retrieved, with a decay factor of 0.995.
• Importance: a 1 to 10 score the model assigns when the memory is created, so "ate breakfast" scores low and a breakup scores high.
• Relevance: cosine similarity between the memory's embedding and the current query.

Reflection is a second kind of memory. When the summed importance of recent events passes a threshold (150 in their implementation), the agent asks itself what high-level questions its recent memories raise, retrieves evidence, and writes conclusions, such as a belief that one character is dedicated to his research, with pointers to the memories that support them. In practice agents reflected roughly two or three times a simulated day. Reflections are stored in the same stream and can be reflected on again, which builds a tree of increasingly abstract beliefs.

The evaluation was about believability, not accuracy. In interviews ranked by 100 human evaluators, the full architecture scored highest, and removing reflection, then planning, then memory made agents progressively less believable. The authors also report failures: agents sometimes failed to retrieve the right memories, and sometimes embellished their knowledge with details that never happened. Instruction tuning made agents overly formal and overly cooperative. Reflection is powerful because it turns raw events into beliefs; it is risky for the same reason, because a wrong belief is now stored and cited.

MemGPT: memory tiers like an operating system

Charles Packer and colleagues borrowed the idea of virtual memory. The prompt is treated as fast, scarce main context; everything else is external context in databases.

• Main context has three parts: fixed system instructions, a writable working context for key facts about the user and task, and a FIFO queue of recent messages.
• Recall storage holds the full message history that has been evicted from the queue.
• Archival storage holds arbitrary text, such as documents, searchable by the agent.

The model manages its own memory through function calls: it can write to working context, search recall or archival storage and page results into the prompt. When prompt tokens pass a warning threshold (70% of the window in their example), the system inserts a "memory pressure" message so the model can save what matters; at the flush threshold (100%) the queue manager evicts messages and writes a recursive summary. A function call can request a heartbeat to chain several calls before replying.

On a deep memory retrieval task built from the Multi-Session Chat dataset, where the agent must answer a question about something said several sessions ago, MemGPT raised accuracy from 32.1% to 92.5% and from 35.3% to 93.4% on two models compared with the same models using a fixed context and a lossy summary. It also handled document question answering beyond the window by paging search results from archival storage. The paper measured retrieval, not whether stored facts stay true.

How do production agents implement the same ideas?

The research patterns map almost one to one onto features you can install today. Each description below comes from the product's own documentation, read on 23 October 2026.

Hermes Agent: memory files, skills, a background review and a curator

Hermes Agent is Nous Research's MIT-licensed open source agent. Its README describes a "closed learning loop": agent-curated memory with periodic nudges, autonomous skill creation after complex tasks, skills that improve during use, full-text search over past sessions, and compatibility with the agentskills.io standard. The latest release on GitHub when we checked on 23 October 2026 was v0.22.5, published that day; the project ships several releases a week, so check the defaults below against the docs for your version.

Memory (MemGPT-style working context, but small and bounded). Two files under ~/.hermes/memories/: MEMORY.md for the agent's own notes (2,200 characters, about 800 tokens) and USER.md for a user profile (1,375 characters, about 500 tokens). Both are injected into the system prompt as a frozen snapshot when a session starts, so a write during a session only takes effect next session. Memory does not auto-compact; a write over the cap fails and the agent must consolidate. Entries are scanned before acceptance for prompt injection, credential exfiltration and SSH backdoor patterns and for invisible Unicode. The docs warn against pointing two agent processes at one Hermes home, because their writes would "compound each other's entries" into state nobody authored.

Session search (MemGPT-style recall storage). Every CLI and messaging session is stored in SQLite at ~/.hermes/state.db with FTS5 full-text search, which the agent queries with a session_search tool.

Skills (Voyager-style library, as Markdown plus files). Skills live in ~/.hermes/skills/ and use progressive disclosure: a list of names and descriptions first, the full SKILL.md when needed, specific reference files only on request. The documented template has When to Use, Procedure, Pitfalls and Verification sections. The agent manages its own skills with a skill_manage tool whose actions are create, patch, delete, write_file and remove_file. The system prompt asks it to save a skill after working out a multi-step workflow, after finding the working path past errors, and after a user correction, which is close to Reflexion's "lesson from failure" turned into a reusable procedure.

Background review (an automated Reflexion step). After a turn, a forked review can save memories or patch skills without being asked. It runs on the main model by default, can be pointed at a different model under auxiliary.background_review, and can be turned off with auxiliary.background_review.enabled: false, leaving a manual /refine command.

The curator (forgetting, made explicit). A background maintenance pass tracks how often agent-created skills are viewed, used and patched. Skills unused for 14 days become stale and after 30 days are archived to ~/.hermes/skills/.archive/; it runs at most every 7 days and only after 2 hours idle. An LLM consolidation pass that merges overlapping skills is off by default (curator.consolidate: false). The docs say the curator never auto-deletes, and hermes curator backup and hermes curator rollback snapshot and restore the skills directory.

The defaults that matter for review. Both write gates are off by default: memory.write_approval: false and skills.write_approval: false, so the agent and its background review write freely. A separate content scanner for agent-created skills, skills.guard_agent_created, is also off by default; the configuration docs say it was tripping too often on legitimate workflows. Skills found inside a project repository do not load until you run hermes skills trust, and project skills the scanner rates dangerous are quarantined. We cover the Hermes learning loop in more depth, and the two published CVE records for Hermes Agent (for example CVE-2026-71963), in other posts.

The agentskills.io format

The agentskills.io specification defines a skill as a directory containing at least a SKILL.md file, with optional scripts/ for executable code, references/ for documentation and assets/ for templates. SKILL.md starts with YAML frontmatter. Only two fields are required: name (1 to 64 lowercase letters, digits and single hyphens, matching the directory name) and description (up to 1,024 characters, saying what the skill does and when to use it). Optional fields are license, compatibility, a free-form metadata map and an experimental allowed-tools list of pre-approved tools.

What the format does not define is just as relevant to security: there is no required author, no content hash, no signature and no field recording whether a human reviewed the skill or an agent wrote it. Those have to come from the system that stores the skill.

Anthropic: Agent Skills and the memory tool

Anthropic's documentation describes Agent Skills as filesystem-based packages of instructions, scripts and resources that Claude loads in three levels: metadata (name and description, about 100 tokens per skill, always loaded), the SKILL.md body when triggered (under 5k tokens), and bundled files or scripts only as needed, where a script's output enters context but its code does not. Custom skills can be created in Claude Code (in ~/.claude/skills/ or a project's .claude/skills/), uploaded through the Claude API's Skills endpoints, or added in claude.ai settings. On the API, skills run in a sandboxed container with no network access and no runtime package installation.

The docs' security section tells users to use skills only from trusted sources, to audit every bundled file, and to "treat like installing software." It notes that even trustworthy skills can be compromised if their external dependencies change.

Separately, Anthropic documents a memory tool that lets Claude create, read, update and delete files under a /memories directory across conversations. It runs client side: Claude requests operations and your application executes them against storage you control. The docs make the application responsible for safeguards, including stripping sensitive data, capping file sizes, expiring files that have not been accessed in a long time, and validating every path so operations cannot escape the memory directory.

OpenAI: skills in Codex and ChatGPT, sessions in the Agents SDK

OpenAI's Codex documentation says its skills "build on the open agent skills standard": a directory with a SKILL.md containing name and description, plus optional scripts/, references/ and assets/, and an optional agents/openai.yaml for appearance and dependencies. Skills use progressive disclosure, starting with names and descriptions within a context budget (by default at most 2% of the context window or 8,000 characters when the window is unknown), and load the full SKILL.md when chosen. Codex reads skills from repository folders (.agents/skills from the working directory up to the repository root), the user's home, admin and system locations, and a built-in skill creator drafts new skills.

The OpenAI Agents SDK documents sessions, built-in session memory that keeps conversation history across agent runs, with implementations such as SQLiteSession. That is the recall layer; what an application promotes into longer-term memory is up to the developer.

Side by side

What evidence is there that these loops help over long horizons, and where do they plateau or drift?

The honest answer is that the evidence is short-horizon. The papers measure improvement over a dozen trials, a few hundred prompting iterations, two simulated days or a handful of chat sessions. None measures an agent learning on real work for months.

Within those horizons there are clear patterns.

Gains come early and flatten. Reflexion's ALFWorld curve shows a jump between the first two trials and a steady climb to near-perfect over 11 more. Self-Refine's quality rises with iterations but the paper caps at four. Voyager without its skill library plateaus; with it, progress continues for longer, but the authors also note tasks it gets stuck on.

The loop stops where the signal stops. Reflexion's WebShop runs and Self-Refine's maths results are the same story: when the agent cannot tell success from failure, more iterations do not help, and Reflexion's MBPP result shows bad tests can make things worse than no loop at all.

Memory accumulates errors as well as lessons. Generative Agents embellished memories. Voyager's critic sometimes judged failures as successes, and those "successes" become saved skills. MemGPT's evaluation checks that the agent can retrieve what was said, not whether what was said, or summarised, is still correct.

Production systems add the missing maintenance, by hand. Hermes's curator archives unused skills on a timer, its memory caps force consolidation, and Anthropic's memory tool docs recommend expiring files that have not been accessed in a long time. These are engineering answers to drift that the papers did not need to solve.

Weights do not move, so neither does the ceiling. A memory or skill loop changes what the model sees, not what it can do. That makes these loops reversible and auditable, which is good, and bounded by the base model, which is the limit. Claims of open-ended improvement need a benchmark, a baseline, a compute budget and an independent measurement; none of the papers here makes such a claim.

What new risks appear at each level?

Every layer that persists between sessions is an input to future behaviour. If someone else can influence what gets written, or the agent writes something wrong, the effect outlives the conversation that caused it. OWASP's Agentic Security Initiative maps several of these in its Agentic AI Threats and Mitigations guide (version 1.1, December 2025).

AgentPoison (Chen et al., July 2024) is a useful reference point because it shows the attack works without touching the model: the authors poisoned the memory or retrieval knowledge base of three kinds of agents so that a trigger in a user instruction retrieved planted demonstrations, while normal queries behaved normally. We do not reproduce attack content here; the point for defenders is that anything retrieved into a prompt is an input that needs the same scrutiny as code. OWASP's Playbook 2 for memory poisoning recommends validating candidate memory insertions, logging memory access, isolating sessions, requiring source attribution for memory updates, requiring outside validation before memories persist across sessions, and keeping snapshots so you can roll back.

For deeper treatments, see our posts on AI agent memory security risks, memory poisoning and memory hardening, and why agent skills need the same governance as packages.

A small self-improving loop you can audit

The clearest way to see what these systems store is to build a toy version. The script below needs only Python 3 and the standard library. It stands in a fixed function for the model so it is reproducible, uses fixed unit tests as the outside evaluator, keeps Reflexion-style verbal reflections in a three-item sliding window, and saves a passing solution as a Voyager-style code skill. The one addition the papers did not have is a review state: the skill is recorded with a content hash, an author and its evaluator, and it refuses to load until a human approves it.

import hashlib
from collections import deque

# Stand-in for a model call: returns a candidate implementation.
# A real agent would send the task plus the reflection memory to an LLM.
def generate(task, memory):
    if not memory:
        return "def slugify(s):\n    return s.lower().replace(' ', '-')\n"
    return ("import re\n"
            "def slugify(s):\n"
            "    s = re.sub(r'[^a-z0-9]+', '-', s.lower())\n"
            "    return s.strip('-')\n")

# The evaluator is external to the model: fixed unit tests.
TESTS = [("Hello World", "hello-world"),
         ("  Trim me!  ", "trim-me"),
         ("a--b", "a-b")]

def evaluate(src):
    ns = {}
    exec(src, ns)  # demo only: real agents run candidate code in a sandbox
    return [(i, want, ns["slugify"](i)) for i, want in TESTS
            if ns["slugify"](i) != want]

def reflect(failures):
    # Reflexion-style verbal feedback, stored as text, not as weights.
    return "; ".join(f"input {i!r} gave {got!r}, expected {want!r}"
                     for i, want, got in failures)

memory = deque(maxlen=3)          # sliding window, as in the Reflexion paper
skills = {}                       # name -> record

for trial in range(1, 5):
    src = generate("slugify", list(memory))
    failures = evaluate(src)
    print(f"trial {trial}: {len(TESTS) - len(failures)}/{len(TESTS)} tests pass")
    if not failures:
        skills["slugify"] = {
            "sha256": hashlib.sha256(src.encode()).hexdigest()[:16],
            "author": "agent",
            "evaluator": f"{len(TESTS)} unit tests",
            "status": "pending-review",   # nothing loads until a human approves
            "source": src,
        }
        break
    memory.append(reflect(failures))
    print("  reflection stored:", memory[-1])

def load_skill(name):
    rec = skills[name]
    if rec["status"] != "approved":
        raise PermissionError(f"{name} is {rec['status']}; review it first")
    ns = {}
    exec(rec["source"], ns)
    return ns[name]

rec = skills["slugify"]
print("skill saved:", {k: v for k, v in rec.items() if k != "source"})
try:
    load_skill("slugify")
except PermissionError as e:
    print("blocked:", e)
rec["status"] = "approved"        # the human review step
print("after approval:", load_skill("slugify")("Self Improving Agents!"))

Running it prints:

trial 1: 1/3 tests pass
  reflection stored: input '  Trim me!  ' gave '--trim-me!--', expected 'trim-me'; input 'a--b' gave 'a--b', expected 'a-b'
trial 2: 3/3 tests pass
skill saved: {'sha256': '32cee4ac87f734ba', 'author': 'agent', 'evaluator': '3 unit tests', 'status': 'pending-review'}
blocked: slugify is pending-review; review it first
after approval: self-improving-agents

Three things to notice. The improvement came entirely from the evaluator's specific failures being written back into the prompt; swap the tests for "looks fine to me" and the loop has nothing to learn from. The skill is code, so it inherits every property of code, including its imports. And the metadata (hash, author, evaluator, review status) is what lets you answer later where this procedure came from and whether anyone checked it. Real systems need the same fields, stored somewhere the agent cannot rewrite.

Step by step: put a review gate and a rollback path on a learning agent

This walkthrough uses Hermes Agent because its controls are documented and open source, but the same five steps apply to any agent that writes memory or skills. Check each key against the documentation for your installed version; these were read on 23 October 2026.

Step 1: see what the agent already learned

Before changing anything, look at the current state.

cat ~/.hermes/memories/MEMORY.md
cat ~/.hermes/memories/USER.md
ls -la ~/.hermes/skills/
hermes curator status        # last run, counts, pinned list

Read every memory entry. Anything you do not recognise, or that reads like an instruction rather than a fact, deserves a closer look.

Step 2: turn on staging for memory and skill writes

Edit ~/.hermes/config.yaml:

memory:
  write_approval: true          # stage memory writes for review (default: false)

skills:
  write_approval: true          # stage every skill create, patch, delete and file write (default: false)
  guard_agent_created: true     # content scanner on agent skill writes (default: false)

display:
  memory_notifications: verbose # show a preview of what the background review changed

curator:
  enabled: true
  consolidate: false            # keep the LLM merge pass off unless you review its output

With the gates on, writes from messaging platforms, scripts and the background review are staged; skill writes always stage, and in the interactive CLI foreground memory writes prompt inline. Review them with:

/memory pending
/memory approve <id>
/memory reject <id>
/skills pending
/skills diff <id>
/skills approve <id>
/skills reject <id>

Read the full diff for every skill change on the CLI or dashboard rather than in a chat bubble, where the docs say diffs are truncated. Staged skill writes sit under ~/.hermes/pending/skills/ until you decide.

Step 3: put approved skills and memory under version control

The curator has its own snapshots, but git gives you a history you can diff and audit independently of the agent.

cd ~/.hermes/skills
git init
git add -A
git commit -m "baseline: skills as of $(date +%F)"

# after each review session
git add -A
git commit -m "approved skill changes $(date +%F)"
git log --stat -5

Keep ~/.hermes/.env, state.db and anything holding credentials out of any repository you push. Memory files can contain personal details; if you version them, keep that repository local or encrypted.

Step 4: back up, then know how to roll back

Warning: a rollback replaces the current skills directory with an older snapshot, and any skill created since that snapshot is lost unless you saved it. Take a fresh backup first.

# 1. Back up current state before any rollback
tar czf ~/hermes-skills-backup-$(date +%Y%m%d-%H%M).tgz -C ~/.hermes skills memories
hermes curator backup

# 2. List snapshots, then restore a specific one
hermes curator rollback --list
hermes curator rollback --id <timestamp>

The same warning applies to resetting the git history from step 3. Back up first, then:

# Warning: discards uncommitted skill changes. Back up first (see above).
git -C ~/.hermes/skills log --oneline -10
git -C ~/.hermes/skills reset --hard <known-good-commit>

An archived skill can be brought back individually with hermes curator restore <skill> instead of a full rollback.

Step 5: treat skill scripts and their dependencies as code

A skill with a scripts/ folder is a small program. If its scripts install Python packages, pin them with hashes and generate an SBOM for the environment they run in.

# pin exact versions with hashes
pip install pip-tools
pip-compile --generate-hashes -o requirements.txt requirements.in
pip install --require-hashes -r requirements.txt

# CycloneDX SBOM of the environment the scripts run in
pip install cyclonedx-bom
cyclonedx-py environment --output-format JSON --output-file skill-env.cdx.json

Run skills that execute code inside a container backend rather than the local backend. The Hermes configuration docs list local as the default terminal.backend and say it runs commands directly on your machine with no isolation; docker runs them in a container with dropped capabilities. Then gate changes to the skills repository the same way you gate application code: review, tests, and a dependency check in CI.

Where Safeguard fits, honestly

Safeguard does not judge whether a reflection is true, does not read an agent's memory files and does not monitor Hermes Agent or any other specific agent. The part of this problem Safeguard addresses is the software supply chain the learning loop produces: the scripts in skill folders, the packages those scripts and the agent itself install, and the code a coding agent writes into your repositories. Those are open source components and source files like any others, and they need an inventory, provenance and a patch path.

Concretely, Safeguard generates and analyses SBOMs in SPDX and CycloneDX, matches components against vulnerability databases with reachability analysis, evaluates policies and gates (block, warn, notify or require approval), and records attestations and signatures (SLSA, Sigstore, in-toto). Those are generally available. Autonomous remediation, where Griffin opens tested fix pull requests, is also generally available and ships switched off, so you choose what runs on its own. If you keep approved skills in a git repository as in step 3 above, that repository is a codebase Safeguard can scan like any other, for example with safeguard scan --dir . from its root. AI-BOM discovery, which inventories AI SDK use and models across code and cloud, and MCP Guardrail, which proxies an agent's MCP tool traffic, are in early access.

Common mistakes

• Treating a self-check as a test. If the same model writes the code, the tests and the verdict, a pass means less than it looks. Reflexion's MBPP result is the cautionary example.
• Leaving write gates at their defaults. In Hermes both write_approval settings default to false, and the background review writes after turns you did not ask it to learn from.
• Reading the gist, not the diff. A one-line summary of a skill change hides the procedure and scripts that actually changed.
• Letting two agents share one memory store. Their writes compound into state neither of them, nor you, authored.
• Saving logs as skills. Incident narratives and chat excerpts in a skill become noise that loads into every future task. The Hermes docs ask for "lessons, not logs."
• Unpinned installs inside skill scripts. A script that installs the latest version of a package is a different program every time it runs.
• No backup before rollback or reset. Restoring a snapshot or resetting git history throws away recent good changes along with the bad ones.
• Assuming improvement compounds forever. The papers show plateaus and failure modes within short horizons; budget for review and pruning, not just growth.
• Reading benchmark gains as general capability. ALFWorld, HotPotQA, HumanEval and Minecraft results are specific to those setups and the models of 2023.

FAQ

What is a self-improving AI agent?

An agent that changes something it keeps between tasks, such as memory, notes, saved skills or its own code, based on its experience, so later runs go better. In nearly all current systems the model weights do not change; the agent improves what it puts in front of the model.

Do self-improving agents retrain their own model?

Not in the systems covered here. Reflexion, Self-Refine, Voyager, Generative Agents and MemGPT all work by prompting a fixed model. Hermes Agent's learning loop writes memory files and skills; it does not fine-tune the model it calls. Self-training methods that do update weights are a separate family of research.

What is the difference between Reflexion and Self-Refine?

Self-Refine improves one output within a single task by having the model critique and rewrite it, and keeps nothing afterwards. Reflexion turns a failure signal from an evaluator into a written lesson and stores it in memory for the next attempt at the task.

What is an agent skill library?

A store of reusable procedures the agent can retrieve and run later. Voyager stored skills as JavaScript functions indexed by embeddings. Today's common format is a folder with a SKILL.md file of instructions plus optional scripts and references, as defined at agentskills.io and used by Hermes Agent, Anthropic and OpenAI.

Why are agent skills stored as code?

Code can encode a long sequence of actions, call other skills, and run the same way each time, so the agent does not have to re-derive the procedure. Voyager's authors also argue it avoids the forgetting that can come with fine-tuning. The trade-off is that a saved skill is executable and needs review like any other code.

Can an AI agent's memory be poisoned?

Yes, conceptually and in research settings. If content the agent reads can influence what it saves, a false fact or instruction can persist into future sessions. OWASP lists memory poisoning as threat T1 for agentic systems, and the AgentPoison paper demonstrated poisoning of agent memory and retrieval stores. Write approval, content scanning, source tracking and rollback are the main defences.

Is Hermes Agent safe to run?

It depends on configuration. Hermes documents several security layers, including command approval, container backends and memory scanning, but its memory and skill write gates and its agent-skill content scanner are off by default, and its default local terminal backend runs commands on your machine with no isolation. Turn on write approval, use a container backend for untrusted work, and keep backups.

Does self-reflection make language models more accurate?

Only when there is a reliable signal to reflect on. With external feedback such as unit tests, environment rewards or a correct-answer check, reflection helped in the papers reviewed here. Without it, research published at ICLR 2024 and in TACL 2024 found self-correction often fails and can make reasoning worse.

How do I roll back what an agent has learned?

Keep snapshots and version history before you need them. In Hermes, hermes curator backup and hermes curator rollback snapshot and restore the skills directory, staged writes can be rejected before they land, and a git repository over the skills folder gives you a diffable history. Always back up the current state before restoring an older one.

Fix it without waiting for upstream

A self-improving agent installs packages, writes scripts into its skill folders and edits your code, and every one of those artifacts becomes part of a software supply chain that someone has to inventory and patch. The agent's own dependency tree, the plugins it loads and the scripts it saves are open source components that receive CVEs like any others, so the same fix pipeline applies to them.

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation covers eight categories, including dependencies, licenses, security posture and findings. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds. The safeguard-sh/gold-scan-action GitHub Action can fail a build on a known exploited vulnerability, on any known vulnerability, or never, so a version you have just removed cannot quietly come back in next week's merge.

3. Gold Registry: start from components that are already fixed

registry.safeguard.sh provides hardened, drop in replacements for upstream packages and container images. Every Gold artifact is held continuously to zero exploitable critical or high CVEs, zero malware and a completed license check, with SLSA L3 or higher provenance, a Sigstore signature and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. This fork and patch flow is in early access today, so ask which packages it covers before you plan around it.

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

The first demo is a real Safeguard CLI session. You ask it to fix the lodash CVE in a repository; it reads the finding and the advisory, asks before each command, edits package.json to lodash 4.17.21, runs npm install and npm test with your approval, prints a summary and pushes nothing. That approve-each-step pattern is the same one this article recommends for an agent's own skill writes: a coding agent should not change your dependencies without showing you the change.

Click through it at your own pace, or open it full screen.

The second demo connects Claude to the Safeguard MCP server with OAuth and shows how you choose which Safeguard tools Claude can use. You add Safeguard from Claude's connector directory rather than writing a custom server configuration. For an assistant that learns and acts on your behalf, choosing exactly which tools it may call is the tool-level version of least privilege.

Click through it at your own pace, or open it full screen.

The third demo sets up the Safeguard local runner on your own machine, approves it with a device code, then scans from the terminal. If you keep agent-authored skills and scripts in a local repository, this is how you would scan that repository without sending it anywhere first.

Click through it at your own pace, or open it full screen.

To try it on your own code, download Safeguard and follow the CLI documentation.

Quick check

Sources

• Reflexion: Language Agents with Verbal Reinforcement Learning (arXiv 2303.11366)
• Self-Refine: Iterative Refinement with Self-Feedback (arXiv 2303.17651)
• Generative Agents: Interactive Simulacra of Human Behavior (arXiv 2304.03442, UIST '23)
• Voyager: An Open-Ended Embodied Agent with Large Language Models (arXiv 2305.16291)
• MemGPT: Towards LLMs as Operating Systems (arXiv 2310.08560)
• Large Language Models Cannot Self-Correct Reasoning Yet (arXiv 2310.01798, ICLR 2024)
• When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (arXiv 2406.01297, TACL 2024)
• AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases (arXiv 2407.12784)
• Hermes Agent repository and README, Nous Research
• Hermes Agent releases (v0.22.5, 23 Oct 2026)
• Hermes Agent docs: Persistent Memory
• Hermes Agent docs: Skills System
• Hermes Agent docs: Curator
• Hermes Agent docs: Configuration
• Agent Skills specification, agentskills.io
• Anthropic: Agent Skills overview
• Anthropic: Memory tool
• OpenAI: Agent Skills in Codex
• OpenAI Agents SDK: Sessions
• OWASP GenAI Security Project: Agentic AI Threats and Mitigations
• CycloneDX Python SBOM tool usage

Last reviewed: 23 October 2026. Hermes Agent ships several releases a week; check each configuration key and default against the documentation for your installed version before relying on it.