It depends on the product and how it attaches. If the agent fails open, your app keeps running without protection. If it fails closed, the app stops or blocks traffic. Neither is wrong, but you should know which you have and test it before an incident.
Two outcomes, two risks
| Behavior | What users see | What security sees |
|---|---|---|
| Fail open | Nothing, the app works | Protection silently absent |
| Fail closed | Errors or downtime | Protection never bypassed |
Fail open protects availability and risks a quiet gap. Fail closed protects the control and risks an outage. The right choice depends on the application. A public marketing page and a payments service will not make the same call.
What the Java documentation tells us
Java is a useful reference, though other runtimes differ. The java.lang.instrument documentation says that with a startup agent, the JVM aborts if the agent cannot be loaded or its premain method throws an uncaught exception. For an agent started after the VM is already running, failures do not abort the JVM, and an uncaught exception is ignored, though the JVM may log it.
So the same agent technology can behave as fail closed or fail open depending on how it is attached. That is a deployment decision hiding in a flag.
Python offers a caution of its own
Python audit hooks, a mechanism some runtime tools use to observe actions, come with a clear warning in the documentation. They are for collecting information, not for enforcing security, and malicious code can trivially disable or bypass hooks added this way. Registration can also fail silently if an existing hook raises certain exceptions. This is a statement about Python's built in hooks, not about any specific product, but it shows why you should ask each vendor how their control resists tampering inside the process it protects.
How to test it
- In a test environment, start the app with the agent configured normally and confirm protection works.
- Break the agent on purpose: wrong path, corrupted file, or incompatible version.
- Observe whether the app starts and whether anything alerts.
- Repeat for a crash while running, if you can simulate one.
- Record the result in the runbook, with who gets paged.
Make silent failure loud
If you choose fail open, add a heartbeat. The agent should report in on a schedule, and a missing report should alert someone. Without it, fail open turns into a gap nobody knows about.
Decide per service
Write the policy once and apply it per tier. Critical customer facing services may justify fail closed with strong redundancy. Internal tools may accept fail open with a heartbeat. Document exceptions.
A worked example: two services, two policies
A company runs a public marketing site and an internal payment service. For the marketing site, the team chooses fail open with a heartbeat, because an outage costs more than a short window without runtime protection, and the site holds little sensitive data. For the payment service, the team chooses fail closed behind redundant instances, accepting that a failed agent on one instance removes it from rotation while others carry traffic.
The point is not that one choice is smarter. The point is that each was a deliberate decision, written in a runbook, and tested by deliberately breaking the agent in staging.
Failure modes to rehearse
- The agent file is missing at startup.
- The agent version is incompatible with the runtime version.
- The agent loses its connection to its management console.
- The agent exhausts memory or crashes while running.
- A rule update fails to download.
For each, record what the application does, what the on call engineer sees, and how long recovery takes. The first time you learn these answers should not be during an incident.
Heartbeats and alerts
A heartbeat is a regular signal from the agent to a place you monitor. Alert when it is missing for longer than an agreed period. Also alert when a deployed instance has no agent at all, by comparing your deployment inventory with the agents reporting in. That second check catches the case where nothing started.
Documenting the decision
Put the policy where engineers will find it: the service catalog or the runbook. Include the tier, the chosen behavior, the reason, the test date, and the owner. Review it when the service changes tier.
What leadership should hear
Executives rarely need the technical detail. They need three sentences: which services fail open and which fail closed, why, and how fast the team learns when protection is missing. Give them a one page table by tier, with the test date for each row. That turns a hidden engineering assumption into a visible risk decision that someone senior has agreed to, which is exactly where it belongs.
Testing the alert path
A heartbeat is only useful if the alert reaches a person. Once a quarter, silence a staging agent on purpose and time how long it takes for the right person to be notified. Record the time. If the alert goes to a channel nobody watches or a mailbox that is not monitored overnight, fix the routing then, not during a real failure.
Try it yourself
Click through it at your own pace, or open it full screen.
Notice how a monitored system raises a signal when something changes. The same idea applies to an agent: you want a signal when it goes quiet.
Quick check
Which is safer, fail open or fail closed?
Neither in general. Fail open favors uptime, fail closed favors the control. Choose by service risk and test the choice.
How do I know the agent is actually running?
Use a heartbeat or a known test event that you check on every deploy.
Can I test failure behavior safely?
Yes, in a staging environment that mirrors production. Break the agent on purpose and watch what happens. Never run that experiment first in production.
For rollout steps, read monitor mode versus block mode. Safeguard's free scan at gold.safeguard.sh/scan covers dependency risk without adding anything to your runtime.
Sources
Last reviewed: 19 October 2026.
