A scheduled AI task can fail by erroring, by timing out, by being skipped, or by running and producing a wrong result. Ask any vendor how each case is reported, who gets the alert, whether a retry can repeat a side effect, and what the run log keeps.
Four ways a run goes wrong
- Hard error. A tool call fails, a credential expired, or the service was unavailable. This is the easy case, if an alert exists.
- Timeout or budget stop. The task ran out of time or usage allowance part way through. Half finished work is the dangerous variant.
- Skipped run. The device was off, the schedule was paused, or a quota was hit. Nothing ran and nothing says so.
- Wrong result. The task completed and reported success, but the output was incorrect or incomplete. No error exists to alert on.
Vendors tend to handle the first well and the last not at all, since a model cannot reliably flag its own mistakes. That gap is why a human check on outputs stays part of the design.
Retries and repeated side effects
A retry is only safe if repeating the task does no harm. Think of two tasks. One reads open tickets and writes a summary to a document, replacing the old summary. Running it twice gives the same state. The other emails a customer list. Running it twice sends two emails.
Ask these questions:
- Does the platform retry automatically, and how many times?
- Can you turn retries off per task?
- Does a retry restart from the beginning or from where it stopped?
- Is there a way to mark a task as safe or unsafe to repeat?
For tasks with side effects, prefer a design where the task checks whether the work was already done before doing it. That is a property of how you write the task, not just of the product, but the product should let you set retry limits.
What the run log should show
A useful run log answers, after the fact, what happened without guessing. Look for:
| Field | Why you want it |
|---|---|
| Start and end time, with time zone | Detect skips and overruns |
| Identity used | Attribute actions |
| Tools called, with arguments | See what the task actually did |
| Outcome per step | Locate partial failure |
| Output or a pointer to it | Verify results |
| Model and settings used | Explain changed behavior |
Also check retention and export. A log you cannot export or keep for your required period will not help in an incident review.
Alerts that reach a person
Even a detailed log is useless if nobody reads it. Ask where failure alerts go: email, chat, a webhook into your paging tool. Confirm that the alert names the task, the owner and the failed step. Then test it by breaking a task on purpose, for example by revoking its credential, and see who finds out and how fast.
OWASP's notes on improper output handling list insufficient monitoring and logging of model outputs among the conditions that raise impact. The same applies to unattended runs: if you cannot see outputs and failures, small problems grow.
A minimal acceptance test
Before approving a task type, run these four checks: force an error, force a timeout, pause the device or schedule for one slot, and feed it a deliberately wrong input. Record what you see in each case. If one of the four produces no visible signal, either add your own monitoring or limit that task type to low impact work.
Safeguard's policies and gates can help with the change side, enforcing rules on what automated changes may be merged, while the vendor's own alerts cover run health.
A worked example: the quarterly report that went missing
Imagine a task that compiles a list of expiring vendor documents on the first of each month and emails it to the compliance lead. In month three, the connector credential expired. The task ran, found it could not read the document store, and produced an empty list. The email arrived saying nothing was expiring. Nobody noticed for two months.
This is the fourth failure type, the wrong result, and it shows why alerting on errors is not enough. The fix has three parts:
- Make emptiness suspicious. Have the task state how many records it read, not only what it found. "Read 0 documents" is a visible warning sign.
- Add an expected range. If the list usually has 20 to 40 entries, flag runs outside it.
- Alert on credential expiry. Many connectors support expiry notices. Turn them on and send them to the task owner.
Step by step: an acceptance test for failure handling
- List every failure type from the section above that could apply to the task.
- For each, plan a way to cause it safely: revoke a test credential, set an impossible time limit, pause the device, or feed an empty input.
- Run the test in a trial tenant with dummy data.
- Write down what the platform showed, who was notified and how long it took.
- Compare the result to what you would want in production, and add your own monitoring for gaps.
| Failure | How to cause it | What you want to see |
|---|---|---|
| Hard error | Revoke a test token | Alert within minutes naming the task |
| Timeout | Set a very short limit | Alert, plus a note that work was partial |
| Skipped run | Pause the schedule | A visible skipped entry in history |
| Wrong result | Feed an empty input | A count in the output that exposes it |
Who owns the alert
Alerts need an owner and a backup. A mailbox that nobody reads is not a control. Give each task a primary owner and a secondary, and review the list when people change roles. Treat a task whose owner has left as an incident of its own: pause it until someone takes it over.
Try it yourself
Click through it at your own pace, or open it full screen. Look at how a recurring check surfaces a change, and ask how your own task failures would surface.
Quick check
Can a task fail even if it reports success?
Yes. A model can produce a plausible but wrong result. Spot check outputs and compare against a known source.
Should every task retry automatically?
No. Retry read-only and repeat-safe tasks. For tasks that send or change things, limit retries and make the task check for earlier completion.
How do I know a successful run was correct?
Check outputs against a source you trust, include counts in the output so empty results stand out, and sample runs on a regular schedule.
Sources
Last reviewed: 19 October 2026.
