Your scheduled jobs run at three in the morning, with broad credentials, unattended, and nobody finds out when one stops running.
They are the least observed code in production and frequently the most privileged, because a nightly job that reconciles accounts, expires records, sends invoices and purges data needs permissions no request handler would be given.
This post is what to get right about work that runs on a timer. For whoever inherited the cron entries.
A job that stops running fails silently
This is the property that makes everything else worse. A failing endpoint produces errors, alerts and complaints. A job that stops running produces nothing, because its output was always invisible.
The failures that hide this way:
- The schedule was removed during a migration and nobody noticed.
- The job throws early every night and exits non-zero into a void.
- It runs and processes zero records because a query or a path changed.
- It takes longer than its interval, so instances overlap or the scheduler skips runs.
- The credential expired.
Several of these are security-relevant rather than merely operational. A retention job that silently stops means you are keeping data you promised to delete. A job that expires sessions or access grants means revocation quietly stopped working. A certificate renewal job means an outage on a date decided months ago.
Alert on absence, not on failure. A job that has not reported success within its expected window should page someone. That is the only alert shape that catches the silent cases, and it is usually one heartbeat call at the end of the job plus a watchdog.
The credentials are the interesting part
Scheduled work accumulates permissions because each new task adds one and nothing removes any. The nightly job ends up able to read every customer's data, write to most tables, call three third parties and delete things.
Two changes worth making:
One identity per job, not a shared batch account. When something anomalous happens you want to know which job did it, and when you narrow permissions you want to narrow them for one task rather than for all of them.
Scope to the work. A job that sends invoices needs read on invoices and write on a delivery log. It does not need the permissions of the job that purges data, and putting them in one identity means a bug in either has the reach of both.
Overlap and idempotency
A job that occasionally runs longer than its interval will eventually run twice at once. If it is not safe to do so, you get duplicate charges, duplicate emails, or two workers fighting over the same records.
Take a lock, with an expiry longer than the job's worst case, and decide explicitly what a second instance should do: exit quietly is usually right. Make the work idempotent anyway, because the lock will fail at some point and at-least-once is the honest assumption.
Input from outside is still input
A job that reads a file from a bucket, consumes a queue, or fetches a feed is processing data it did not generate, in a context with high privilege and no user watching.
Everything you would apply to a request handler applies here: validate the shape, bound the size, treat fields as untrusted, and do not let the input decide how much work to do. A batch importer that loops over a caller-supplied count is the same bug as an unbounded API parameter, in a process with more permissions.
Changing a schedule is a change
Cron entries and scheduler configurations frequently live outside the process that governs everything else: edited on a host, changed in a console, adjusted by whoever was debugging.
That means an attacker who can modify a schedule has persistence, running as your most privileged identity, on a timer, and it will look exactly like normal activity. It also means your change management has a gap, which is the same gap as console edits and feature flags.
Keep schedules in version control, apply them through your pipeline, and alert on changes made outside it.
What to check
# Which jobs exist, and when did each last succeed?
# If you cannot answer the second half, that is the finding.
# On a host, what is actually scheduled?
crontab -l; ls -la /etc/cron.d/ 2>/dev/null
systemctl list-timers --all 2>/dev/null | head -20
# For each job: what identity does it use, and what can that identity do?
# For each job: is there an alert if it does not run?
The most valuable output of this exercise is usually a list of two or three jobs nobody can explain, still running, still credentialed.
The concession
Per-job identities and absence alerting are real work, and for a handful of low-consequence jobs the shared account and the hope that someone notices is a reasonable trade. Building observability for a job that resizes thumbnails is effort spent poorly.
The triage is by what the job touches. Anything deleting data, moving money, sending customer communications, or enforcing an expiry gets its own identity and an absence alert. Everything else can share. The list that qualifies is usually short, and it is usually the one nobody has looked at.
The implication
Scheduled jobs are the part of your system where nobody is watching, the privileges are highest, and failure is silent by construction.
Ask when each of yours last succeeded. The ones you cannot answer for are doing either nothing or something, and you have no way to tell which.
