The % sign that silently kills cron jobs (and three of its friends)
The Formward TeamFormward, Stockholm6 min read
The most expensive failures in operations are rarely loud. A crashed process pages someone; a job that silently never starts pages nobody, and scheduled jobs are where this failure mode concentrates, because nothing is watching the moment they run. While hardening the scheduled jobs behind our own infrastructure, we studied a class of cron failures that share one property: every one of them can pass every manual check you think to run, and still never execute in production. This post walks through the four mechanisms using a single composite anatomy, a nightly database backup that never runs while everything looks fine, because that is the job where silent failure hurts most.
Mechanism one is a carriage return. A shell script written on Windows and committed without line-ending rules ships to Linux with CRLF endings, and its shebang line no longer names /usr/bin/env bash. It names bash followed by an invisible \r, a program that does not exist. Execution dies instantly with "env: 'bash\r': No such file or directory", and it dies before the script's own logging is reached, so the failure writes nothing anywhere. The fix is mechanical and permanent: a .gitattributes rule forcing LF on anything that executes on Linux. The subtle part is coverage, because the obvious rule targets *.sh, and the wrapper scripts that cron tends to call are often extensionless. The glob misses exactly the file that matters.
Suppose the script is fixed, and suppose the verification is genuinely diligent: run the full cron command by hand under sh, with env -i to simulate cron's empty environment. It exits zero. The backup appears. The dump even restores cleanly into a scratch database with matching row counts. Every one of those checks is real, and a reasonable engineer writes "verified" after them. The trap is that none of them touched the one parser that will actually run the line every night.
Because mechanism two is cron's own parser, and it has a rule almost nobody remembers until it bites: in the command field of a crontab, an unescaped % is not a percent sign. crontab(5) states it plainly: percent signs are translated to newlines, and everything after the first one is delivered to the command as standard input. Now consider a perfectly innocent timestamp like $(date -u +%FT%TZ) inside that command. Cron truncates the line at $(date -u +, which leaves an unterminated quote and an unterminated command substitution, and the resulting fragment dies as a shell syntax error before reaching the actual job. Worse: if the command's >> logfile 2>&1 redirect sits after the first %, the redirect is truncated away with everything else. The failure has no log by construction. Nothing ran, and nothing recorded that nothing ran.
This is why the sh-based verification above proves nothing about cron. No shell implements the % translation, because it is not a shell rule; it lives in cron's reading of the crontab line, before any shell is invoked. The only test that exercises it is a real scheduled fire: a throwaway entry a few minutes in the future, and then waiting to see whether an artifact actually appears.
Which is where mechanism three likes to hide. Write that test entry with a UTC time on a server configured for a local timezone and it will simply never fire while you watch, because cron schedules in the host's local time. This one at least fails visibly within minutes, but it has a quieter cousin: documentation that says a job runs at "03:17 UTC" when the crontab means 03:17 local. On a Stockholm box those are two hours apart, and every runbook, alert window and retention calculation inherits the error.
The durable fix for the % class is not a cleverer escape sequence. Escaping (\%) works until someone edits the line, and crontab command fields are edited by people who have no reason to know the rule exists. The robust move is to remove the class: put the entire command body in a script file, which has no % rule, and let the crontab line invoke only that path plus a log redirect. Then make the rule unforgettable mechanically rather than socially, with a CI check that fails if a % ever appears in the command field of a shipped crontab. And when you write that check, control-test it against a known-broken line first. A guard that never fails on the bug it exists to catch is decoration.
Mechanism four is the structural one, and it is what allows the first three to persist: a scheduled job that has no way to say "I did not run". Success writes a log line, but the truncated command above writes nothing, and nothing is exactly what a healthy quiet night looks like. For unattended systems, the absence of errors is not evidence of success; silence has to be made into an alarm. The standard pattern is a dead-man's switch: the job pings a URL only on full success, and monitoring fires when the pings stop arriving. It is the only mechanism on this list that catches failure modes you have not thought of yet, which is what makes it part of the definition of done for any scheduled job: not "it works", but "someone finds out when it does not".
The checklist, then, for anything cron runs that you would miss: enforce LF endings on every executable file, including the extensionless ones your *.sh glob does not match. Treat cron as its own parser with its own rules, %, local time, an empty environment, none of which your shell will reproduce. Verify with a real scheduled fire, because a test that does not exercise the failing layer converts an outage into a documented non-outage. And arm a dead-man's switch, because logs only record what ran.
We run a form backend, which means the data behind our scheduled jobs is other people's form submissions, and the backup job is the one we hold to this checklist hardest. Our security page describes what we actually run, stated plainly, and the one question this post should leave you asking of any vendor, ours included, is the question that cuts through every backup claim: when did a scheduled, unattended run last produce an artifact, and when was one last restored? Everything else is a script and good intentions.