Automation is great. Until it breaks and nobody gets paid • The Register https://www.theregister.com/2023/04/14/on_call/?

td=rt-3a

SIGN IN / UP

SOFTWARE 54

Automation is great. Until it breaks and nobody

gets paid
An ill-considered cron job turned into a nasty 2AM job

Simon Sharwood Fri 14 Apr 2023 // 07:31 UTC

ON CALL With Friday upon us, and a weekend next on the schedule, The Register once
again brings you an instalment of On Call, our weekly reader-contributed tales of being
dragged out at all hours to fix failures inflicted by the foolish, flummoxed, or fatuous.

This week, meet a reader we’ll Regomize as “Hugh”, who in the early 2000s scored a
contract as a Linux admin for a global auto manufacturer.

Hugh spent regular weeks on call and told us those times were “sure to bring at least one
sleepless night doing battle against failed software, or hardware.”

One of those incidents started at 2:00 AM when Hugh’s pager pinged with news that a
host used by the HR team was in trouble.

Hugh did the shake yourself awake and turn on the laptop in the middle of the night thing
and logged in to inspect the system, which clearly needed a reboot. So Hugh initiated a
power cycle, watched it reboot without incident, then ran a series of tests. That effort
produced nothing untoward, so Hugh prepared to retire for the night once again.

But just as he was about to hit the sack, a new alert arrived. The same host was in trouble

1 of 6 15/04/2023, 21:12
Automation is great. Until it breaks and nobody gets paid • The Register https://www.theregister.com/2023/04/14/on_call/?td=rt-3a

again. Again, the system showed no sign of distress, and a reboot again brought it back
to life This time Hugh decided to run some extra checks to make sure he hadn’t missed
anything the first time.

That extra time paid off because exactly five minutes after reboot the host locked up
again.

Hugh decided that 300 second interval was a clue, so when the system came back to life,
he disabled cron, the ubiquitous job scheduler found in Unix-esque systems.

Next, Hugh started looking for evidence of any scheduled jobs.

2 of 6 15/04/2023, 21:12
Automation is great. Until it breaks and nobody gets paid • The Register https://www.theregister.com/2023/04/14/on_call/?td=rt-3a

And found one.

“It's MASSIVE, and its time stamp was ... about five minutes in the past,” Hugh told On
Call.

MORE CONTEXT
Techie called out to customer ASAP, then: Do nothing

Uptime guarantees don't apply when you turn a machine off, then on again, to 'fix' it

Errors logged as 'nut loose on the keyboard' were – ahem – not a hardware problem

Techie fired for inventing an acronym – and accidentally applying it to the boss

A little investigation led him to a script that he described as designed to “append itself to
his crontab each time it runs, then execute his target script 16384 times, and copy itself
again.”
SIMILAR TOPICS 54 COMMENTS
“The job in question was to collect timesheets from various sources, and take that to
On Call

3 of 6 15/04/2023, 21:12
Automation is great. Until it breaks and nobody gets paid • The Register https://www.theregister.com/2023/04/14/on_call/?td=rt-3a

More like these

payroll.”

TIP US payroll
But the OFF system, and the host Hugh was trying to fix, did not enjoy that influx of info
Send us news
and fell over.

Which was bad for Hugh seeing as he was now wide awake at 2:00AM, and also because
the function of this crappy cron job was to collect time sheet info for contractors.

Techie called
Contractors out to customer ASAP,
like Hugh. Uptime guarantees don't apply when
then: Do nothing you turn a machine off, then on again,
ON CALL
“Folks Service
were level agreement
not happy when theyshould
did really to 'fix'
not get paid it
on time,” Hugh told On Call, rounding
specify services, not just arrivals
out his tale with news that the chap who wroteON theCALL The chap who took the support call for
script and had his cron privileges
the SEV-1 incident survived – just
revoked.
SOFTWARE 8 days | 102 STORAGE 15 days | 121

What has automation messed up in your life? Click here to send On Call an email and
we’ll automatically consider it for a future On Call. ®
Errors logged as 'nut loose on the Why AI inference will remain largely on
keyboard' were – ahem – not a the CPU
Sponsored: Why
hardware problem AI inference will remain largely on the CPU
It’s a complex argument, but there are good
ON CALL Five developers named Bob were not reasons why inference shouldn’t head into
good at their jobs accelerators or GPUs
SOFTWARE 22 days | 186 SPONSORED FEATURE

Techie fired for inventing an acronym – Duelling techies debugged printer by

and accidentally applying it to the testing the strength of electric shocks
boss ON CALL Even a hundred-volt jolt couldn’t
ON CALL A tale of how a PEBCAK became a convince one of them that hardware was the
CLE problem
ON-PREM 30 days | 190 ON-PREM 1 month | 108

Service
NETWORKSdesk tech saved consultancy
1 month | 83 PC tech TECH
PERSONAL turns doctor to diagnose
2 months PC's
| 216

4 of 6 15/04/2023, 21:12
Automation is great. Until it breaks and nobody gets paid • The Register https://www.theregister.com/2023/04/14/on_call/?td=rt-3a

Capita from VPN meltdown, got a constant crashes as a case of arthritis

smack for it ON CALL Customer with a magnetic personality
ON CALL Maybe he shouldn’t have built a also had a significantly magnetic wristband
naughtily-named website where users could get
the fix?

If you have a fan, and want this What's up with IT, Doc? Rabbit hole
company to stay in business, bring it reveals cause of outage
to IT now ON-CALL Who would willingly chews a 6:00AM
ON CALL It ain’t half hot in the datacenter when Sunday callout?
an errant aircon engineer leaves the tech team to
take the heat
ON-PREM 2 months | 143 ON-PREM 2 months | 164

Cleaner ignored 'do not use tap' sign, No, you cannot safely run a network
destroyed phone systems ... and the operations center from a corridor
entire building ON CALL Rack full of kit predictably became a
ON CALL The natural enemy of the IT pro is the network disruption center
builder – they’ll cover you in dust, hose you down,
or worse
ON-PREM 3 months | 160 OFF-PREM 2 months | 83

5 of 6 15/04/2023, 21:12
Automation is great. Until it breaks and nobody gets paid • The Register https://www.theregister.com/2023/04/14/on_call/?td=rt-3a

The Register Biting the hand that feeds IT

About Us

Our Websites

Your Privacy

Copyright. All rights reserved © 1998–2023

6 of 6 15/04/2023, 21:12