All our apps, databases and services have a planned downtime for off-line backup and other maintenance. Some services are failed over to another server (the Web never sleeps) and some aren’t (our in-house staff goes home and sleeps.) All maintenance showed up as failures.  A meeting was called to confront everyone responsible for the servers.

Item one:  Our in-house user interface was down from 3AM to 4:08AM. How could I account for this? I knew where to look:   in the crontab for that server.   I was stopping the user interface every night while the database was checked for integrity and snapshotted.  Everyone in the room had a similar story to tell:  Services were being stopped while snapshots were taken and other maintenance that required a quiet system.  As far as the public was concerned there was no outage.

The meeting was dismissed. We all left feeling like the well oiled machine of our infrastructure had survived the scrutiny and something would be done about the uptime report-making.  That is not what happened. I got an email from Philip an hour later:  “Please move the user interface outage to 3 hours earlier.”

It must be noted that all automated system administration  tasks are carefully planned in coordination with other systems that may be affected.   I realized no one had been consulted about possible conflicts. Probably no one would be in the office at midnight when Philip wanted this operation  to begin but the database would still be running as well as many other cron tasks throughout the organization (why am I even pretending this can be justified.)

Up to now the various crontabs have been executing in harmony and trouble-free. We are being asked to hastily move things around for the purpose of this pie chart.

Sure enough, when I went into Philip’s office to discuss this, there was Cecil projecting the pie chart of outages on the wall. One of the database administrators was already there and he was looking peeved. He had an email in his hand which he turned so I could read it. I recognised the short request with an AM after the time. That was all I needed to see. He was also being asked to shift his cron jobs around.

Philip was ready to explain his actions.  “If we move all these slices” he wiggled his laser pointer across the 1AM – 5AM sector “over to here,” lots of wiggling in the 0AM – 1AM sector “then all our downtime will be condensed into one hour.”

This was not satisfactory to me or the database admin. The only way this could happen is taking all servers off-line (including fail-overs) from 0AM – 1AM and that is when our American cousins are doing all their ecommerce with us (midnight in Melbourne is 10AM in New York, 3PM in London). We all pleaded to take no action until all admins could meet. We have to drag this out into meeting after meeting. It is our only hope.

I alone understand that Philip is fighting for his corporate life here. The justification for his existence comes from making changes and reporting that he was responsible for them. He wanted to meet with his boss, show the old and new pie charts side by side. His boss would be very impressed. Only the cussed sysadmins were in the way of this dynamic leveraging of  synergy.

We are still resisting but I am considering the downside of caving in to Philip’s plan and letting hell break loose with failed backups and corrupted snapshots and angry customers but that would create a lot of cleanup for all the sysadmins (a couple of us might get fired as scape-goats.) Also Philip would also heroically come to the rescue. Revert to the original regimen, and report “mission accomplished” to his boss yet again while the sysadmins struggle to hold the infrastructure together with their bare hands.

Every course of action is a lose-lose for the sysadmins and a win-win for Philip.  Cecil has started tagging along with Philip everywhere.  This is a bad synergy.