All our apps, databases and services have a planned downtime for off-line backup and other maintenance. Some services are failed over to another server (the Web never sleeps) and some aren’t (our in-house staff goes home and sleeps.) All maintenance showed up as failures.  A meeting was called to confront everyone responsible for the servers.

Item one:  Our in-house user interface was down from 3AM to 4:08AM. How could I account for this? I knew where to look:   in the crontab for that server.   I was stopping the user interface every night while the database was checked for integrity and snapshotted.  Everyone in the room had a similar story to tell:  Services were being stopped while snapshots were taken and other maintenance that required a quiet system.  As far as the public was concerned there was no outage.

The meeting was dismissed. We all left feeling like the well oiled machine of our infrastructure had survived the scrutiny and something would be done about the uptime report-making.  That is not what happened. I got an email from Philip an hour later:  “Please move the user interface outage to 3 hours earlier.”

It must be noted that all automated system administration  tasks are carefully planned in coordination with other systems that may be affected.   I realized no one had been consulted about possible conflicts. Probably no one would be in the office at midnight when Philip wanted this operation  to begin but the database would still be running as well as many other cron tasks throughout the organization (why am I even pretending this can be justified.)

Up to now the various crontabs have been executing in harmony and trouble-free. We are being asked to hastily move things around for the purpose of this pie chart.

Sure enough, when I went into Philip’s office to discuss this, there was Cecil projecting the pie chart of outages on the wall. One of the database administrators was already there and he was looking peeved. He had an email in his hand which he turned so I could read it. I recognised the short request with an AM after the time. That was all I needed to see. He was also being asked to shift his cron jobs around.

Philip was ready to explain his actions.  “If we move all these slices” he wiggled his laser pointer across the 1AM – 5AM sector “over to here,” lots of wiggling in the 0AM – 1AM sector “then all our downtime will be condensed into one hour.”

This was not satisfactory to me or the database admin. The only way this could happen is taking all servers off-line (including fail-overs) from 0AM – 1AM and that is when our American cousins are doing all their ecommerce with us (midnight in Melbourne is 10AM in New York, 3PM in London). We all pleaded to take no action until all admins could meet. We have to drag this out into meeting after meeting. It is our only hope.

I alone understand that Philip is fighting for his corporate life here. The justification for his existence comes from making changes and reporting that he was responsible for them. He wanted to meet with his boss, show the old and new pie charts side by side. His boss would be very impressed. Only the cussed sysadmins were in the way of this dynamic leveraging of  synergy.

We are still resisting but I am considering the downside of caving in to Philip’s plan and letting hell break loose with failed backups and corrupted snapshots and angry customers but that would create a lot of cleanup for all the sysadmins (a couple of us might get fired as scape-goats.) Also Philip would also heroically come to the rescue. Revert to the original regimen, and report “mission accomplished” to his boss yet again while the sysadmins struggle to hold the infrastructure together with their bare hands.

Every course of action is a lose-lose for the sysadmins and a win-win for Philip.  Cecil has started tagging along with Philip everywhere.  This is a bad synergy.

He will complain about me to my manager.  It is another example of privileged individuals who are not required to use the helpdesk.  This would have been far better handled if the helpdesk had intercepted this call and tried to follow the guidelines set out by Simon Tatham here.

In my first IT job in  Melbourne the helpdesk was taken very seriously. If a helpdesk ticket was generated you would have to respond within 15 minutes or you were in trouble. You had to talk to the user either in person or over the phone and tell them you were aware of the issue and would soon have it resolved. At this place helpdesk tickets are often ignored. We all know that no one will ever follow up. The users sense this and avoid using the helpdesk and prefer to stalk the technicians.

The people who have no clout can be made to call the helpdesk.  We just ignore them otherwise.  We will probably ignore them even after they have put in a ticket.  A lowly mail room clerk approached me to fix his printer.  He had a PC solely for the purpose of logging shipments.  His printer had not worked for 10 days.  I cleared the jam, cleared his print queue, restarted his spooler, then I went over to the helpdesk office to let them know to close the ticket.  They let me know there was no ticket.  “We don’t help them.” was the explanation.  There was some feud or disrespect that had not been settled.   How long can a company go on like this?

These events are connected by a clear failure of management.  Where I worked before we had a CIO and that is why priorities were established and rigidly enforced.  Philip is not going to enforce anything. That is akin to making waves and the successful manager knows to keep a low profile in all matters other than reporting successes. Here, CIO is an empty title that has no rank and can enforce no policies. It was bestowed on Philip to justify a rise in pay.

Barry

Melbourne

Some time later a situation arose that I could not handle in a timely manner:  A new server was giving memory errors. I told Philip we should bring in the vendor. We were having frequent slowdowns and core dumps. I realized there was a lot of work involved in pinning it down (especially not having parts to swap out) and we should set the vendor on it who had all spare parts and diagnostics ready and we were under warranty so there was no cost to us.
Philip determined that a contractor was out of the question and he couldn’t understand why we could not handle this ourselves.

I set about handling it. I analyzed all the logs and the core dumps. I Looked for hardware issues. I patched the system and the firmware.
I realised that half the RAM was being taken offline shortly after a reboot. I took out half the RAM. The problem went away (the defective DIMM was in the half I took out. That was just dumb luck.) The vendor will not replace the RAM without their man diagnosing it. So I have to replace all the RAM and it’s back to the vendor option. In any case, much time was wasted by disregarding my first diagnosis. I am blamed for not realising this sooner. Philip often hints that this outage is somehow my failure. And so this is how time is wasted and reputations tarnished. I almost suspect that the vendor option was rejected out of hand because it was my suggestion. I really don’t want to believe that.

There is no real lesson here.  There is nothing I could have done differently, other than having a screaming fit.  I could say that in the ideal scenario, where technicians made the technical decisions, the matter would have been expedited quickly and silently. In this case the manager made the wrong technical decision but blamed the technician for the outcome anyway. It is a small step from that to actually delegating technical decisions to the technician and then being fully justified in blaming him for the outcome.

Barry