What is the smallest change you made to production purely for your personal convenience?

We classify lots of production changes as BAU (e.g. requiring no paperwork) because it's the only way to survive in a large company while getting the job done.

We have a two week change process and even the smallest change is extremely draining.

  • Each environment is its complete own process. e.g. What I describe bellow starts in DEV and repeats for TEST, QA, and PROD (the last of which is set into strict change windows).

  • Each change requires a mass amount of documentation. This is entered into a very slow web GUI. It goes through multiple steps, each requiring authorisation from 3 other people (followed by a usually mandatory meeting). You can't fill ahead, and those authorisations can come at any time during your work day (or a day later, or more).

  • Consequently the most minor change requires dozens of steps, authorisations, documents, waiting time, follow-ups and meetings. These are staggered and quickly build up sapping time for any actual focused work. You also have to coordinate this with other staff/teams who are responsible for their systems also. Consequently even doing one or two changes throughout the environment can grind all of our productive work to a halt because management does not understand you can only juggle so many interruptions at once.

The only way we survive is by defining a category of our tasks as business-as-usual (BAU) and so outside of the change process, and this involves production. But I find it hard to draw the line between what we declared those were and what would fall into the category of "personal convenience".

Anything that avoids the change process is personal convenience, but also necessary.

For example we can and do enable/disable logging as required to investigate issues. We can restart our monitoring service. We can install monitoring software if it's missing, as is often the case because the people who's responsibility this really is don't give a fuck (and it doesn't require a reboot). We can add/remove/disable logins. We can alter the backup strategies and delete backups to free disk space, etc.

As DBAs here's other stuff like minor alterations to database settings (shrinking log files where we have to, enabling page checksums, and turning off auto close etc). Technically those can have various deleterious effects but we are aware of them where need be.

/r/sysadmin Thread