The importance of the inspect and adaption cycle

At my work, we pride ourselves on our ability as an development team
to continually improve not only our engineering practices but also our planning,
testing, and communication.

Recently we performed an operation on one of our primary databases. This
involved creating a new slave database for our backups and other read only
services, promoting the current slave to be the primary as it had more resources
available to it, then decommission the old master.

This process was given a two hour window to be completed. The team involved
performed this work smoothly in just over an hour.

Post-work, we met as a team to perform a short retrospective; a common "meeting"
in the Agile/Scrum process which gives the team the chance to review what was
done, what worked, and what could be improved next time.

Here was the outcome from our retrospective:

Things we did well

• Planning and documentation processes have improved immensely over the last couple of maintenance windows
• Having the team onsite yet again proved to be superior to remote - communications and responsiveness to issues was great

Things we can improve

• Obtaining a maintenance window was problematic. Given my work is a global business, finding a convenient and appropriate time for a maintenance window has always proved challenging
• Generate and document success test cases that are going to be used to verify the system is ok in advance. This will help prevent us tripping over application bugs on the day, which could hamper the successful completion of maintenance

Actions

• Develop and maintain a maintenance window calendar that the business has access to so that if a window is needed a call can be made without involving engineering
• All maintenance windows should have a dress rehearsal done before a proposed window is requested. This is to ensure the team are across the process that needs to be done and more accurate timings are defined for the outage window

As agile practitioners its vital that we continually look at ways to improve on what was done so that next time it can be done better.