We deploy on Friday
Recently there was an heated debate on Twitter about deploying on Friday, started by this tweet. It was very interesting to notice how people opinions could be neatly split into either “no you don’t” and “yes you should” parties.
The “no you don’t” party can be summed up like this: no matter how small and apparently harmless the change, no matter how many testing and automation tools you have in place, there’s always going to be a risk that something goes wrong during a deploy, and if it does on Friday you may have to remain at work (or come back to work) outside normal business hours. So: don’t do it.
The “yes you should” party says that if you fear that something will go wrong during a deploy, it means your automation / testing / monitoring infrastructure is not good and THAT is a problem that you should fix immediately.
At Refactory, we kinda agree with the “yes you should” party and we like to expose some additional considerations on the topic. Let’s take a virtual walk together into an imaginary garden while I expose such additional considerations.
The path that leads to Friday deploys
For a start, let’s agree that it is not acceptable to fear your deploy procedure. You may have valid reasons for being scared (legacy code, complex systems, critical services) but it is something that you have to fix. Let’s agree that, unless you can deploy at any time without being worried, your system needs to be better. Let’s agree that we all aim for peace of mind and for happy weekends. That settled, we can move forward.
1.
The first step to improve your system is to build tests. The more automated tests you have in your code, the less the chance that you introduce a bug with any new change. Automated tests should be fast enough that you can run them often. Each time you edit something, you should make sure there’s a test for that and – if not – write it immediately. And here’s another pro tip: automated tests should be part of your deploy procedure. Something wrong in the test? No deploy. Which brings us to automation.
2.
The second step is to automate as much as possible. Sometimes it’s a pain to write and maintain additional scripts, configurations, services… but deploy should really be a single click (or a single command if you’re a just-command-line-thank-you person like we are). There are tons of tools that can help you with that. We mostly use a combination of deploy keys, Ansible, bash scripts and a Capistrano-like strategy. Again: automate EVERYTHING. Copy files. Run migrations. Tag commit in git. Send notifications. Whatever you do manually during deploy, put it into a deploy script. Hey, you’re a developer, aren’t you? You’re SUPPOSED to automate things.
3.
The third step is to monitor your production system. You should set alerts on system metrics and application metrics so you’re the first to know if anything is wrong. Systems can fail even if no deploys were made, so the more critical is the reliability of your system, the more important is that you monitor it at all time. If you monitor carefully, you may notice symptoms and fix things way before the actual problem explodes.
4.
The last step, which should be obvious but sometimes is not, is: make your system easy to fix. Keep your code clean; keep your design healthy; refactor when needed. In most of our projects, when the inevitable bug happens, it’s trivial to fix it or to write a workaround. Because while firing up your laptop outside business hours to fix a production bug may be acceptable, spending all weekend doing so it’s not.
Wrap up
You should not be scared to deploy on Friday. Making your deploys safer and faster will help you stay less worried and more focused at any time. It will improve not only your Friday, but any day of your week.
P.S. if you’d like some help in improving your existing project, that’s exactly what Refactory does. Get in touch!