Top 8 ways to piss off your Ops team
No development team can succeed without a killer operations team. But the truth of the matter is, most developers do things every day that piss off their Ops team and make the working relationship harder to deal with – when in fact they should be doing the opposite.
So what exactly might your development team do that makes things more complicated for your Ops team? Here’s a list of the top eight ways you likely piss off your Ops team without even realizing it – plus tips on how to turn the situation around.
1) Requiring a manual delivery process with constant monitoring
Delivering new code that’s so fragile and brittle you have to keep an eye on it for the entire delivery process is a surefire way to piss off your Ops team. It requires them to drop everything else (talk about disruptive!) and will only become even more of an issue as your team moves to start delivering more, faster – an inevitable trend in today’s environment.
In most cases, this situation occurs when you have a manual delivery process without any automation via tooling or scripts, but it could happen even if you do have Continuous Delivery processes and tooling in place. When you do have that automation and continue to call out special cases that require extra close monitoring, it’s all that more frustrating for your Ops team.
What to do instead: Aim to build more stable and reliable code that can be released through automation without any additional monitoring. Doing so will increase productivity for everyone from your development team to your Ops team and can help focus that time saved on creating more value for customers in the products you develop and the ways you deliver them.
2) Requiring regular off-hour releases, especially if it involves an outage
What if you’re doing things that are so dangerous to do in production that you have to do them when people aren’t using your application? It might be no sweat for you, but it means your Ops team has to take time out of their off hours to do this. And while some off-hours work is part of the job for most Ops teams, having to take that time regularly due to such risky releases is definitely not what they signed up for. Plus, it might even mean downtime for your customers if it requires taking your entire application offline.
What to do instead: As we move into a Continuous Delivery world, it’s important to make sure you have the ability to release new code in a fully automated way, which will ease the burden on your Ops team. Additionally, you should never make changes that actively break code already running in production. Rather, you should aim to make your releases complementary to what’s running in production, including making sure that code is forward and backward compatible.
3) Making changes that are not forward and backward compatible
Purposely breaking the system? That’s sure to make your Ops team life more difficult. And there are a lot of ways you can do that: You can change your database and move a field that everyone uses, you can require an outage until a new release is fully deployed, you can discover a new data structure doesn’t work with your existing code and have to release a hotfix or you can change APIs in a way that break how different external and internal services talk to each other. No matter how you do it, the result will definitely be a pissed off Ops team.
What to do instead: Make sure everything you release is both forward and backward compatible. In a Continuous Delivery environment, so many changes get rolled out over time that it’s inevitable both new code and old code will run simultaneously. This environment requires you to think both forward and backward about how you build and release new code to ensure it doesn’t actively break anything. Doing so means recognizing that local tests may not always fully replicate what’s live in production when you’re operating in a Continuous Delivery environment. Another best practice to help this situation is to make database changes separate from code changes.
4) Constantly changing the build requirements
Imagine your Ops team just took the time to carefully put together a system for build requirements that does exactly what it’s supposed to do based on your team’s direction. Now imagine your team comes back to them shortly after they’ve finished with an entirely new set of requirements. For instance, what if your system is built around one app being deployed but suddenly you want to enable microservices? Constantly requiring your Ops team to go back and redo their work is highly disruptive to their own commitments and, quite frankly, frustrating.
What to do instead: Introduce a system designed for constantly changing requirements. For example, Spaceship is constantly updating to the latest best practices for Continuous Delivery, and that means we recognize the need to change build requirements without a hassle. As a result, Spaceship can easily adapt to your latest configurations to make this situation headache-free for everyone involved.
5) Releasing major new features without using feature flags
Every time you release a new feature, it doesn’t matter how much QA you’ve done – there will always be something you missed. It’s just how things are. That’s why we have feature flags to help limit the audience who sees the feature as you continue to work out the kinks. But if you go without a feature flag and roll out the new feature to your entire audience at once, as things start breaking, so will your Ops team’s patience.
Beyond pissing off your Ops team, the inevitable errors and performance issues that will spread to your entire customer base all at once will also result in a terrible experience for your users.
What to do instead: Quite simply, use feature flags. They’re a simple development tool but one that’s absolutely necessary in a Continuous Delivery environment. They can act as a release toggle so that your team can hide specific functionality for all or some users, enabling you to ship partial features to production without affecting users in a negative way. And not only do they reduce risk, but they also increase productivity – talk about a win-win!
6) Ignoring your CI system or (gasp!) not having a CI system
Every system has a loophole and taking that path might lead to a faster end result, but it’s also guaranteed to wreak havoc. After all, the system is there for a reason. So even if it’s easy to change your code in a way that doesn’t break tests so you can get around CI checks or, worse yet, just ignore a failed test by commenting it out, you absolutely should not. You might think you understand what the error is saying, but maybe you don’t – and then you’re just going to push out broken code.
In general, manipulating the system by changing your code or ignoring the CI checks is a good way to piss off your Ops team as well as other developers and your QA team because it weakens the tests and reduces confidence in the system.
What to do instead: First, make sure you have a CI system in place. Second, configure your CI system so that people can’t reduce coverage in code, reduce the amount of tests run or mark tests as ignored. Add in checks so that an additional third party is alerted any time a user attempts to take one of those actions, that way they can review and either agree with the action or prevent them from skirting the system.
7) Not doing any load testing or performance monitoring
Just because your code is correct doesn’t mean it runs well. You could release a brand new feature, ensure it works properly and then roll it out from behind a feature flag to go to your entire customer base only to find it’s causing daily outages or system slowdowns. The culprit could be a number of things, but it’s something you can diagnose in advance with load testing and ongoing performance monitoring as you roll it out to more and more customers. Whatever the cause, your Ops team will have to bear the brunt of these slowdowns, and if you don’t give them code that runs well, you’re putting them between a rock and a hard place.
What to do instead: Always run load testing before a release and then follow that through with performance monitoring. Along the way, keep in mind that if something is slow during testing, it will be slow during production too. Importantly, you usually have to extrapolate that slowdown to understand how it will impact all of your customers at once, since load testing only mimics a portion of the production environment. The same goes for performance monitoring too, because a minimal slow down that occurs when only a small subset of users have the new release will continue to get bigger as you roll it out to more and more people.
8) Not providing useful metrics or logging with your new features
Let’s say you just released a new feature that sends email, but you don’t monitor that those emails actually get sent or even how many get sent on a regular basis. How will you know what went wrong? It could be your code or it could be the email server, but without proper logs you won’t be able to identify the issue early on, let alone trace the source of the issue.
Without this visibility, your Ops team won’t be able to tell you what’s going on and therefore won’t be able to solve the problem easily. This has the added impact of making your own life more difficult too, since it means your Ops team won’t have the insight to share any feedback with you about how your code is working.
What to do instead: Your Ops team needs good introspection into the code to understand what’s going on so they can do their jobs effectively. If you provide that insight via detailed metrics and logging of activities, they can typically solve problems on their own because they can identify where the issue is and come up with solutions. This results in faster solutions to problems, plus better insight back to you from the Ops team that you can use to improve efforts going forward (and of course a better working relationship).
Hello Ops team, it’s your developers and we’d like to start over
If you’ve done any of these eight things, chances are you’re not a fan favorite among your Ops team. But that doesn’t mean you can’t turn things around. There are several steps you can take, including those noted above, to get into the good graces of your Ops team. And best of all, this will not only help improve your working relationship, but also deliver a better end result for your users.
Ready to get started? Contact Spaceship today to discover how we can help.