All Stars

My son recently participated in an All Star tournament for his Little League. Watching him through the tournament, the confidence he built throughout the regular season was clearly abandoned for feelings of alienation and a general lack of passion.

When we feel the pressure of highly capable people around us, it can affect our mindset. We strive to be great in our given discipline yet when we achieve greatness and are subsequently thrust to the stage, we sometimes wither into our former selves.

Every game has an important, elevated “next level” stage like an All Star game in Little League or an executive leadership meeting for a new Director. It’s in these situations that you’ll inevitably notice that some of the players are not shaken by the heightened pressure but instead thrive on it. When others crumble as the game intensifies, these individuals exude a calmness under pressure which, when it becomes contagious enough, can spur on the team’s ability to succeed in spite of their nerves.

What, I wonder, causes some of us to break under pressure versus see the pressure as just another game to be played? Perhaps the only way to build this mindset is to have played those high pressure games enough to not be fazed by them.

Perhaps it’s on the leaders of great teams, then, to practice high pressure situations on the fields of greatness (real or simulated), in order to build comfort and coolness under pressure. If we can do that, maybe our entire team can be All Stars.

Advice for the New Engineering Manager

When I first started leading software development teams, I was extremely naive about what it meant to be a manager. My father was a small businessman and lead his company and its employees with an unmistakable authority and command-and-control style. I was naturally influenced by this approach early in my career and applied it to some of my first teams. What a disaster! I’ve since learned this and many other lessons about what it means to lead engineers, developers, and product managers in their day-to-day work.

So, here are some opinions on engineering management I think could help the newly promoted.

Let Go, But Stay Current (or, Know Your Strengths)

There is a lot of content on the subject of whether engineering leads or managers should continue coding, architecting, and being an individual contributor. It’s very tempting for the new manager to believe she can continue to code. After all the source code is right there, still just one git commit -m 'Fixed typo' away from being improved upon. While I think the answer can vary based on the company, the team, and the product, I would argue it’s better to get out of the code for the most part.

I was never a particularly strong developer; often leaving tasks to the last minute, under-scoping work, struggling in the pre-Stack Overflow days to understand harder concepts, and so on. But I did learn that the most effective engineers are commonly on the most effective teams and so I instead began gravitating toward making sure teams around me were healthy. This often meant negotiating time to reduce technical debt (often of my own creation¬Į\_(„ÉĄ)_/¬Į), upgrading development systems and IDEs, and automating tasks like builds and releases. This became a strength overcoming any weaknesses in my coding ability and I began to take satisfaction in a team’s increased ability to perform without my direct contribution to the code.

So, while I believe the first thing to get right in your new role is to focus on the team’s health, I also believe it’s important to stay current on technology and programming methodologies being deployed by your team. As a manager, you will be asked to break ties all the time (or just shepherd conversations in one direction or another) and it’s best to know a thing or two about the current issues being considered. Don’t be the manager who defers tough decisions by bringing up the good ole’ days when you wrote that CORBA interface that made the company so much money in 2003. It’s irrelevant today and, anyway, your code probably had to be thrown out at some point.

Manage Your Team’s Mindset

Today’s workplace can be a stressful place. We seek to stay productive throughout the day, ever seeking the elusive “flow state.” Meanwhile, a successful software team is often an endless fountain of distraction. The more users adore your software, it seems, the faster and louder are their complaints about a bug you just shipped or how a feature misses the mark. What a¬†catch 22! How can we achieve high levels of productivity when doing so causes more chances to be distracted?!?!?

My strategy: maximize¬†mindfulness. The software team that can consistently drown out distractions, face urgent “fires” without slipping into endless firefighting, and find peace standing in the front of the mountain of work called “the backlog” is the team that has the correct mindset. Fostering a team’s healthy mindset is now a primary job for you as the engineering manager. They cannot do it on their own.

To encourage a healthy mindset on an engineering team:

  • Assure the team that there is always time to do things correctly, even if it has be a little later than today for the sake of shipping the current solution.
  • Foster a sense of reasonable compromise and give-and-take, even in the face of seemingly intractable trade-offs resulting from the first point.
  • Let engineers pick their own tools or build new ones if necessary.
  • Listen to their needs and observe #allthefeels. Effective engineers will solve problems right up to and including the point of exhaustion, frustration, and even rage. Don’t let it happen.
  • Coach engineers to strive for high quality on every line of code, but ensure they are comfortable shipping things before perfectionism kicks in.

A healthy mindset in an engineer is a precious gift, freeing up the manager to focus on the company’s goals, customer needs, and recruiting new engineers to the team.

Master The Ability to Connect The Work to the Mission, Vision, and Roadmap

This lesson applies to pretty much every role in a technology company, but the impact to an engineering team is compounded many times over. Specifically, the engineering manager must master connecting the company mission, vision, product strategy, and feature roadmap to his or her team’s day-to-day engineering efforts. When a team doesn’t understand why they’re working on something,¬†it breeds conflict, distraction, busywork, and analysis paralysis.

“We all know that these functions must work together toward a common set of goals that involve a satisfied customer and a money-making product. The roadmap is a critical opportunity — frequently missed — to articulate those goals and rally everyone around them. Before you can establish specific goals, you need a vision, a product vision in particular that guides the direction, not only for the goal setting but also for all activity on the product.”

from Product Roadmapping by Evan Ryan; Michael Connors; Bruce McCarthy; C. Todd Lombardo

Like the crew of a ship in the middle of the ocean, a team constantly looks to the captain to tell them where they’re sailing and why. Like a remote island with the promise of treasure chests full of gold, a finished and successful software project should be something each team member desperately wants to attain. So, the engineering manager has to clarify what is so great about the software to be built, the brilliance of the roadmap, what a feature’s purpose is in the world, and how amazing it will be to finish the project.

There is a lot of overlap here with the Product Manager role, so common in startup technology companies. But I believe the engineering manager has a responsibility to make sure these connections are made whether the PM is succeeding in doing so or not.

Meetings Suck. You Will Now Create Them. So, You Suck. But It’s OK.

Dear New Engineering Manager,

Sorry, but your calendar is now officially f*cked.

Best,
The Gods of Management

Remember the days when you had long, uninterrupted blocks of time to just think about a software problem, design a solution, implement it, write tests first and iterate on the final solution? Yeah, those days are gone. But don’t make the mistake of thinking that your whole team has to suffer your newfound lack of whitespace on your calendar. Their time to dedicate to such things should be high, even as yours dwindles to zero.

For more useful and meaningful insights on this topic, just see Chapter 5 of Managing Humans by Michael Lopp. In fact, just read the whole book.

There are many different lessons that can be learned in the span of a management career, but perhaps the biggest thing to remember is that your new goal is really to build better engineers. By removing impediments like meetings, paying for expensive conferences, and coaching individuals through tough architectural challenges, you are not just helping them build a better software product, but also a better self.

“Management is the opportunity to help people become better people. Practiced that way, it’s a magnificent profession.”
– Clayton Christensen

So, welcome to management and soak in the inevitable fatigue, the constant stress, and, if you’re lucky, the chance to build a team that will do the greatest work of their lives.

Go Forth and Deploy Things ūü§Ė

For many of us in software of a certain age, making changes to and releasing software has traditionally been a perilous activity. Customers had to be hand-held, release sequences carefully orchestrated, and contingencies planned in case of failure. And so, the best releases were done by the best release engineers (remember that title?)–typically release engineers who were most averse to risk. Thankfully, we now know there’s a better way.

alex-knight-199368
The world’s most effective release engineer.¬†Photo by¬†Alex Knight¬†on¬†Unsplash

Software today is now¬†ideally¬†deployed into cloud-based infrastructures, with little or no customer involvement, with no risk of downtime, and–most critically–completely automated. The risk of breakage is minimized as the duration of Agile iterations dwindles to zero, ensuring that when breakages occur, they can be mitigated in minutes. What was once a massively delicate and stress-filled activity, can be automated into oblivion and reduced to an afterthought. This is the realm of¬†continuous deployment and delivery and it is outstanding.

As teams I’ve worked with over the years have taught me, there are some key considerations in ensuring that continuous delivery is achievable, while maintaining everyone’s sanity.

Careful Code Review

Changes made to the code have always needed peer review. But in an environment where, once a changed has the approval of peers and quality assurance, it is immediately shipped off to the customer, the importance of thorough code review increases dramatically. This is not to say that perfection is the goal (an all too common aim of naive, less experienced programmers), but rather that every change is evaluated for it’s quality and thoroughness relative to it’s potential for negative impact on the application.

Release in the Middle of the Day

If you are waiting until off-hours to release your code, it’s probably a sign that your deployment process is fragile and not correctly mitigating the risk of downtime. Fear that you will break the application at peak usage breeds an overall fear of¬†any¬†change, much less a complex and innovative one. This, of course, has exceptions and is subject to the idiosyncrasies of your industry, product, and customer needs. But as a general rule, the best time to release software is when everyone is ready to monitor and turn around any fixes.

Of course, there are ways to avoid the impact of breakages during peak usage altogether, such as rolling updates, Green/Blue deployments, canary releases, and even very new approaches like Houston.

This is not an uncontroversial subject so flexibility on execution here is necessary. Go with what the team is comfortable with, but also make decisions that build confidence in automated releases at any time of the day.

How Fast Can You Fix Your Mistakes

When continuously releasing software to production, there is an increased need to be alerted, automatically, when an issue has occurred. The human rigor typically applied to big-bang releases–such as manual canary testing, full regression test, and so on–may not be possible given the frequency of releases. As such, when a breakage¬†does¬†occur, it’s critical to have a systematic approach for detecting and redeploying changes (or rolling back, as last resort).

A common model for managing security incidents applies to continuous delivery, as well. This graphic illustrates a few key KPIs that should be kept in mind.

IMG_4021.PNG

While this graphic is intended to describe responding to security incidents, I like to imagine the same metrics apply when the incident in question is a planned release of your software that goes wrong. By managing the time to detect a problem independently from the time to recovery, a team ensures the most effective reaction times in the case of a failed release.

Stop Penalizing Failure

It is not a matter of if, but when, a software release will go wrong. How the team reacts to this failure is a good indication of whether or not they are ready for continuous deployment. In my professional past, I have been very negative and punitive in my feedback when projects are not successfully deployed. Healthier cultures and mentorship have shown me that a team that can calmly and quickly correct problems on their own is much more desirable.

‚ÄúIf you want something new, you have to stop doing something old‚ÄĚ
‚Äē Peter F. Drucker

Infrastructure as Code == ūüĒ•

When your entire application can be reasoned about, there is a sense that the risk of any given code change can be more easily reasoned about as well. For instance, if the configuration of an application clearly denotes that a production Web server cluster is set to deploy as a “rolling update” with a minimum of six instances, then the deployment team knows what to expect as the rollout proceeds.

Continuously integrating, testing, and releasing your application to customers is exhilarating and liberating, as long as you manage the risks of something going wrong. If you can identify and eliminate or minimize these risks, the benefits to your developers and ops people are massive.

So go forth and deploy things! Or, more accurately, build robots to deploy things.