The Eng Review Episode #1: Build, Buy or Away Team?
About this series
There are a lot of excellent system design newsletters out there. Most of them cover understanding an existing system in depth or designing a new system from scratch. Both are extremely valuable in their own right but this series is different.
One of your roles as a senior leader is to make decisions. Sometimes you have to make design decisions weighing trade offs when there is no clear right answer. Sometimes you have to make decisions navigating the technical and human elements in the equation. You also have to own the consequences of your decisions as an engineering leader.
In case you are thinking these are manager/director/VP problems, think again! Staff+ ICs need to make these decisions too!
In this series I cover real engineering projects. I go into the details of the tradeoffs we encountered and decisions made. Some are my stories and some are ones I’ve collected from colleagues.
Each episode has 2 sections:
The Choice: In this section I lay out the background and context for you along with the decision question. You can put yourself in the decision maker’s shoes and try to think of what questions you’d ask or what decision you’d make and how you’d explain it to the team. This section is for all ChaiTime readers.
The Outcome: In this section I talk about what actually happened. Some are success stories, some failures but many in-between. I share my own lessons learned - technical and leadership related - so you can learn from my experiences! This section is for my paid subscribers only since we will also be discussing it further in the Substack chat.
The Choice
Google has a large organization called Core that manages internal infrastructure used by all engineers across the company. This includes Build/Test/Deploy systems, CI/CD release pipelines, various frameworks for web and mobile development (think internal versions of SaaS kits), data management infra and more.
When I was an eng director in Google Workspace, one of the products that Core offered was called Integration Test Suite (ITS). ITS was the newer version of an older test framework and had many features to make writing integration tests easier. But it lacked a few critical features we needed in Workspace to cover the unique needs of our products. A couple of examples:
UDP support for Google Meet tests
Understanding and management of different types of domain accounts (enterprises, educational, government)
The ITS team did not have the engineering bandwidth to support every custom feature request across the entire company from every product area. So we had the following choices.
Option 1 - Get ITS to prioritize our features ahead of everyone else
Recommendation:
Make the argument that Workspace is an important product area with millions of users and needs these features ASAP. Mobilize VP-level escalations to prioritize our features over everyone else's in the ITS roadmap.
Pros:
If it works, the ITS team can deliver all the features we need quickly.
Cons:
Workspace is important but not that important compared to Search or Ads at Google so someone else could outrank us.
The escalation game may end with no winners. Even if we win, we’d lose because ITS might have to drop other important work to fulfill the escalation decision and won’t view us as good partners in future.
Option 2 - Build missing features in-house
Recommendation:
Build the missing features within our team.
Pros:
We can build exactly what we want as quickly as we want.
Cons:
This will add a maintenance burden to our team.
The new modules may not integrate very seamlessly with ITS since we’d be trying to duct-tape them on top using our outsider knowledge. This could lead to developer friction ultimately affecting our product development speed and/or quality.
Option 3 - Send an Away Team
Recommendation:
Send a small team of engineers from Workspace to ITS for a fixed period of time. For that period they will be part of ITS and learn their codebase, understand the rationale behind certain design choices and also participate in the ITS oncall rotation. They will build these features for Workspace in the ITS codebase.
Pros:
Smoother integration with ITS since the features are part of the product, built with the product architecture knowledge and not duct-taped on top.
No maintenance burden for Workspace.
Cons:
ITS could be left holding a large bag of code they did not write long after the Away team departs.
The Away Team could take a long time to come up to speed on the ITS codebase and contribute these features. This is valuable time that could be spent doing other development work for Workspace.
You are the engineering director in charge of Workspace Developer Productivity and you are tasked with making this decision. Which will you choose and why?
Only 5 days left for my Maven course paid subscriber lottery!
Upgrade to paid subscriber now to be considered for the lottery, get access to exclusive templates, enjoy content like this and also be part of the ChaiTime Substack chat!