The Eng Review Episode #1: Build, Buy or Away Team?

About this series

There are a lot of excellent system design newsletters out there. Most of them cover understanding an existing system in depth or designing a new system from scratch. Both are extremely valuable in their own right but this series is different.

One of your roles as a senior leader is to make decisions. Sometimes you have to make design decisions weighing trade offs when there is no clear right answer. Sometimes you have to make decisions navigating the technical and human elements in the equation. You also have to own the consequences of your decisions as an engineering leader.

In case you are thinking these are manager/director/VP problems, think again! Staff+ ICs need to make these decisions too!

In this series I cover real engineering projects. I go into the details of the tradeoffs we encountered and decisions made. Some are my stories and some are ones I’ve collected from colleagues.

Each episode has 2 sections:

• The Choice: In this section I lay out the background and context for you along with the decision question. You can put yourself in the decision maker’s shoes and try to think of what questions you’d ask or what decision you’d make and how you’d explain it to the team. This section is for all ChaiTime readers.

• The Outcome: In this section I talk about what actually happened. Some are success stories, some failures but many in-between. I share my own lessons learned - technical and leadership related - so you can learn from my experiences! This section is for my paid subscribers only since we will also be discussing it further in the Substack chat.

The Choice

Google has a large organization called Core that manages internal infrastructure used by all engineers across the company. This includes Build/Test/Deploy systems, CI/CD release pipelines, various frameworks for web and mobile development (think internal versions of SaaS kits), data management infra and more.

When I was an eng director in Google Workspace, one of the products that Core offered was called Integration Test Suite (ITS). ITS was the newer version of an older test framework and had many features to make writing integration tests easier. But it lacked a few critical features we needed in Workspace to cover the unique needs of our products. A couple of examples:

• UDP support for Google Meet tests

• Understanding and management of different types of domain accounts (enterprises, educational, government)

The ITS team did not have the engineering bandwidth to support every custom feature request across the entire company from every product area. So we had the following choices.

Option 1 - Get ITS to prioritize our features ahead of everyone else

Recommendation:

Make the argument that Workspace is an important product area with millions of users and needs these features ASAP. Mobilize VP-level escalations to prioritize our features over everyone else's in the ITS roadmap.

Pros:

• If it works, the ITS team can deliver all the features we need quickly.

Cons:

• Workspace is important but not that important compared to Search or Ads at Google so someone else could outrank us.

• The escalation game may end with no winners. Even if we win, we’d lose because ITS might have to drop other important work to fulfill the escalation decision and won’t view us as good partners in future.

Option 2 - Build missing features in-house

Recommendation:

Build the missing features within our team.

Pros:

• We can build exactly what we want as quickly as we want.

Cons:

• This will add a maintenance burden to our team.

• The new modules may not integrate very seamlessly with ITS since we’d be trying to duct-tape them on top using our outsider knowledge. This could lead to developer friction ultimately affecting our product development speed and/or quality.

Option 3 - Send an Away Team

Recommendation:

Send a small team of engineers from Workspace to ITS for a fixed period of time. For that period they will be part of ITS and learn their codebase, understand the rationale behind certain design choices and also participate in the ITS oncall rotation. They will build these features for Workspace in the ITS codebase.

Pros:

• Smoother integration with ITS since the features are part of the product, built with the product architecture knowledge and not duct-taped on top.

• No maintenance burden for Workspace.

Cons:

• ITS could be left holding a large bag of code they did not write long after the Away team departs.

• The Away Team could take a long time to come up to speed on the ITS codebase and contribute these features. This is valuable time that could be spent doing other development work for Workspace.

You are the engineering director in charge of Workspace Developer Productivity and you are tasked with making this decision. Which will you choose and why?

---

Only 5 days left for my Maven course paid subscriber lottery!

Upgrade to paid subscriber now to be considered for the lottery, get access to exclusive templates, enjoy content like this and also be part of the ChaiTime Substack chat!

---

The Outcome

What happened

Our final decision was a hybrid of 80% Option 3, 15% Option 1 and 5% Option 2!

Here is how it played out.

• We did need ITS to agree to prioritize our Away Team since if every product area sent an Away Team, ITS could still get overwhelmed with training these teams and maintaining a lot of new code they wrote. So we needed a little bit of Option 1.

• We contributed tests and probers for the new components we wrote for ITS and agreed to continue owning their monitoring and actioning. So this was a little bit of Option 2.

• We did send an Away Team to work with ITS for almost a year and they were able to contribute many critical features useful to Workspace in the ITS codebase.

My learnings

Ask a lot of questions

We had a series of meetings to finalize the Away Team’s role and scope before they started. Asking questions helped us both gain clarity and address surprises. Some questions to consider:

• How does the hosting team avoid bad code from the Away Team?

• Who will oncall/maintain the code after the Away Team goes back home?

• What about the time and energy needed to train the Away Team to be productive in a new codebase?

It's not a headcount loss, it's a knowledge gain

Don’t look at the Away TEam’s time as a headcount loss. In this case, my team gained valuable knowledge that they brought back with them. Directly, they deeply knew the internals of one of our key dependencies. Indirectly, they gained exposure to another organization’s engineering culture. Both helped us become better as an organization!

Not just an internal big-tech problem

I saw this story play out many times with external customers during my time in Google Cloud. I am also encountering this dilemma as a CTO of an early-stage startup now. If you are the CTO of a company and are considering a new software that almost fits your wishlist but not quite, then instead of falling into the Build-vs-Buy trap, you can consider an Away Team from your engineering organization to work with the software provider’s eng team for a better integration. My team at Stripe worked with many customers this way to provide a better experience on both sides!