Why Teams That Ship Self-Hosted Keep Hiring Forward-Deployed Engineers

A month per customer, even when you’re good at it

Gil Feig, the CTO of Merge, talked on a podcast about how they got into customer-hosted deployments. Their first one took 18 weeks. They kept at it and got that down to 4 to 6 weeks, and they want to reach 2. Even after enough repetitions to cut the time by two-thirds, a single deployment still runs about a month. And Gil’s bigger point was that the deployment isn’t the expensive part over time. He puts building the integration at about 20 percent of the work and maintaining it at the other 80.

Lifetime cost of a customer-hosted deployment

That lines up with what we see. Each deployment is real work, and the maintenance behind it never lets up. Most of it comes down to a few things.

You can’t see into your own software anymore

You’ve probably got good monitoring for your own SaaS, but the copy sitting in your customer’s account sends almost none of that back to you. You find out something is wrong late, or you find out from the customer.

One security scanning company told us their self-hosted scanner would run into threading and out-of-memory problems they had no way to see. Their heartbeat reported the scanner as healthy or unhealthy and nothing more. A CI/CD vendor with hundreds of on-prem installs said they troubleshoot by asking customers to send screenshots. A search company that runs mostly on-prem admitted they sometimes hear about a customer’s outage from the customer.

You own the machines your SaaS runs on, so you can watch everything your software does. In your customer’s account you own none of it, and the isolation they need (their data stays put, you get no standing access) is the same barrier that stops your telemetry. 

Ten customers, ten different versions

You ship one version to ten customers, and a few months later you have ten deployments that don’t quite match. Customers change things. They upgrade when they feel like it, if they upgrade at all.

A devops vendor told us their on-prem customers upgrade about once a year, so the server version trails their cloud version by a quarter or two. A bioinformatics company keeps a separate enterprise build, cherry-picked from their SaaS, and spends real effort fighting version drift across their installs. A data platform put it plainly: every time they add a component to the product, someone has to go back to every customer and roll it out.

On a public BYOC panel, the ClickHouse team called coordinating maintenance windows across customer environments cumbersome, and the groundcover team said they built their own orchestration on top of Flux, using Temporal, just to push upgrades out with any confidence. If teams like that are hand-building it, nobody has a clean answer yet.

What happens when something breaks?

When something breaks, you need to get in. Usually you don’t have standing access, and often your contract says you shouldn’t. So every incident starts with a negotiation to get through the door.

The teams who handle this well do it the same way: no standing access, and short-lived access that’s time-bound, approved, and logged. ClickHouse gives its engineers limited, time-bound access over a Tailscale tunnel. groundcover makes the customer invite them in, with no impersonation by default. Merge streams anonymized logs and alerts out, and when an alert fires they kick off an approval process for temporary access. As Gil Feig described building it: you need a joint approval system and time-bound access you can revoke, and you build all of it yourself.

Even that isn’t enough for some customers. ClickHouse said their most security-conscious customers won’t accept just-in-time access at all, which leaves the vendor building something even more locked down.

Each new customer is a little different from the last

Your first customer is on AWS. The next one wants their own container registry and wants every package scanned before it’s applied. The one after that is on Azure. The one after that is air-gapped. None of these is a big ask on its own. Stack them up and your product turns into a pile of one-offs.

One vendor found some of their services were tied to AWS base images, so they told early customers it had to run on AWS. A company with customers on AWS, Azure, OCI, and Google Cloud maintains a version for each cloud by hand. Multi-tenant SaaS spreads this kind of work across one shared system. Here there’s no shared system underneath, just customers, so the work never amortizes.

Everyone is hiring forward-deployed engineers

This role used to belong to Palantir. Now nearly every enterprise and AI company that ships into customer environments is hiring forward-deployed engineers, or selling them as a paid service on top of the license.

When each deployment is a hand-built one-off you can’t see into and can’t upgrade cleanly, the way you make it work is to put a person next to it. Forward-deployed engineers are how vendors cover the gaps: they stand up the install, wire up the networking, chase the drift, and get on the call when something breaks that the vendor can’t see from the outside.

For the hardest accounts, that’s the right call. But it’s headcount that grows with your customer count. One search company we talked to runs its entire on-prem fleet on about one platform engineer and sells forward-deployed engineers as an add-on to cover the rest.

What you can actually get rid of

None of this means you shouldn’t offer BYOC. The demand is real, especially from regulated buyers, and for a lot of these teams it’s how they win enterprise deals at all. Some of the cost is also just real.

What you can get rid of is the repeated work: hand-authoring each environment, shipping services that can’t report back, and having nothing coordinating upgrades across your installs.

That’s what Tensor9 handles, and it maps to the four problems above. You define your stack once, and Tensor9 compiles it into a deployment for each target environment, so there’s one defined version, adapted per environment, instead of a hand-built one-off. Your customer runs a short setup that stands up an appliance and a controller in their own environment, reaching out to you, with nothing reaching inbound. When you cut a new release, Tensor9 recompiles it and the appliance applies it in place, so your installs don’t drift apart. Operator access runs through a just-in-time, time-bound, audited path your customer can gate with approvals. And the deployment’s logs and metrics route into the tools you already run, not a separate Tensor9 console.

Tensor9 doesn’t erase all of it. A customer who won’t run any vendor-managed component, or an environment with no outbound path at all, still needs a conversation. What it removes is the repeated, hand-built work, which is most of the ongoing cost. 

Because it’s software, taking on another customer looks more like a configuration change than another forward-deployed engineer to hire.

What to measure

If you already offer self-hosted, focus on two numbers. How many weeks does a new customer deployment actually take, start to finish, and where do those weeks go? And how many live deployments can one engineer carry before things start slipping? Merge’s path, 18 weeks down to 4 to 6 with a target of 2, is a fair yardstick.

If keeping all of this running is eating more of your team every quarter, it’s worth seeing the alternative to hand-building each deployment before you staff up another forward-deployed engineer. 

Poke around the Tensor9 sandbox. Inside, you can run one example stack through Tensor9 and watch a single definition become a deployment for several different environments, without rebuilding it for each. You can also try setting up telemetry and operating those environments.