Running Oumi Where the Data Can’t Leave
Share this:
In just a few weeks, Oumi used Tensor9 to stand up their entire platform in a fresh AWS account that behaves like a customer’s. This post is about what Oumi didn’t have to build to get there and about the engineering compromises introduced by a SaaS product that has to run somewhere you don’t control.
Oumi builds a platform for working with AI models: fine-tuning, evaluating, and deploying open LLMs and VLMs with a commercial product on top of an Apache-licensed open source core. It is a multi-tenant SaaS hosted in Oumi’s own AWS environment. Their users are teams and organizations building on open models rather than closed APIs, and increasingly that means enterprises in regulated industries, finance especially, who want the full hosted platform but can’t send their data to it.
The hosted product doesn’t work for these customers, because their data can’t leave their environment. So Oumi came to us: what would it take to run their entire platform inside a customer’s environment?
The right answer to this is not necessarily foresight: building a product requires making decisions and assumptions about where it will run and what the environment will look like. Carefully constructing the infrastructure expecting it to run anywhere is, arguably, counterproductive for SaaS platforms (unless that’s the point of the platform, of course). But at some point you have to build the product itself and it will ultimately have ties to its environment.
What Oumi brought to us was a well-designed and expertly executed set of infrastructure tailored to the core assumption of a multi-tenant SaaS product. That assumption has changed. Our goal was to give them a new install of their platform in a separate AWS account, effectively emulating what a customer would see. This install, which we refer to as an appliance, allowed Oumi to rapidly iterate on their application code to tailor it for a BYOC installation.
“We wanted to enable regulated customers to build and own their AI with Oumi without spending countless hours on cloud deployments. With Tensor9 we got our platform running in an account that looked like one of theirs, and could spend our time on the application changes that actually needed us, not on rebuilding our infrastructure for someone else’s environment.”
Matthew Persons, Co-Founder, Oumi
In order to do this, we had to prepare automation for their deployment ranging from artifact storage to secrets management. All the while we could discuss how the new reality of a BYOC deployment changes the engineering compromises they had already established in their infrastructure.
The appliance controller always connects outbound to the vendor controller. Deployments and operational commands function over that link.
Artifact Delivery
Oumi’s platform has a container-based API backend along with a server-side rendered frontend. Both sets of artifacts must be delivered to the appliance, but they’re private artifacts and there’s an existing build, test, and release process for them. These artifacts must be delivered automatically during a deployment, never mirrored by hand, and never exposed publicly.
In Oumi’s case, their application depends on both container images and object blobs. During deployment, the Tensor9 vendor controller in their account mediates a copy of the resource from their production account into the appliance account. The deployment specifies the specific container and object versions to replicate. Only the vendor controller, which is infrastructure inside an account Oumi also owns, requires access to the artifacts.
Secrets Management
Oumi’s application, like many SaaS platforms these days, relies on a range of third party services for a variety of reasons: dependencies core to the functionality, operations, monitoring, and analytics. Some of these dependencies become optional in an appliance, which becomes an application-level change. Some are critical, but the API keys associated with those dependencies would be provided by Oumi’s customer, and not Oumi. Tensor9’s secret handling provides a mechanism to sort secrets by ownership.
BYOC Engineering Challenges
Deploying your application into your customer’s environment means the limits imposed on it will be different than your SaaS offering. Your customer’s threat model is likely different from yours and they likely have a different set of compliance obligations they must meet. The controls you’ve placed on your application for SOC 2 compliance may be insufficient in the context of your customer’s environment.
Although Oumi’s SaaS platform is available on the public internet, their customers have different tolerances for availability and access. Is it acceptable for an API to be accessible on the public internet with strong authentication, or does it need to be restricted to certain corporate IPs, or even behind a VPN? Can the site’s static artifacts still be published to a global CDN or do they need to be served from a VPC-connected server?
Even outbound connections originating from Oumi’s application to resources on the Internet may require negotiation as well. Oumi’s platform builds AI models and can pull model files from HuggingFace at runtime or during the normal course of its operation make API calls to LLM providers like Anthropic or OpenAI.
On top of all of these considerations, how do you deploy and operate your application in such an environment? Your deployment mechanism also assumes a certain amount of access. Deploying changes to your infrastructure requires widely scoped access into the account.
“The hardest part of BYOC is building a deployment model flexible enough to adapt to each customer’s infrastructure constraints. With Tensor9, we could quickly stand up representative environments, iterate across different configurations, and see where Oumi needed to adapt. That helped us preserve key platform capabilities, including the Oumi Agent, across regulated environments.”
Min Song, Engineer, Oumi
Your Stack Stays Yours
Tensor9 provides primitives that help solve these architectural challenges. During deployments, Tensor9’s appliance controller (inside your customer’s environment) only ever establishes a connection to the Tensor9 controller in your account outbound, never inbound, and your deployments and operational requests traverse this link.
Oumi’s stack stays Oumi’s. Their infrastructure-as-code remains backward compatible and still deploys their multi-tenant SaaS while only growing to express what a BYOC install needs: which secrets belong to the customer, where artifacts come from, and the infrastructure differences each environment imposes. Tensor9 provides the deployment machinery and Oumi can focus on the application work.
“Every customer has unique needs for their deployments. Tensor9 gave us the ability to easily meet customers where they are–running model training and inference on infrastructure they control.”
Matthew Persons, Co-Founder, Oumi
Customers come to us at different points along their path to BYOC. Some have already started re-architecting for other environments while others are sizing up the work for the first time. Either way, running your product in a customer’s environment doesn’t have to mean forking your architecture or standing up a team to operate it. Keep your stack, tell us the constraints your customer imposes, and we’ll work out what it takes to run it where their data lives.
