Observability

Observability in Tensor9 enables you to monitor all your customer appliances from a single observability platform. Logs, metrics, and traces flow from each customer’s infrastructure to your control plane, which then forwards telemetry to your observability sink, giving you unified visibility across all deployments regardless of where they run.

How observability works

When you deploy applications through Tensor9, each customer appliance runs in isolated infrastructure (their AWS account, Google Cloud project, or private environment). Without observability, you would have no visibility into how these appliances are performing, whether deployments succeeded, or how customers are using your application. Tensor9’s observability system solves this by collecting telemetry from each appliance and forwarding it to your centralized observability platform:

Telemetry generation

Resources in the customer appliance (containers, Lambda functions, databases, load balancers) generate logs, metrics, and traces during normal operation.

Collection

Tensor9 uses steady-state permissions to collect telemetry from appliance resources. Collection agents run within the appliance and use read-only permissions to access logs and metrics.

Forwarding

Collected telemetry is forwarded from the customer appliance to your control plane over secure connections. Your control plane then forwards the telemetry to your configured observability sink (Datadog, New Relic, Grafana, etc.).

Analysis

Your team monitors all customer appliances from your observability platform. You can track deployment health, investigate incidents, analyze usage patterns, and troubleshoot issues across all customers from one place.

Observability collection uses steady-state permissions, which are always active and read-only. Customers do not need to approve observability access - it runs continuously to ensure you maintain visibility into appliance health.

What you can observe

Tensor9 collects comprehensive telemetry from customer appliances:

Application logs

Logs from your application components running in customer appliances:

Container logs: stdout/stderr from containers running in EKS, GKE, AKS, or private Kubernetes
Function logs: Execution logs from Lambda, Cloud Functions, or Azure Functions
Application logs: Custom application logs written to CloudWatch, Cloud Logging, or other logging services

Logs include the instance_id and appliance metadata, allowing you to filter and correlate logs across customers.

Infrastructure metrics

Performance and health metrics from infrastructure resources:

Compute metrics: CPU, memory, network for containers, VMs, or functions
Database metrics: Connections, queries per second, replication lag, storage utilization
Storage metrics: Object count, storage used, request rates for S3, GCS, or Azure Blob Storage
Load balancer metrics: Request count, latency, error rates, healthy/unhealthy targets

Custom metrics

Application-level metrics you instrument in your code:

Business metrics: User signups, API calls, feature usage
Performance metrics: Request duration, queue depth, cache hit rates
Error tracking: Exception rates, failed operations, validation errors

Distributed traces

Request traces across your application components:

Cross-service traces: Track requests across microservices, databases, and external APIs
Performance analysis: Identify slow operations and bottlenecks
Dependency mapping: Visualize how services communicate within an appliance

Observability sinks

Your observability sink is the platform where telemetry from all customer appliances is aggregated. Telemetry flows from each appliance to your control plane, which then forwards it to your configured observability sink. Tensor9 supports forwarding telemetry to popular observability platforms:

Sink Type	Telemetry Types	Configuration
Datadog	Logs, metrics, traces	API key and site configuration
New Relic	Logs, metrics, traces	License key and account ID
Grafana	Logs (Loki), metrics (Prometheus), traces (Tempo)	API token and endpoint URLs
Sumo Logic	Logs, metrics	HTTP collector endpoint and access credentials
Elasticsearch	Logs, metrics	Cluster endpoint and credentials
OpenTelemetry	Logs, metrics, traces	OTLP endpoint URL and optional authentication
CloudWatch	Logs, metrics, traces (X-Ray)	Cross-account role for centralized monitoring
Google Cloud Logging/Monitoring	Logs, metrics, traces (Cloud Trace)	Service account for centralized monitoring
Azure Monitor	Logs, metrics, traces (Application Insights)	Managed identity for centralized monitoring

You configure your observability sink per app in your control plane, and Tensor9 automatically configures telemetry forwarding for all appliances of that app.

Configuring observability

Observability configuration has two parts: configuring your observability sink via the Tensor9 CLI, and optionally configuring telemetry forwarding in your origin stack.

Configure your observability sink

Set up your observability platform credentials in your control plane using the Tensor9 CLI:

# Example: Configure Datadog as your observability sink for an app
tensor9 observability sink configure \
  -appName my-app \
  -sinkType Datadog \
  -credentialType DatadogApiKey \
  -credential <YOUR_DATADOG_API_KEY> \
  -datadogSite us5.datadoghq.com

All telemetry from your appliances flows through your control plane to this configured observability sink. Tensor9 automatically collects basic infrastructure telemetry (resource state, deployment events, health checks) even if you don’t explicitly configure logging in your origin stack, and forwards everything to this sink for all appliances of the specified app.

Configure telemetry in your origin stack

In your origin stack, you can configure telemetry collection however you like - define log groups, add instrumentation libraries, configure metrics exporters, or use any telemetry approach that works for your application. Tensor9 analyzes your origin stack during compilation and automatically configures the appropriate telemetry routing for each appliance based on your observability sink configuration. Here are two common examples:

Example 1: Send telemetry to CloudWatch in your control plane

Define resources with CloudWatch logging in your origin stack:

# Example: Lambda function with CloudWatch logs
resource "aws_lambda_function" "api" {
  function_name = "myapp-api-${var.instance_id}"
  handler       = "index.handler"
  runtime       = "nodejs18.x"
}

resource "aws_cloudwatch_log_group" "api" {
  name              = "/aws/lambda/myapp-api-${var.instance_id}"
  retention_in_days = 7
}

During compilation, Tensor9 detects the CloudWatch log group and automatically configures log forwarding from each appliance to CloudWatch in your Tensor9 AWS account (your control plane).

Example 2: Send telemetry to Datadog via your control plane

Configure resources with Datadog instrumentation in your origin stack:

# Example: Lambda function with Datadog instrumentation
resource "aws_lambda_function" "api" {
  function_name = "myapp-api-${var.instance_id}"
  handler       = "index.handler"
  runtime       = "nodejs18.x"
  layers        = [var.datadog_lambda_layer_arn]

  environment {
    variables = {
      DD_API_KEY       = var.datadog_api_key
      DD_SITE          = var.datadog_site
      DD_TRACE_ENABLED = "true"
      DD_SERVICE       = "myapp-api"
      DD_ENV           = var.instance_id
    }
  }
}

During compilation, Tensor9 strips out the Datadog credentials from your origin stack so they are never deployed to customer appliances. Telemetry flows from each appliance back to your control plane without credentials, and your control plane then forwards the telemetry to your Datadog account using the credentials you configured via tensor9 observability sink configure. These are just examples - you can configure telemetry collection in your origin stack using any approach, and Tensor9 will handle the deployment details appropriately for each appliance.

Observability across form factors

Observability collection adapts to each appliance’s form factor:

Form Factor	Log Collection	Metrics Collection	Trace Collection
AWS	CloudWatch Logs	CloudWatch Metrics, resource-specific metrics	X-Ray or application instrumentation
Google Cloud	Cloud Logging	Cloud Monitoring, resource-specific metrics	Cloud Trace or application instrumentation
Azure	Azure Monitor Logs	Azure Monitor Metrics, resource-specific metrics	Application Insights or application instrumentation
DigitalOcean	Logs via Fluent Bit/Fluentd	Prometheus metrics	OpenTelemetry Collector
Private Kubernetes	Logs via Fluent Bit/Fluentd	Prometheus metrics	OpenTelemetry Collector
On-prem	Logs via Fluent Bit/Fluentd	Prometheus metrics	OpenTelemetry Collector

Tensor9 provisions the appropriate collection infrastructure for each environment during compilation.

Appliance identification

All telemetry includes metadata that identifies which appliance generated it:

instance_id: Unique identifier for the appliance (e.g., acme-corp-prod)
customer_name: Customer name associated with the appliance
app_name: Your application name
form_factor: Where the appliance runs (aws, gcp, azure, private-k8s, digitalocean)
vendor_version: The release version deployed to the appliance

This metadata allows you to filter, group, and correlate telemetry across customers, form factors, and versions.

Example: Filtering logs by customer

In Datadog:

service:myapp-api env:acme-corp-prod

In Grafana Loki:

{instance_id="acme-corp-prod", service="myapp-api"}

Unified dashboards

With telemetry from all appliances flowing to your observability sink, you can create unified dashboards that aggregate metrics across customers:

Deployment health: Track successful vs. failed deployments across all appliances
Performance trends: Compare response times and error rates across customers
Resource utilization: Monitor database CPU, storage usage, function execution counts
Version adoption: See which customers are running which versions

You can also create customer-specific dashboards filtered to a single instance_id for troubleshooting individual appliances.

Telemetry and customer data

Your responsibility: Tensor9 does not guarantee that your logs do not contain customer data. It is your responsibility as the vendor to ensure that your application does not log sensitive customer data (PII, financial information, proprietary content, or customer business data) that will be forwarded to your observability sink.

Observability telemetry should contain application logs and infrastructure metrics - not customer business data. While logs may include operational metadata (timestamps, user IDs, API endpoints, error codes), they should never include sensitive customer information. You must take precautions to prevent customer data from appearing in logs:

Sanitize logs: Remove or redact sensitive information before logging. Never log request/response payloads containing customer data.
Use structured logging: Log metadata and identifiers, not full payloads. Log user_id: 12345 instead of the entire user object.
Configure log levels: Use DEBUG/INFO for development, WARN/ERROR for production. Avoid verbose logging that may capture customer data.
Review what you collect: Audit what data flows to your observability sink. Test your logging to ensure no customer data leaks through.
Filter at the source: Configure log filters to exclude patterns that may contain sensitive data (credit card numbers, SSNs, API keys).

Tensor9 forwards whatever telemetry your application emits - it is your responsibility to ensure that telemetry does not contain customer data.

Observability permissions

Telemetry collection requires steady-state permissions in customer appliances. These permissions are:

Read-only: Cannot modify infrastructure or customer data
Always active: Observability runs continuously without customer approval
Scoped to vendor resources: Can only access resources deployed by your application

Example steady-state role for observability in AWS:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams",
        "logs:GetLogEvents",
        "cloudwatch:GetMetricData",
        "cloudwatch:ListMetrics"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/tensor9:instance": "${var.instance_id}"
        }
      }
    }
  ]
}

This role allows reading logs and metrics only from resources tagged with the appliance’s instance_id.

Alerting and incident response

Once telemetry flows to your observability platform, you can configure alerts that notify your team when issues occur across customer appliances:

Deployment failures: Alert when a deployment to any appliance fails
High error rates: Notify when error rates exceed thresholds
Performance degradation: Alert on slow response times or database latency
Resource exhaustion: Warn when databases approach storage limits

Alerts can include the instance_id and customer name, allowing you to quickly identify which customer is affected and route incidents to the right team.

Best practices

Never log customer data

It is your responsibility to ensure your application does not log sensitive customer data (PII, financial information, customer business data). Tensor9 forwards whatever telemetry your application emits - it does not filter or sanitize logs for customer data. Implement log sanitization in your application code, avoid logging request/response payloads, and regularly audit what data flows to your observability sink.

Tag all resources with instance_id

Ensure every resource in your origin stack is tagged with the instance_id variable. This enables filtering telemetry by appliance and ensures observability permissions are correctly scoped.

Permissions Model: Understanding steady-state permissions for observability
Appliances: Customer environments where telemetry is collected
Deployments: Tracking deployment success through observability
Operations: Using observability to inform remote operations

Tensor9

Getting Started

Fundamentals

Your Origin Stack

Your Customer's Environment

Integrations

tensor9 CLI

How observability works

What you can observe

Application logs

Infrastructure metrics

Custom metrics

Distributed traces

Observability sinks

Configuring observability

Configure your observability sink

Configure telemetry in your origin stack

Example 1: Send telemetry to CloudWatch in your control plane

Example 2: Send telemetry to Datadog via your control plane

Observability across form factors

Appliance identification

Example: Filtering logs by customer

Unified dashboards

Telemetry and customer data

Observability permissions

Alerting and incident response

Best practices

Tensor9

Getting Started

Fundamentals

Your Origin Stack

Your Customer's Environment

Integrations

tensor9 CLI

​How observability works

​What you can observe

​Application logs

​Infrastructure metrics

​Custom metrics

​Distributed traces

​Observability sinks

​Configuring observability

​Configure your observability sink

​Configure telemetry in your origin stack

​Example 1: Send telemetry to CloudWatch in your control plane

​Example 2: Send telemetry to Datadog via your control plane

​Observability across form factors

​Appliance identification

​Example: Filtering logs by customer

​Unified dashboards

​Telemetry and customer data

​Observability permissions

​Alerting and incident response

​Best practices

​Related topics

How observability works

What you can observe

Application logs

Infrastructure metrics

Custom metrics

Distributed traces

Observability sinks

Configuring observability

Configure your observability sink

Configure telemetry in your origin stack

Example 1: Send telemetry to CloudWatch in your control plane

Example 2: Send telemetry to Datadog via your control plane

Observability across form factors

Appliance identification

Example: Filtering logs by customer

Unified dashboards

Telemetry and customer data

Observability permissions

Alerting and incident response

Best practices

Related topics