OTEL E2E Testing

Run OTEL E2E tests to validate trace and metrics export from all Porch components

This guide explains how to run the automated OTEL E2E test locally and in CI to validate that Porch components (porch-server, porch-controllers, porch-function-runner) correctly export traces and metrics to observability backends.

Overview

The OTEL E2E testing validates:

  • ✅ All Porch components export traces to Jaeger
  • ✅ All components export metrics to Prometheus
  • ✅ OTEL SDK initializes without errors
  • ✅ Real Porch operations are properly instrumented

Scope: v1alpha2 CRD test suite

  • Both v1alpha1 and v1alpha2 APIs use the same underlying server and controller components (porch-server, porch-controllers, function-runner)
  • OTEL instrumentation is at the component level, not the API layer
  • A single v1alpha2 test suite is sufficient to validate telemetry export from all shared components
  • Uses lightweight “Init” specs and one render test to exercise core operations efficiently

Prerequisites

  • Running Kind cluster with Porch deployed (see Local Development Environment)
  • kubectl configured to access the cluster
  • jq command-line JSON processor (for validation output)

Resource Requirements

Note on resource bumps: Porch components have increased resource allocations to accommodate OTEL telemetry collection:

  • porch-server: 500m CPU request, 512Mi memory request (was lower; increased to prevent timeout issues during telemetry export)
  • porch-function-runner: Similar increases to ensure stable operation while exporting traces/metrics

These resource changes apply system-wide (not just OTEL tests) to provide stable baseline capacity for observability operations. If tighter resource constraints are needed for your environment, adjust deployments/porch/*.yaml before deploying.

Local Testing

Run OTEL E2E Test

Single command deploys Porch, monitoring stack, runs E2E test, and validates OTEL:

make test-e2e-otel

This:

  1. Deploys v1alpha2 stack with DB cache
  2. Deploys monitoring (Prometheus, Grafana, Jaeger)
  3. Runs lightweight E2E test (~20-30 sec) to exercise components
  4. Validates traces/metrics from all 3 components
  5. Leaves deployment running for manual debugging (see Access Observability UIs)

Total time: ~25-30 minutes

Validation Steps

The validate-otel.sh script checks:

  1. ✅ Jaeger pod is running
  2. ✅ porch-server metrics endpoint responds (port 9464)
  3. ✅ OTEL initialized in logs (no critical errors)
  4. ✅ Traces exist in Jaeger (with retries)
  5. ✅ Prometheus has scraped all 3 components
  6. ✅ Lists all components exporting traces

Example output:

=== OTEL Validation Checks ===
=== 1. Checking Jaeger availability ===
✓ Jaeger pod found: jaeger-66f5c97cb8-skllc
=== 2. Checking porch-server metrics endpoint ===
✓ Metrics endpoint responding
=== 6. Services reporting traces to Jaeger ===
✓ Porch components exporting traces:
  - porch-server
  - porch-controllers
  - porch-function-runner

Access Observability UIs

While the test stack is running, access services via port-forward:

Jaeger (Distributed Tracing)

# Terminal 1
kubectl port-forward -n porch-monitoring deployment/jaeger 16686:16686

# Terminal 2 - visit http://localhost:16686
# Query service: "porch-server", "porch-controllers", or "porch-function-runner"

Prometheus (Metrics)

# Terminal 1
kubectl port-forward -n porch-monitoring deployment/prometheus 9092:9090

# Terminal 2 - visit http://localhost:9092
# Query: http_server_requests_total, grpc_server_requests_total

Grafana (Dashboards)

# Terminal 1
kubectl port-forward -n porch-monitoring deployment/grafana 3001:3000

# Terminal 2 - visit http://localhost:3001
# Default credentials:
# User: porch
# Password: retrieve from secret using:
#   kubectl get secret -n porch-monitoring grafana-admin-creds \
#     -o jsonpath='{.data.password}' | base64 -d

CI Integration

GitHub Actions Workflow

The OTEL E2E tests run automatically in GitHub Actions.

File: .github/workflows/porch-e2e-otel-weekly.yaml

Triggers:

  • Scheduled: Every Sunday at 2 AM UTC
  • Manual: Trigger via GitHub Actions UI (workflow_dispatch)

What it does:

  1. Builds Porch images (parallel)
  2. Deploys v1alpha2 stack
  3. Deploys monitoring
  4. Runs lightweight E2E test
  5. Validates OTEL infrastructure
  6. Reports which components export traces/metrics

Duration: ~45 minutes total

Test Selection

Init and Render tests (v1alpha2 CRD):

  • Operations: Create/initialize packages + simple render pipeline
  • Why: Lightweight, exercises core package operations and function-runner traces, no heavy rendering overhead
  • Duration: ~30-45 seconds
  • Traces generated: API calls, package initialization, status updates, function-runner invocations
  • Function-runner coverage: Render pipeline test invokes the set-namespace KRM function, generating function-runner traces

This test suite is sufficient to verify that:

  • All components are healthy and receiving traffic
  • All components export traces and metrics
  • OTEL pipeline works end-to-end
  • Server-level and function-runner instrumentation is working

Make Targets

# Deploy + test + validate + leave running for debugging
make test-e2e-otel

Troubleshooting

No traces in Jaeger

  1. Check Porch components are running:

    kubectl get pods -n porch-system

  2. Check OTEL env vars are set:

    kubectl get deployment porch-server -n porch-system \
  -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="OTEL_EXPORTER_OTLP_TRACES_ENDPOINT")].value}'

    Should output: http://jaeger-otlp.porch-monitoring.svc.cluster.local:4317

  3. Check Jaeger OTLP endpoint is reachable:

    kubectl logs -n porch-system deployment/porch-server | grep -i "jaeger\|otlp\|export"

Metrics endpoint not responding

  1. Check porch-server pod is healthy:

    kubectl get pod -n porch-system -l app=porch-server
kubectl logs -n porch-system -l app=porch-server | grep "ERROR\|error"

  2. Verify port-forward is working:

    kubectl port-forward -n porch-system deployment/porch-server 9464:9464 &
curl http://localhost:9464/metrics | head -10

Validation script fails

  • Prometheus metrics query fails: Metrics may still be collecting. Check manually at http://localhost:9092 (Prometheus UI)
  • No services in Jaeger: E2E test may not have exercised components. Check test logs for errors
  • Export errors in logs: Usually transient (connection timeout during startup). Check if components eventually recover

Cleanup

# Remove monitoring stack
./scripts/monitoring/deploy-monitoring.sh remove

# Destroy Porch deployment
make destroy

Next Steps

Last modified October 13, 2026: Add OTEL E2E testing infrastructure (#1224) (5498d947)