A hand-drawn character turning OpenTelemetry signals into Prometheus metrics

Self-Hosting OpenTelemetry, Part 1: Metrics with Prometheus and Grafana

Yulei Chen

16 min

Metrics tell you that something changed before you have to read a single log line. A request counter can reveal a traffic spike, while a duration histogram shows whether an endpoint is gradually getting slower.

In the first part of this OpenTelemetry series, you'll instrument a small Node.js API, send its metrics to an authenticated OpenTelemetry Collector, and store them in Prometheus. Grafana gives you a place to query and graph the result.

The app only knows about OTLP, the OpenTelemetry Protocol. It does not know which metrics database sits behind the Collector. That separation becomes useful in part two, where we add logs and Loki, and part three, where we add traces and Tempo.

What you'll deploy

Node.js app -- OTLP/HTTP + bearer token --> OpenTelemetry Collector
                                                   |
                                                   | batched OTLP/HTTP
                                                   v
                                              Prometheus
                                                   ^
                                                   |
                                                Grafana

Service	Access	Purpose
Node.js app	Public	Produces test traffic and OpenTelemetry metrics
OpenTelemetry Collector	Private	Authenticates, buffers, and routes telemetry
Prometheus	Private	Stores and queries metrics
Grafana	Public	Visualizes the metrics

Deploy every service on the same Sliplane server. Services on the same server can reach each other, while the Collector and Prometheus remain unavailable from the public internet.

This tutorial uses Prometheus's native OTLP receiver instead of making the Collector expose a scrape endpoint. The app pushes to the Collector, the Collector batches the data, and Prometheus receives it over OTLP.

1. Create the repository

Create a GitHub repository named otel-demo. We’ll build the following project structure step by step:

otel-demo/
├── .gitignore
├── app/
│   ├── Dockerfile
│   ├── app.ts
│   ├── instrumentation.ts
│   ├── package-lock.json
│   └── package.json
├── collector/
│   ├── Dockerfile
│   └── collector-config.yaml
└── prometheus/
    ├── Dockerfile
    └── prometheus.yml

We'll add loki/ and tempo/ later in the series. Keeping each service in its own directory lets you select a different Dockerfile and context directory for each Sliplane service.

2. Instrument the Node.js app

Create app/package.json:

app/package.json

{
  "name": "otel-demo-api",
  "version": "1.0.0",
  "private": true,
  "scripts": {
    "start": "tsx --import ./instrumentation.ts app.ts"
  },
  "dependencies": {
    "@opentelemetry/api": "1.9.1",
    "@opentelemetry/exporter-metrics-otlp-proto": "0.222.0",
    "@opentelemetry/resources": "2.11.0",
    "@opentelemetry/sdk-metrics": "2.11.0",
    "@opentelemetry/sdk-node": "0.222.0",
    "tsx": "4.23.15"
  }
}

This follows the official OpenTelemetry TypeScript example: tsx runs the TypeScript files, and Node's --import flag loads the instrumentation first. The versions are pinned so a future package release does not silently change the example.

From the app directory, generate and commit the lockfile:

npm install

Next, create app/instrumentation.ts:

app/instrumentation.ts

import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-proto';
import { defaultResource, resourceFromAttributes } from '@opentelemetry/resources';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { NodeSDK } from '@opentelemetry/sdk-node';

const endpoint = (process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4318')
  .replace(/\/$/, '');
const token = process.env.OTEL_AUTH_TOKEN;

if (!token) {
  throw new Error('OTEL_AUTH_TOKEN is required');
}

const resource = defaultResource().merge(resourceFromAttributes({
  'service.name': process.env.OTEL_SERVICE_NAME || 'otel-demo-api',
  'service.version': process.env.npm_package_version || '1.0.0',
  'deployment.environment.name': process.env.NODE_ENV || 'development',
}));

const metricReader = new PeriodicExportingMetricReader({
  exporter: new OTLPMetricExporter({
    url: `${endpoint}/v1/metrics`,
    headers: { Authorization: `Bearer ${token}` },
  }),
  exportIntervalMillis: 5000,
});

const sdk = new NodeSDK({
  resource,
  metricReaders: [metricReader],
});

sdk.start();

process.once('SIGTERM', () => {
  sdk.shutdown()
    .then(() => process.exit(0))
    .catch((error) => {
      console.error('OpenTelemetry shutdown failed', error);
      process.exit(1);
    });
});

The exporter sends a batch every five seconds. Its endpoint includes /v1/metrics, as required by the OTLP HTTP exporter. The tsx --import command starts this SDK before it loads the application code. OpenTelemetry must initialize first so libraries obtain real telemetry providers instead of no-op implementations.

Create app/app.ts:

app/app.ts

import { type Attributes, metrics } from '@opentelemetry/api';
import { createServer } from 'node:http';

const meter = metrics.getMeter('otel-demo-api');
const requests = meter.createCounter('demo_http_requests', {
  description: 'Number of HTTP requests handled by the demo app',
});
const duration = meter.createHistogram('demo_http_request_duration_milliseconds', {
  description: 'HTTP request duration in milliseconds',
});

const server = createServer(async (request, response) => {
  const startedAt = performance.now();
  const url = new URL(request.url ?? '/', `http://${request.headers.host ?? 'localhost'}`);
  const route = url.pathname === '/work' ? '/work' : url.pathname;
  let statusCode = 200;

  if (url.pathname === '/health') {
    response.writeHead(200).end('ok');
  } else if (url.pathname === '/work') {
    await new Promise((resolve) => setTimeout(resolve, 50 + Math.random() * 250));
    statusCode = url.searchParams.get('fail') === 'true' ? 500 : 200;
    response.writeHead(statusCode, { 'content-type': 'application/json' });
    response.end(JSON.stringify({ ok: statusCode === 200 }));
  } else {
    statusCode = 404;
    response.writeHead(statusCode).end('not found');
  }

  const attributes: Attributes = {
    'http.route': route,
    'http.response.status_code': statusCode,
  };
  requests.add(1, attributes);
  duration.record(performance.now() - startedAt, attributes);
});

const port = Number(process.env.PORT || 3000);
server.listen(port, '0.0.0.0', () => {
  console.log(`Demo API listening on port ${port}`);
});

The two attributes have a small, predictable set of values. Avoid putting user IDs, raw URLs, or request IDs on metrics: every distinct label combination creates another time series.

Finally, create app/Dockerfile:

app/Dockerfile

FROM node:24.15.0-alpine

WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY app.ts instrumentation.ts ./

ENV NODE_ENV=production
EXPOSE 3000
CMD ["npm", "start"]

The container keeps tsx as a runtime dependency because it runs the .ts files directly.

3. Configure the Collector

Create collector/collector-config.yaml:

collector/collector-config.yaml

extensions:
  bearertokenauth/otlp:
    token: ${env:OTEL_AUTH_TOKEN}
  health_check:
    endpoint: 0.0.0.0:13133

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
        auth:
          authenticator: bearertokenauth/otlp

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 256
    spike_limit_mib: 64
  batch:
    timeout: 5s
    send_batch_size: 1024

exporters:
  otlphttp/prometheus:
    endpoint: ${env:PROMETHEUS_OTLP_ENDPOINT}
    tls:
      insecure: true

service:
  extensions: [bearertokenauth/otlp, health_check]
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/prometheus]

The bearer-token authenticator rejects requests without the shared token. The memory limiter protects the Collector under pressure, while the batch processor groups small exports before sending them to Prometheus. These components and their ordering follow the Collector resiliency guidance.

Create collector/Dockerfile:

collector/Dockerfile

FROM otel/opentelemetry-collector-contrib:0.161.0

COPY collector-config.yaml /etc/otelcol-contrib/config.yaml
CMD ["--config=/etc/otelcol-contrib/config.yaml"]

We use the Collector's contrib distribution because it includes the bearer-token authentication extension.

4. Enable Prometheus's OTLP receiver

Create prometheus/prometheus.yml:

prometheus/prometheus.yml

global:
  scrape_interval: 15s
  evaluation_interval: 15s

storage:
  tsdb:
    out_of_order_time_window: 30m

otlp:
  promote_resource_attributes:
    - service.name
    - service.version
    - deployment.environment.name

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]

The out_of_order_time_window setting is recommended for OTLP ingestion because Collector retries can deliver older samples after newer ones. Promoting a small set of resource attributes makes fields such as service.name available as Prometheus labels. Prometheus translates dots to underscores, so service.name becomes service_name.

Create prometheus/Dockerfile:

prometheus/Dockerfile

FROM prom/prometheus:v3.14.0

COPY prometheus.yml /etc/prometheus/prometheus.yml
RUN /bin/promtool check config /etc/prometheus/prometheus.yml

CMD ["--config.file=/etc/prometheus/prometheus.yml", "--storage.tsdb.path=/prometheus", "--storage.tsdb.retention.time=15d", "--storage.tsdb.retention.size=5GB", "--web.listen-address=0.0.0.0:9090", "--web.enable-otlp-receiver"]

--web.enable-otlp-receiver activates Prometheus's OTLP HTTP endpoint. Pinning the image tag and validating the configuration during the build makes upgrades deliberate.

Before committing, create a .gitignore in the repository root so dependencies and local secrets are not added to Git:

.gitignore

node_modules/
.env
.env.*
.DS_Store

Commit and push all files before you start deploying.

5. Deploy Prometheus

In Sliplane, create a service named prometheus-example from the GitHub repository. Open Advanced Settings to configure the Dockerfile path and context directory, then use:

Setting	Value
Service name	`prometheus-example`
Dockerfile path	`prometheus/Dockerfile`
Context directory	`prometheus`
Public access	Disabled
`PORT` environment variable	`9090`
Persistent volume	Mount at `/prometheus`

Deploy it, then copy its internal hostname. We use prometheus-example.internal below; replace it with the value shown in your service settings.

The volume preserves the time-series database between deployments. The 5 GB retention setting is a budget for stored blocks rather than a strict cap on all disk use, so leave room for the write-ahead log and compaction.

6. Deploy the Collector

Create another GitHub service named collector-example on the same server. Open Advanced Settings to configure the Dockerfile path and context directory:

Setting	Value
Service name	`collector-example`
Dockerfile path	`collector/Dockerfile`
Context directory	`collector`
Public access	Disabled

Generate an authentication token. You will use the same value for the Collector and the app:

openssl rand -hex 32

Add these environment variables:

PORT=13133
OTEL_AUTH_TOKEN=replace-with-the-generated-token
PROMETHEUS_OTLP_ENDPOINT=http://prometheus-example.internal:9090/api/v1/otlp

Mark OTEL_AUTH_TOKEN as secret. The exporter appends /v1/metrics to the base endpoint, producing Prometheus's full /api/v1/otlp/v1/metrics path.

Once the Collector is healthy, copy its internal hostname for the app configuration.

The shared token travels over HTTP inside Sliplane's private network in this tutorial. Keep both services on the same server. If telemetry crosses an untrusted network or you expose the Collector publicly, terminate TLS and use a production-grade authentication gateway instead.

7. Deploy the app

Create the third GitHub service as app-example. Open Advanced Settings to configure the Dockerfile path and context directory:

Setting	Value
Service name	`app-example`
Dockerfile path	`app/Dockerfile`
Context directory	`app`
Public access	Enabled
Health check	`/health`

Add the app's environment variables:

PORT=3000
OTEL_EXPORTER_OTLP_ENDPOINT=http://collector-example.internal:4318
OTEL_AUTH_TOKEN=replace-with-the-same-generated-token
OTEL_SERVICE_NAME=otel-demo-api
OTEL_TRACES_EXPORTER=none

Mark the token as secret. OTEL_TRACES_EXPORTER=none prevents the Node SDK from creating an unused default trace exporter in this metrics-only part of the series.

After deployment, create a few successful and failed requests from your terminal:

for i in {1..20}; do curl -s https://app-example.sliplane.app/work > /dev/null; done
for i in {1..5}; do curl -s https://app-example.sliplane.app/work?fail=true > /dev/null; done

Wait at least five seconds for the first export batch.

8. Connect Grafana to Prometheus

Deploy the Grafana preset on the same server. Keep its volume mounted at /var/lib/grafana, then open its public Sliplane domain.

Deploy Grafana >

Sign in to Grafana with the default username admin and password admin. Grafana will ask you to change the password after your first login.

Then:

Open Connections > Data sources.
Add a Prometheus data source.
Set its URL to http://prometheus-example.internal:9090.
Select Save & test.

Grafana makes this request from its server, so the browser does not need direct access to Prometheus.

Open Explore, select the new data source, and try these PromQL queries.

Total requests:

demo_http_requests_total{service_name="otel-demo-api"}

Requests per second by route and status:

sum by (http_route, http_response_status_code) (
  rate(demo_http_requests_total{service_name="otel-demo-api"}[5m])
)

95th-percentile request duration:

histogram_quantile(
  0.95,
  sum by (le, http_route) (
    rate(demo_http_request_duration_milliseconds_bucket{service_name="otel-demo-api"}[5m])
  )
)

Create a Grafana dashboard and turn each query into a time-series panel. You now have an application emitting metrics through a reusable OpenTelemetry pipeline.

Troubleshooting

The query returns no data

Wait for the app's five-second export interval.
Confirm that the app and Collector use the same token.
Check that the app uses the Collector's internal hostname, not localhost.
Confirm PROMETHEUS_OTLP_ENDPOINT ends in /api/v1/otlp, without /v1/metrics.
Inspect the app and Collector deployment logs for 401, 404, or export errors.

Prometheus rejects out-of-order samples

Confirm that storage.tsdb.out_of_order_time_window is present and that Prometheus was redeployed after the configuration change.

Labels or metric names look different

OTLP-to-Prometheus translation normalizes characters and adds Prometheus suffixes. For example, the monotonic counter demo_http_requests appears as demo_http_requests_total, and resource attribute service.name becomes service_name when promoted.

Next: send logs to Loki

The Collector can receive all three telemetry signals on the same authenticated endpoint. In part two, we'll add the OpenTelemetry Logs SDK, route log records to Loki, and query them in Grafana without changing the application's network architecture.

Self-Hosting OpenTelemetry, Part 1: Metrics with Prometheus and Grafana

Deploy your observability stack on Sliplane