A hand-drawn character weaving OpenTelemetry log records into Loki

Self-Hosting OpenTelemetry, Part 2: Logs with Loki and Grafana

Yulei Chen - Content-Engineerin bei sliplane.ioYulei Chen
15 min

Plain-text logs are easy to write and surprisingly hard to search. Structured logs give every record consistent fields, such as the route, response status, and service name, so you can filter them without parsing sentences.

In part one of this series, we sent Node.js metrics through an authenticated OpenTelemetry Collector to Prometheus. This tutorial adds OpenTelemetry logs to the same app and Collector, then stores them in Loki.

                                     +--> Prometheus (metrics)
Node.js app -- OTLP/HTTP --> Collector
                                     +--> Loki (logs)
                                              ^
                                              |
                                           Grafana

The OpenTelemetry JavaScript logs API and SDK are still marked as development, while JavaScript traces and metrics are stable. Pin the package versions, test upgrades, and keep that maturity difference in mind before relying on this exact logging integration in production.

Before you start

Complete the metrics tutorial first. You should already have this repository layout and four running services:

otel-demo/
├── app/
├── collector/
└── prometheus/

The app, Collector, Prometheus, and Grafana must remain on the same Sliplane server. This tutorial adds a private Loki service and reuses the Collector's OTLP HTTP endpoint and bearer token.

1. Add the OpenTelemetry logging packages

In the repository's app directory, install the logging API, SDK, and OTLP exporter:

npm install --save-exact \
  @opentelemetry/api-logs@0.222.0 \
  @opentelemetry/exporter-logs-otlp-proto@0.222.0 \
  @opentelemetry/sdk-logs@0.222.0

Replace app/instrumentation.ts with this combined metrics-and-logs setup:

app/instrumentation.ts
import { OTLPLogExporter } from '@opentelemetry/exporter-logs-otlp-proto';import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-proto';
import { defaultResource, resourceFromAttributes } from '@opentelemetry/resources';
import { BatchLogRecordProcessor } from '@opentelemetry/sdk-logs';import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { NodeSDK } from '@opentelemetry/sdk-node';

const endpoint = (process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4318')
  .replace(/\/$/, '');
const token = process.env.OTEL_AUTH_TOKEN;

if (!token) {
  throw new Error('OTEL_AUTH_TOKEN is required');
}

const headers = { Authorization: `Bearer ${token}` };const resource = defaultResource().merge(resourceFromAttributes({
  'service.name': process.env.OTEL_SERVICE_NAME || 'otel-demo-api',
  'service.version': process.env.npm_package_version || '1.0.0',
  'deployment.environment.name': process.env.NODE_ENV || 'development',
}));

const metricReader = new PeriodicExportingMetricReader({
  exporter: new OTLPMetricExporter({
    url: `${endpoint}/v1/metrics`,
    headers,  }),
  exportIntervalMillis: 5000,
});

const logProcessor = new BatchLogRecordProcessor({  exporter: new OTLPLogExporter({    url: `${endpoint}/v1/logs`,    headers,  }),});
const sdk = new NodeSDK({
  resource,
  metricReaders: [metricReader],
  logRecordProcessors: [logProcessor],});

sdk.start();

process.once('SIGTERM', () => {
  sdk.shutdown()
    .then(() => process.exit(0))
    .catch((error) => {
      console.error('OpenTelemetry shutdown failed', error);
      process.exit(1);
    });
});

Both exporters use the same Collector base address and credentials, but each uses its signal-specific OTLP path.

2. Emit structured log records

Replace app/app.ts with this version:

app/app.ts
import { type Attributes, metrics } from '@opentelemetry/api';
import { logs, SeverityNumber } from '@opentelemetry/api-logs';import { createServer } from 'node:http';

const meter = metrics.getMeter('otel-demo-api');
const logger = logs.getLogger('otel-demo-api');
const requests = meter.createCounter('demo_http_requests', {
  description: 'Number of HTTP requests handled by the demo app',
});
const duration = meter.createHistogram('demo_http_request_duration_milliseconds', {
  description: 'HTTP request duration in milliseconds',
});

const server = createServer(async (request, response) => {
  const startedAt = performance.now();
  const url = new URL(request.url ?? '/', `http://${request.headers.host ?? 'localhost'}`);
  const route = url.pathname === '/work' ? '/work' : url.pathname;
  let statusCode = 200;

  if (url.pathname === '/health') {
    response.writeHead(200).end('ok');
  } else if (url.pathname === '/work') {
    await new Promise((resolve) => setTimeout(resolve, 50 + Math.random() * 250));
    statusCode = url.searchParams.get('fail') === 'true' ? 500 : 200;
    response.writeHead(statusCode, { 'content-type': 'application/json' });
    response.end(JSON.stringify({ ok: statusCode === 200 }));
  } else {
    statusCode = 404;
    response.writeHead(statusCode).end('not found');
  }

  const elapsed = performance.now() - startedAt;
  const attributes: Attributes = {
    'http.route': route,
    'http.response.status_code': statusCode,
  };

  requests.add(1, attributes);
  duration.record(elapsed, attributes);

  const failed = statusCode >= 500;  logger.emit({    severityNumber: failed ? SeverityNumber.ERROR : SeverityNumber.INFO,    severityText: failed ? 'ERROR' : 'INFO',    body: failed ? 'Request failed' : 'Request completed',    attributes: {      ...attributes,      'http.request.method': request.method ?? 'UNKNOWN',      'http.server.request.duration_ms': Math.round(elapsed),    },  });});

const port = Number(process.env.PORT || 3000);
server.listen(port, '0.0.0.0', () => {
  console.log(`Demo API listening on port ${port}`);
});

The body remains readable, while the attributes hold the fields you will filter and aggregate. Notice that the log contains the route template rather than the full URL. This avoids creating a distinct value for every query string or resource ID.

The console.log startup message is not exported by this code. OpenTelemetry's log API emits explicit structured records; it does not automatically replace or capture console output.

3. Add Loki to the repository

Add a loki directory:

otel-demo/
├── app/
├── collector/
├── loki/
│   ├── Dockerfile
│   └── loki-config.yaml
└── prometheus/

Create loki/loki-config.yaml:

loki/loki-config.yaml
auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory
  replication_factor: 1

schema_config:
  configs:
    - from: 2024-04-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

storage_config:
  filesystem:
    directory: /loki/chunks

compactor:
  working_directory: /loki/compactor
  retention_enabled: true
  delete_request_store: filesystem

limits_config:
  allow_structured_metadata: true
  retention_period: 168h

Loki's native OTLP endpoint stores selected resource attributes as indexed labels and keeps the other OpenTelemetry attributes as structured metadata. TSDB schema v13 and structured metadata are required for this path. The from date must be in the past, so do not replace it with a future deployment date.

This is a single-binary, filesystem-backed configuration with seven days of retention. It is appropriate for learning and a small single-node setup. Grafana recommends object storage for production Loki deployments that need stronger durability or horizontal scaling.

Create loki/Dockerfile:

loki/Dockerfile
FROM grafana/loki:3.7.8

COPY loki-config.yaml /etc/loki/local-config.yaml
CMD ["-config.file=/etc/loki/local-config.yaml"]

4. Route logs through the Collector

Replace collector/collector-config.yaml with this combined configuration:

collector/collector-config.yaml
extensions:
  bearertokenauth/otlp:
    token: ${env:OTEL_AUTH_TOKEN}
  health_check:
    endpoint: 0.0.0.0:13133

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
        auth:
          authenticator: bearertokenauth/otlp

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 256
    spike_limit_mib: 64
  batch:
    timeout: 5s
    send_batch_size: 1024

exporters:
  otlphttp/prometheus:
    endpoint: ${env:PROMETHEUS_OTLP_ENDPOINT}
    tls:
      insecure: true
  otlphttp/loki:    endpoint: ${env:LOKI_OTLP_ENDPOINT}    tls:      insecure: true
service:
  extensions: [bearertokenauth/otlp, health_check]
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/prometheus]
    logs:      receivers: [otlp]      processors: [memory_limiter, batch]      exporters: [otlphttp/loki]

The Collector's OTLP HTTP exporter appends /v1/logs, so Loki receives records at /otlp/v1/logs. This is Loki's recommended native OpenTelemetry ingestion path; an old, Loki-specific Collector exporter is no longer necessary.

Commit and push the Loki files and the app and Collector changes.

5. Deploy Loki on Sliplane

Create a service named loki-example from the same GitHub repository. Open Advanced Settings to configure the Dockerfile path and context directory:

SettingValue
Service nameloki-example
Dockerfile pathloki/Dockerfile
Context directoryloki
ServerThe same server as the other services
Public accessDisabled
PORT environment variable3100
Persistent volumeMount at /loki

Deploy Loki and copy its internal hostname. The remaining examples use loki-example.internal as a placeholder.

Loki has no authentication in this configuration because it is only reachable through the private network. Do not expose this service publicly without adding an authenticating reverse proxy.

6. Update the Collector and app services

Add Loki's internal endpoint to the Collector service:

LOKI_OTLP_ENDPOINT=http://loki-example.internal:3100/otlp

Redeploy the Collector, then redeploy the app with its new lockfile and source code. Keep OTEL_TRACES_EXPORTER=none for now—we enable tracing in part three.

7. Generate and inspect logs

Send another mix of successful and failed requests:

for i in {1..20}; do curl -s https://app-example.sliplane.app/work > /dev/null; done
for i in {1..5}; do curl -s https://app-example.sliplane.app/work?fail=true > /dev/null; done

In Grafana, add Loki as a data source:

  1. Open Connections > Data sources.
  2. Add a Loki data source.
  3. Set its URL to http://loki-example.internal:3100.
  4. Select Save & test.

Open Explore, choose Loki, and run:

{service_name="otel-demo-api"}

Loki converts the OpenTelemetry resource attribute service.name to the indexed label service_name. Filter down to failed requests with:

{service_name="otel-demo-api"} | severity_text = "ERROR"

Count errors over time:

sum(count_over_time({service_name="otel-demo-api"} | severity_text = "ERROR" [5m]))

Expand a result in Explore to inspect its structured metadata, including http_route, http_response_status_code, and http_server_request_duration_ms. Loki normalizes dots in OpenTelemetry attribute names to underscores.

Keep labels under control

Loki labels build its stream index. Low-cardinality fields such as service name, environment, and severity can be useful labels. High-cardinality fields such as request IDs, user IDs, and raw URLs should remain structured metadata.

The native OTLP mapping already makes conservative choices. Resist the temptation to promote every attribute to a label: too many streams increase memory use, storage overhead, and query cost.

Also decide what must never reach Loki. Redact passwords, authorization headers, session tokens, personal data, and secrets in the application before emission. A private endpoint protects transport access; it does not make sensitive log content safe to store.

Troubleshooting

Loki is healthy but Explore shows no streams

  • Confirm the app was redeployed after the logging packages were committed.
  • Check the Collector logs for errors from the otlphttp/loki exporter.
  • Ensure LOKI_OTLP_ENDPOINT ends in /otlp, not /otlp/v1/logs.
  • Generate new requests and widen Grafana's time range.

Loki cannot write to its volume

Confirm the persistent volume is mounted at /loki, then inspect the Loki service logs for the exact filesystem error.

A field does not appear as a label

That is usually expected. Most OpenTelemetry attributes are structured metadata rather than indexed labels. Expand the log record in Explore or use a structured-metadata filter instead of putting every field in the stream selector.

Next: correlate logs with traces

Metrics show that requests are failing and logs add the error context, but neither shows the complete request path. In part three, we'll add automatic and manual Node.js spans, send them to Tempo, and jump from a Loki log record to its trace in Grafana.

Deploy your observability stack on Sliplane

Run your app, Collector, Loki, Prometheus, and Grafana together with private service-to-service networking.