Skip to content

A complete local observability stack (LGTM: Loki, Grafana, Tempo, Mimir/Prometheus) with OpenTelemetry Collector

License

Notifications You must be signed in to change notification settings

justinabrahms/lgtm

Repository files navigation

LGTM Observability Stack

Docker Docker Compose

A complete observability stack using Docker Compose with Loki (logs), Grafana (visualization), Tempo (traces), and Prometheus (metrics), plus OpenTelemetry Collector for receiving OTEL data.

Services

Quick Start

Using the runfile (recommended)

# Start the stack (with automatic port conflict detection)
./runfile start

# View logs
./runfile logs

# Check status
./runfile status

# Stop the stack
./runfile stop

# Clean everything (removes all data)
./runfile clean

# Check for port conflicts without starting
./runfile check

Using docker-compose directly

# Start the stack
docker-compose up -d

# View logs
docker-compose logs -f

# Stop the stack
docker-compose down

# Stop and remove volumes (clean slate)
docker-compose down -v

Access Grafana

Open http://localhost:3000 in your browser. Authentication is disabled for local development.

All data sources (Prometheus, Tempo, Loki) are pre-configured and ready to use.

Sending OTEL Data

From Your Application

Configure your application to send OpenTelemetry data to:

gRPC endpoint: localhost:4317 HTTP endpoint: http://localhost:4318

Example: Node.js

const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { OTLPMetricExporter } = require('@opentelemetry/exporter-metrics-otlp-grpc');

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: 'grpc://localhost:4317',
  }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: 'grpc://localhost:4317',
    }),
  }),
});

sdk.start();

Example: Python

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(endpoint="localhost:4317", insecure=True)
    )
)

Example: Go

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/sdk/trace"
)

exporter, _ := otlptracegrpc.New(
    context.Background(),
    otlptracegrpc.WithEndpoint("localhost:4317"),
    otlptracegrpc.WithInsecure(),
)

tp := trace.NewTracerProvider(
    trace.WithBatcher(exporter),
)
otel.SetTracerProvider(tp)

Using curl (for testing)

# Send a test trace via HTTP
curl -X POST http://localhost:4318/v1/traces \
  -H "Content-Type: application/json" \
  -d '{
    "resourceSpans": [{
      "resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "test-service"}}]},
      "scopeSpans": [{
        "spans": [{
          "traceId": "5B8EFFF798038103D269B633813FC60C",
          "spanId": "EEE19B7EC3C1B174",
          "name": "test-span",
          "startTimeUnixNano": "1544712660000000000",
          "endTimeUnixNano": "1544712661000000000"
        }]
      }]
    }]
  }'

Docker Network Access

If your application is running in another Docker container, use the Docker network name lgtm_default and connect to:

  • otel-collector:4317 (gRPC)
  • otel-collector:4318 (HTTP)

Or add your containers to the same network:

# In your application's docker-compose.yml
networks:
  default:
    external:
      name: lgtm_lgtm

Exploring Data in Grafana

  1. Traces: Go to Explore → Select "Tempo" datasource → Search for traces
  2. Metrics: Go to Explore → Select "Prometheus" datasource → Query metrics
  3. Logs: Go to Explore → Select "Loki" datasource → Query logs
  4. Create Dashboards: Create custom dashboards combining all data sources

Troubleshooting

Check if services are running

docker-compose ps

View OTEL Collector logs

docker-compose logs otel-collector

Test OTEL Collector connectivity

curl http://localhost:8888/metrics

Reset everything

docker-compose down -v
docker-compose up -d

File Structure

.
├── docker-compose.yml           # Main compose file
├── otel-collector-config.yml    # OTEL Collector configuration
├── tempo.yml                    # Tempo configuration
├── loki-config.yaml             # Loki configuration
├── prometheus.yml               # Prometheus configuration
├── grafana-datasources.yml      # Grafana datasources
├── runfile                      # Convenience script for managing the stack
├── test-logs.sh                 # Test script for sending logs
├── LICENSE                      # MIT License
└── README.md                    # This file

Environment Variables

You can customize ports and settings by creating a .env file:

GRAFANA_PORT=3000
PROMETHEUS_PORT=9090
TEMPO_PORT=3200
LOKI_PORT=3100
OTLP_GRPC_PORT=4317
OTLP_HTTP_PORT=4318

Next Steps

  • Configure your applications to send OTEL data to the collector
  • Create Grafana dashboards for your services
  • Set up alerts in Grafana based on metrics
  • Add more scrapers to Prometheus configuration if needed

About

A complete local observability stack (LGTM: Loki, Grafana, Tempo, Mimir/Prometheus) with OpenTelemetry Collector

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 2

  •  
  •  

Languages