Realtime scalable map using Golang and Kafka. Other tools: Redis, gRPC, WebSocket, GORM, PostgreSQL

• Overview
• Features
• Project Structure
• Architecture
• Getting Started
  • Prerequisites
  • Installation
  • Usage
  • Testing
• API Documentation
• Project Roadmap
• Contributing
• License

Realtime Map is a scalable, distributed system for real-time location tracking and visualization. The system processes location data from IoT devices (OBU - On-Board Units) through a Kafka-based streaming pipeline, stores it in Redis for fast retrieval, and provides real-time visualization through WebSocket connections.

The architecture leverages microservices pattern with separate components for data ingestion, processing, storage, analytics, and presentation. Built with Go, it provides high-performance real-time location tracking suitable for fleet management, logistics, and IoT applications. The analytics service persistently stores location history in PostgreSQL with PostGIS support for geospatial queries and analysis.

• Real-time Location Tracking: Process GPS coordinates from multiple devices simultaneously
• Scalable Architecture: Microservices-based design with horizontal scaling capabilities
• High-Performance Data Processing: Kafka-based streaming for handling high-throughput location data
• Fast Data Retrieval: Redis clustering for sub-millisecond location lookups
• Real-time Visualization: WebSocket-powered live map updates using Leaflet.js
• gRPC API: High-performance API for device communication and data exchange
• RESTful Gateway: HTTP/REST API gateway for easy integration
• Multi-Device Support: Handle thousands of concurrent device connections
• Data Persistence: PostgreSQL with PostGIS for historical data storage and geospatial analytics
• Analytics Service: Dedicated microservice for consuming, processing, and storing location data with database migrations
• Containerized Deployment: Docker Compose setup for easy development and deployment

Devices -> Kafka Topic (obu_positions)  ──┐
                                         │
                                 ┌───────▼────────┐
                                 │  Consumer Pool │  (group of Go workers)
                                 └───────┬────────┘
                                         │
         ┌───────────────────────────────┼──────────────────────────────┐
         │                               │                              │
   Update Redis shard A             Update Redis shard B           Publish to Fan-out
   key: obu:{device_id}              key: obu:{device_id}            (optional)
   (Redis Cluster)                   (Redis Cluster)                 └─> internal pubsub (Redis/ NATS)
                                                                   ┌─> gRPC/WebSocket Servers (stateless, horizontal)
                                                                   └─> Optional: Kafka topic for derived events
Clients connect:
  1) On connect: gRPC server reads snapshot from Redis (get keys / mget / scan by hash tag)
  2) After snapshot: client subscribes to live updates (server pushes) or connects to stream

• Seeder Service: Generates simulated GPS data from multiple devices for testing
• Producer Service: Consumes location data from Kafka, processes it, and stores in Redis
• Analytics Service: Consumes location data from Kafka, processes it, and stores in PostgreSQL with PostGIS for historical analytics and geospatial queries
• API Gateway: Provides gRPC, REST, and WebSocket APIs with web interface
• Go Commons: Shared protocol buffers, database utilities, and common libraries

1. Data Ingestion: Devices send GPS coordinates via gRPC to API Gateway
2. Message Queuing: Location data is published to Kafka topic obu_positions
3. Stream Processing: Producer service consumes from Kafka and processes location updates
4. Caching: Current device positions are stored in Redis for fast retrieval
5. Real-time Updates: WebSocket connections push live updates to web clients
6. Analytics Pipeline: Analytics service consumes from Kafka and stores historical data in PostgreSQL with PostGIS
7. Persistence: Historical data with geospatial indexing enables complex queries and analytics

• Backend: Go 1.21+, gRPC, Protocol Buffers
• Message Broker: Apache Kafka for event streaming
• Caching: Redis cluster for high-performance data access
• Database: PostgreSQL with PostGIS extension for persistent storage and geospatial queries
• ORM: GORM for database operations and migrations
• Frontend: HTML5, JavaScript, Leaflet.js for map visualization
• Infrastructure: Docker, Docker Compose for containerization

Before getting started with realtime-map, ensure your runtime environment meets the following requirements:

• Programming Language: Go 1.21 or higher
• Container Runtime: Docker and Docker Compose
• Protocol Buffers: Protocol Buffers compiler (protoc)
• Build Tools: Make utility
• Message Broker: Apache Kafka (provided via Docker)
• Database: PostgreSQL with PostGIS (provided via Docker)
• Cache: Redis (provided via Docker)

Install realtime-map using the following steps:

1. Clone the Repository:

❯ git clone https://github.com/rauan06/realtime-map
❯ cd realtime-map

2. Install Development Dependencies:

❯ make deps
❯ make buf-install

3. Generate Protocol Buffer Files:

❯ make generate

4. Start Infrastructure Services:

❯ make up
# or manually: docker compose up -d

This will start:

• Apache Kafka broker on port 9092
• PostgreSQL with PostGIS on port 5430
• Analytics service (with automatic database migrations)

5. Build Services (Optional):

❯ go build ./api-gateway/cmd/app
❯ go build ./producer/cmd/app  
❯ go build ./seeder/cmd/app
❯ go build ./analytics/cmd/app

The realtime-map system consists of multiple microservices that work together:

1. Start the Infrastructure:

❯ make up

2. Run the API Gateway:

❯ cd api-gateway && go run cmd/app/main.go

The API gateway provides:

• gRPC server for device communication
• HTTP gateway for REST API access
• WebSocket endpoints for real-time map updates
• Web interface at http://localhost:8080

3. Run the Producer Service:

❯ cd producer && go run cmd/app/main.go

The producer consumes location data from Kafka and processes it for storage and real-time distribution.

4. Run the Analytics Service (runs automatically in Docker):

The analytics service is automatically started via Docker Compose and handles:

• Consuming location data from Kafka
• Running database migrations automatically
• Storing historical location data in PostgreSQL with PostGIS
• Enabling geospatial queries and analytics

To run manually (for development):

❯ cd analytics && go run cmd/app/main.go

5. Run the Data Seeder (for testing):

❯ cd seeder && go run cmd/app/main.go

The seeder generates simulated device location data for testing and demonstration.

Service Endpoints:

• Web Interface: http://localhost:8080
• gRPC API: localhost:50051 (default)
• REST API Gateway: http://localhost:8080/api
• Kafka Broker: localhost:9092
• PostgreSQL with PostGIS: localhost:5430

Run the test suite using the following commands:

Run All Tests:

❯ go test ./...

Run Tests for Specific Service:

❯ cd api-gateway && go test ./...
❯ cd producer && go test ./...
❯ cd seeder && go test ./...
❯ cd analytics && go test ./...

Generate Test Coverage:

❯ go test -coverprofile=coverage.out ./...
❯ go tool cover -html=coverage.out

Integration Testing:

1. Start the infrastructure services: make up
2. Run the seeder to generate test data: cd seeder && go run cmd/app/main.go
3. Start the producer to process data: cd producer && go run cmd/app/main.go
4. The analytics service runs automatically and processes location data
5. Start the API gateway: cd api-gateway && go run cmd/app/main.go
6. Open http://localhost:8080 to see real-time location updates on the map

The system provides a gRPC service for high-performance device communication:

Service Definition:

service LocationService {
  rpc StartSession(DeviceID) returns (google.protobuf.Empty);
  rpc SendLocation(OBUData) returns (google.protobuf.Empty);
  rpc GetDeviceLocations(google.protobuf.Empty) returns (stream OBUData);
}

Message Types:

message OBUData {
  bytes device_id = 1;       // 16-byte UUID
  double latitude = 2;       // GPS latitude
  double longitude = 3;      // GPS longitude  
  google.protobuf.Timestamp timestamp = 4;
}

message DeviceID {
  bytes device_id = 1;       // 16-byte UUID
}

The HTTP gateway provides RESTful access to the gRPC services:

• POST /api/v1/devices/{device_id}/session - Start device session
• POST /api/v1/locations - Send location data
• GET /api/v1/locations/stream - Stream location updates
• GET /api/v1/health - Health check endpoint

Real-time location updates via WebSocket:

• Endpoint: ws://localhost:8080/ws
• Message Format: JSON-encoded OBUData
• Connection: Automatic reconnection with exponential backoff

• Task 1: Implement Kafka producer and data seeder.
• Task 2: Implement Kafka consumer for location data processing.
• Task 3: Basic WebSocket implementation and HTML templates.
• Task 4: Integration with Leaflet.js for map visualization.
• Task 5: Analytics service with PostgreSQL and PostGIS for historical data storage.
• Task 6: Add Grafana dashboards for analytics and monitoring.
• Task 7: Implement device authentication and authorization.
• Task 8: Add geofencing and location-based alerts.
• Task 9: Performance optimization and horizontal scaling.
• Task 10: Comprehensive test coverage and CI/CD pipeline.

• 💬 Join the Discussions: Share your insights, provide feedback, or ask questions.
• 🐛 Report Issues: Submit bugs found or log feature requests for the realtime-map project.
• 💡 Submit Pull Requests: Review open PRs, and submit your own PRs.

This project is protected under the MIT License. For more details, refer to the LICENSE file.