Note: This dashboard is for demonstration purposes only. The data presented is mock data and does not reflect real system metrics.
This document presents a comprehensive observability solution for Atlan's platform, addressing debugging challenges faced by engineering teams. The solution now incorporates AI-powered diagnostic and remediation capabilities to further reduce issue detection and resolution times, making the system more self-healing and less dependent on manual intervention.
The engineering team faces several key challenges in debugging:
- Slow identification of root causes
- Repetitive manual troubleshooting processes
- Inconsistent debugging approaches
- Heavy dependence on experienced engineers
- Outdated and irrelevant logs
- Delayed resolution of known issues
- Knowledge gaps between senior and junior engineers
flowchart TB
subgraph "Atlan Services"
API[API Services\n- \n- Kong\n- ]:::atlan
DB[Databases\n- PostgreSQL\n- Cassandra]:::atlan
Queue[Message Queue\n- Kafka]:::atlan
Other[Other Services\n- Keycloak- \n- etc.]:::atlan
end
subgraph "Observability Data Collection"
direction TB
Instrumentation[OpenTelemetry SDK\nAuto-Instrumentation]:::new
Metrics[Metrics Collection\nPromQL]:::existing
Logs[Log Collection\nFluentBit]:::existing
Traces[Trace Collection\nOpenTelemetry Collector]:::new
end
subgraph "Data Storage & Processing"
Victoria[VictoriaMetrics\nTime Series Database]:::existing
Elastic[Elasticsearch\nLog Storage]:::existing
TraceDB[Jaeger/Tempo\nTrace Storage]:::new
Correlation[Correlation Engine]:::new
end
subgraph "AI Diagnostic & Remediation"
AIGateway[AI Gateway]:::aifeature
DiagnosticModels[Diagnostic Models]:::aifeature
SolutionEngine[Solution Generator]:::aifeature
KnowledgeBase[Knowledge Base]:::aifeature
FeedbackLoop[Feedback Collection]:::aifeature
end
subgraph "Visualization & Analysis"
Grafana[Grafana Dashboards]:::existing
Alerts[AlertManager]:::existing
RunBooks[Automated Runbooks]:::new
SLO[SLO Monitoring]:::new
end
subgraph "Users"
Engineers[Engineering Team]:::users
Ops[Operations Team]:::users
end
API --> Instrumentation
DB --> Instrumentation
Queue --> Instrumentation
Other --> Instrumentation
Instrumentation --> Metrics
Instrumentation --> Logs
Instrumentation --> Traces
Metrics --> Victoria
Logs --> Elastic
Traces --> TraceDB
Victoria --> Correlation
Elastic --> Correlation
TraceDB --> Correlation
Correlation --> AIGateway
AIGateway --> DiagnosticModels
DiagnosticModels --> SolutionEngine
SolutionEngine --> KnowledgeBase
DiagnosticModels --> Grafana
SolutionEngine --> RunBooks
AIGateway --> Alerts
Engineers --> FeedbackLoop
FeedbackLoop --> KnowledgeBase
KnowledgeBase --> DiagnosticModels
Grafana --> Engineers
Alerts --> Engineers
RunBooks --> Engineers
SLO --> Engineers
Grafana --> Ops
Alerts --> Ops
RunBooks --> Ops
SLO --> Ops
classDef atlan fill:#ffefd5,color:#000,stroke:#333,stroke-width:1px
classDef existing fill:#f5b7b1,color:#000,stroke:#333,stroke-width:1px
classDef new fill:#aed6f1,color:#000,stroke:#333,stroke-width:1px
classDef aifeature fill:#d7bde2,color:#000,stroke:#333,stroke-width:1px
classDef users fill:#d5f5e3,color:#000,stroke:#333,stroke-width:1px
flowchart TD
User((End User)) --> FrontEnd[Frontend Application]
FrontEnd --> APIGateway[Kong API Gateway]
subgraph "Service Instrumentation"
APIGateway --> |Request| Heracles[Heracles API Service]
Heracles --> |Query| PostgreSQL[PostgreSQL]
Heracles --> |Search| Elasticsearch[(Elasticsearch)]
Heracles --> |Metadata| Metastore[Metastore]
Heracles --> |Publish Event| Kafka[(Apache Kafka)]
%% Instrumentation points
APIGateway -.-> |Metrics & Traces| TelemetrySDK
Heracles -.-> |Metrics & Traces| TelemetrySDK
PostgreSQL -.-> |Metrics & Traces| TelemetrySDK
Elasticsearch -.-> |Metrics & Traces| TelemetrySDK
Metastore -.-> |Metrics & Traces| TelemetrySDK
Kafka -.-> |Metrics & Traces| TelemetrySDK
end
TelemetrySDK[OpenTelemetry SDK]
TelemetrySDK --> |Metrics| MetricsExporter[Metrics Exporter]
TelemetrySDK --> |Logs| LogExporter[Log Exporter]
TelemetrySDK --> |Traces| TraceExporter[Trace Exporter]
MetricsExporter --> |Push| Prometheus[VictoriaMetrics Scraper]
LogExporter --> |Forward| FluentBit[Fluent Bit]
TraceExporter --> |Export| OTelCollector[OpenTelemetry Collector]
FluentBit --> |Index| LogsElasticsearch[(Elasticsearch for Logs)]
Prometheus --> |Store| VictoriaMetrics[(VictoriaMetrics)]
OTelCollector --> |Store| Jaeger[(Jaeger/Tempo)]
subgraph "Data Processing"
VictoriaMetrics --> |Query| PromQL[PromQL Engine]
LogsElasticsearch --> |Query| LogQL[Log Query Engine]
Jaeger --> |Query| TraceQL[Trace Query Engine]
PromQL & LogQL & TraceQL --> Correlator[Correlation Engine]
end
subgraph "AI Diagnostic System"
Correlator --> |Incident Data| AIGateway[AI Gateway]
AIGateway --> ErrorClassifier[Error Classification Model]
AIGateway --> RootCauseAnalyzer[Root Cause Analyzer]
AIGateway --> PatternDetector[Pattern Detection Model]
ErrorClassifier & RootCauseAnalyzer & PatternDetector --> SolutionGenerator[Solution Generator]
SolutionGenerator --> |Known Solutions| KnownIssueDB[(Known Issue Database)]
SolutionGenerator --> |Novel Solutions| RemediationEngine[Remediation Engine]
KnownIssueDB --> RemediationEngine
RemediationEngine --> |Confidence Score| ActionDecider[Action Decision Engine]
ActionDecider --> |High Confidence| AutoFix[Automated Fix]
ActionDecider --> |Medium Confidence| HumanApproval[Human Approval Flow]
ActionDecider --> |Low Confidence| Suggestion[Solution Suggestion]
end
subgraph "Visualization & Alerting"
PromQL --> Grafana[Grafana]
LogQL --> Grafana
TraceQL --> Grafana
Correlator --> Grafana
AutoFix --> |Execute| Runbook[Runbook Automation]
HumanApproval --> |Approved| Runbook
Grafana --> |SLO Dashboard| SLO[SLO Monitoring]
Grafana --> |View| Engineer((Engineer))
Suggestion --> |Alert with Context| AlertManager[AlertManager]
AlertManager --> |Notifications| Notification[Notification System]
end
Runbook --> |Automated Fix| FixAction[Auto-remediation]
FixAction -.-> |Config Update| APIGateway
FixAction -.-> |Config Update| Heracles
Engineer --> |Feedback| FeedbackCollector[Feedback Collector]
FeedbackCollector --> |Update Models| AIGateway
FeedbackCollector --> |Improve| KnownIssueDB
classDef service fill:#d5e8d4,stroke:#82b366,stroke-width:1px,color:#000
classDef datastore fill:#dae8fc,stroke:#6c8ebf,stroke-width:1px,color:#000
classDef telemetry fill:#ffe6cc,stroke:#d79b00,stroke-width:1px,color:#000
classDef processing fill:#e1d5e7,stroke:#9673a6,stroke-width:1px,color:#000
classDef ai fill:#d7bde2,stroke:#9b59b6,stroke-width:1px,color:#000
classDef visualization fill:#fff2cc,stroke:#d6b656,stroke-width:1px,color:#000
classDef user fill:#f5f5f5,stroke:#666666,stroke-width:1px,color:#000
class User,Engineer user
class APIGateway,Heracles,FrontEnd service
class PostgreSQL,Elasticsearch,Metastore,Kafka,LogsElasticsearch,VictoriaMetrics,Jaeger,KnownIssueDB datastore
class TelemetrySDK,MetricsExporter,LogExporter,TraceExporter,FluentBit,Prometheus,OTelCollector telemetry
class PromQL,LogQL,TraceQL,Correlator processing
class AIGateway,ErrorClassifier,RootCauseAnalyzer,PatternDetector,SolutionGenerator,RemediationEngine,ActionDecider,AutoFix,HumanApproval,Suggestion,FeedbackCollector ai
class Grafana,AlertManager,Notification,Runbook,SLO,FixAction visualization
The enhanced observability stack consists of:
-
Data Collection Layer
- OpenTelemetry SDK integration for application instrumentation
- Metric exporters for VictoriaMetrics (already in Atlan's stack)
- Log aggregation via FluentBit (already in Atlan's stack)
- Distributed tracing collector for end-to-end request visibility
-
Storage & Processing Layer
- VictoriaMetrics for time-series metrics (already in Atlan's stack)
- Elasticsearch for log data (already in Atlan's stack)
- Jaeger or Tempo for distributed trace storage
- Correlation engine for cross-telemetry analysis
-
AI Diagnostic & Remediation Layer
- AI Gateway for service orchestration and API access
- Diagnostic models for error classification and root cause analysis
- Solution generator for automated remediation suggestions
- Knowledge base of known issues and solutions
- Feedback loop for continuous model improvement
-
Visualization & Analysis Layer
- Grafana dashboards (already in Atlan's stack)
- Automated alerting via AlertManager (already in Atlan's stack)
- Custom service health scorecards
- AI-enhanced runbook automation for common issues
-
Automation Layer
- Auto-instrumentation of new services
- AI-powered remediation suggestions
- Automated fix implementation for high-confidence solutions
- Human approval workflow for medium-confidence solutions
- Applications are instrumented with OpenTelemetry SDK
- Telemetry data (metrics, logs, traces) is collected and processed
- Correlation engine connects related events across telemetry types
- AI models analyze correlated data to diagnose root causes
- Solution generator proposes remediation options
- Depending on confidence level, solutions are:
- Automatically implemented (high confidence)
- Sent for human approval (medium confidence)
- Suggested as part of alerts (low confidence)
- Feedback from resolutions improves AI models over time
| Metric Category | Specific Metrics | Rationale |
|---|---|---|
| Request-level | - Request rate - Error rate - Latency percentiles (p50, p95, p99) - Request payload size |
These provide overall health indicators for API endpoints. Percentiles help identify the "long tail" problems affecting some users. |
| Resource utilization | - CPU usage - Memory consumption - Garbage collection metrics - Thread pool utilization |
Resource bottlenecks often cause performance issues. These metrics help identify when scaling or optimization is needed. |
| Dependency metrics | - Database query latency - External API call latency - Cache hit/miss ratio - Queue depth |
Many API issues stem from problems with dependencies. These metrics help isolate which dependency might be causing slowdowns. |
| Business metrics | - Operation success rate - User-impacting errors - SLA compliance |
These connect technical issues to business impact, helping prioritize fixes. |
| Log Level | Purpose | Retention | Examples |
|---|---|---|---|
| ERROR | Critical failures requiring immediate attention | 30 days | - Database connection failures - Authentication failures - Data corruption |
| WARN | Potential issues that don't prevent operation | 14 days | - Retried operations - Deprecated API usage - Performance anomalies |
| INFO | Normal operation milestones | 7 days | - Request started/completed - Major workflow steps - Configuration loaded |
| DEBUG | Detailed troubleshooting information | 2 days | - Parameter values - Intermediate calculation results - Cache operations |
- Structured Logging: All logs should be in JSON format for easier parsing and filtering
- Context Enrichment: Include request ID, user ID, service name, and other relevant context
- Sampling Strategy: Sample debug logs in production to reduce volume
- Log Rotation: Implement based on time and size with appropriate retention policies
- Trace Context Propagation: Ensure all services propagate trace context headers
- Span Attributes: Enrich spans with useful attributes like:
- HTTP method, status code, URL
- Database operation type, query type
- Error details when failures occur
- Custom Spans: Create spans for key business operations
- Sampling Strategy: Sample traces intelligently based on service importance and error status
| Model Type | Purpose | Training Data |
|---|---|---|
| Error Classifier | Categorize errors into known patterns | Historical incident data, error logs |
| Root Cause Analyzer | Identify underlying causes from symptoms | Past incidents with resolved root causes |
| Pattern Detector | Detect anomalies and recurring patterns | Normal and abnormal telemetry data |
| Solution Generator | Suggest or implement fixes | Historical resolutions, runbooks, best practices |
| Confidence Level | Score Range | Action |
|---|---|---|
| High | 85-100% | Automatically implement fix through runbooks |
| Medium | 60-84% | Suggest fix for human approval with implementation path |
| Low | <60% | Surface as suggestion with context in alerts |
-
Configuration Changes: Update service parameters without code changes
- Connection pool sizes, timeouts, retry policies
- Resource allocation, scaling thresholds
- Feature flags and circuit breakers
-
Infrastructure Actions: Make changes to infrastructure
- Service restart or relocation
- Database index rebuilds
- Cache invalidation
- Load balancer configuration
-
Code Fix Suggestions: Provide guidance for code-level fixes
- Anti-pattern identification with solution patterns
- Performance optimization suggestions
- Error handling improvements
- Resolution Success Tracking: Measure if implemented solutions resolved the issue
- Engineer Ratings: Allow engineers to rate solution quality
- Override Tracking: Record when engineers select different solutions
- Improvement Suggestions: Collect engineer improvements to suggested solutions
- Model Update Pipeline: Use feedback to improve future recommendations
stateDiagram-v2
[*] --> Normal: System Stable
state "Monitoring & Detection" as Detection {
Normal --> Anomaly: Anomaly Detected
Normal --> Threshold: Threshold Breach
Normal --> Manual: Manual Report
Anomaly --> AITriage: Incident Created
Threshold --> AITriage: Alert Triggered
Manual --> AITriage: Support Ticket
}
state "AI Triage & Diagnosis" as AITriage {
[*] --> AIAssessment: Analyze Incident Data
AIAssessment --> KnownIssue: Match Found
AIAssessment --> UnknownIssue: No Match
KnownIssue --> HighConfidence: Confidence ≥85%
KnownIssue --> MediumConfidence: 60%≤ Confidence <85%
KnownIssue --> LowConfidence: Confidence <60%
UnknownIssue --> DeepAnalysis: Run ML Analysis
DeepAnalysis --> SolutionGeneration: Find Potential Fix
SolutionGeneration --> MediumConfidence: 60%≤ Confidence <85%
SolutionGeneration --> LowConfidence: Confidence <60%
}
HighConfidence --> AutoRemediation: Automatic Fix
MediumConfidence --> HumanApproval: Engineer Reviews
LowConfidence --> EngineerNotification: Suggest Fix
HumanApproval --> AutoRemediation: Approved
HumanApproval --> EngineerInvestigation: Rejected
EngineerNotification --> EngineerInvestigation: Manual Fix Needed
state "Auto-Remediation" as AutoRemediation {
[*] --> ExecuteFix: Run Fix Playbook
ExecuteFix --> FixSuccess: Fix Successful
ExecuteFix --> FixFailure: Fix Failed
FixSuccess --> Verification
FixFailure --> EngineerInvestigation
}
state "Engineer Investigation" as EngineerInvestigation {
[*] --> ContextExploration: Review AI Analysis
ContextExploration --> RootCauseIdentification: Find Root Cause
RootCauseIdentification --> ManualFix: Implement Fix
ManualFix --> Verification
}
state "Verification" as Verification {
[*] --> MonitorMetrics: Watch Key Metrics
MonitorMetrics --> TestFunctionality: Verify Functionality
TestFunctionality --> ResolutionConfirmation: Confirm Resolution
}
state "Feedback Loop" as FeedbackLoop {
[*] --> CaptureSolution: Record Fix Details
CaptureSolution --> EffectivenessRating: Rate Solution
EffectivenessRating --> ModelUpdate: Update AI Models
ModelUpdate --> KnowledgeBaseUpdate: Update KB
}
ResolutionConfirmation --> FeedbackLoop: Post-Resolution
FeedbackLoop --> Normal: Return to Normal
note right of AITriage
AI models analyze metrics, logs, and traces
to identify patterns and root causes
end note
note left of AutoRemediation
Common fixes: scaling, connection resets,
cache invalidation, config adjustments
end note
note right of FeedbackLoop
Each resolution improves the AI models,
increasing automation over time
end note
-
Slow Root Cause Identification
- Implement distributed tracing to pinpoint bottlenecks
- Create correlation between metrics, logs, and traces
- Deploy AI diagnostics to rapidly analyze correlated data
- Provide context-rich alerts with probable causes
-
Repetitive Manual Troubleshooting
- Create runbooks for common issues
- Develop self-service troubleshooting tools
- Implement AI-powered automation for recurring issues
- Auto-implement high-confidence fixes
-
Inconsistent Debugging
- Standardize observability patterns across services
- Create service templates with built-in observability
- Develop team training on observability tools and approaches
- Use AI to enforce consistent debugging approaches
-
Dependency on Experienced Engineers
- Document tribal knowledge in knowledge base
- Create AI models that learn from expert resolutions
- Build guided remediation workflow for junior engineers
- Provide confidence scores for suggested solutions
-
Outdated and Irrelevant Logs
- Audit and clean up existing logs
- Implement log rotation and retention policies
- Create guidelines for meaningful logging
- Use AI to highlight relevant logs during incidents
To track how much the improvements have helped the team:
-
Quantitative Metrics:
- Mean Time To Detection (MTTD) of issues
- Mean Time To Resolution (MTTR)
- Number of incidents requiring escalation
- Frequency of similar recurring issues
- Percentage of issues resolved automatically
- AI solution acceptance rate
-
Qualitative Metrics:
- Regular surveys of engineering team satisfaction with debugging experience
- "Friction log" collection during incident response
- Peer review of post-incident reports
- Engineer ratings of AI-suggested solutions
-
Automation Metrics:
- Percentage of issues automatically detected before user reports
- Self-service resolution rate
- Dashboard usage statistics
- AI solution confidence scores over time
- Knowledge base growth and utilization
-
Historical Incident Records:
- Past incident reports and resolutions
- Post-mortem documentation
- Identified root causes
- Solution effectiveness records
-
Expert Knowledge Capture:
- Structured interviews with senior engineers
- Documentation of troubleshooting approaches
- Recorded debugging sessions
- Common fix patterns
-
Synthetic Data Generation:
- Simulated failure scenarios
- Load testing results
- Chaos engineering outcomes
- Deliberately induced errors with known resolutions
-
Feedback Collection Pipeline:
- Capture resolution effectiveness
- Record engineer interactions with suggestions
- Track manual overrides and modifications
- Monitor solution performance metrics
-
Model Retraining Strategy:
- Scheduled retraining intervals (weekly/monthly)
- Triggered retraining on feedback thresholds
- A/B testing of model improvements
- Progressive deployment of updated models
-
Knowledge Base Expansion:
- Automated extraction of new patterns
- Solution categorization and tagging
- Cross-referencing with external knowledge sources
- Versioning of solutions for different system states
-
Performance Evaluation:
- Model accuracy metrics
- False positive/negative tracking
- Time saved per incident
- Engineer satisfaction scores
-
Legacy System Integration: Some legacy systems may be difficult to instrument fully
- Mitigation: Use proxy-based monitoring and contextual analysis where direct instrumentation isn't possible
-
Cost Implications: Comprehensive observability with AI increases infrastructure costs
- Mitigation: Implement intelligent sampling, tiered storage policies, and optimize AI resource usage
-
Privacy Considerations: Logging may capture sensitive user data
- Mitigation: Implement data scrubbing, privacy-preserving learning techniques, and compliance with regulations
-
Team Adaptation: Teams will need time to adapt to AI-augmented workflows
- Mitigation: Phased rollout with adequate training, clear documentation, and gradual introduction of AI capabilities
-
AI Confidence Limitations: Not all issues will have high-confidence solutions
- Mitigation: Progressive improvement through feedback loops and clear handoff to human engineers when needed
-
False Positives/Negatives: AI models may sometimes provide incorrect diagnoses
- Mitigation: Conservative confidence thresholds, human verification for critical systems, continuous model improvement
The enhanced observability solution with AI-powered diagnostics and remediation addresses the core debugging challenges faced by Atlan's engineering team. By implementing comprehensive instrumentation, correlation across telemetry types, and automated analysis with intelligent remediation, we can significantly reduce time to resolution and dependence on experienced engineers.
This approach leverages Atlan's existing architecture components while introducing new capabilities for distributed tracing, correlation, and AI-driven problem-solving. The solution creates a semi-autonomous system that not only detects issues but also helps resolve them with minimal human intervention, while continuously learning from each incident.
The phased implementation approach ensures that teams can adapt gradually while beginning to see benefits from early stages, with the AI capabilities becoming more powerful and autonomous as the system learns from real-world usage.