Business-level distributed tracing and analytics for Istio service mesh

Zero-code changes required • Complete request visibility • Advanced troubleshooting

High-performance, asynchronous HTTP session capture built with Rust and WebAssembly (WASM)

Softprobe Agent is a WebAssembly (WASM) plugin for Istio that captures complete HTTP request/response data and sends it to Softprobe for business-level analytics and troubleshooting without modifying application code.

• 🔍 Complete Visibility: Capture full HTTP request/response data across your service mesh
• 🚀 Faster Troubleshooting: Business-level tracing reduces debugging time from hours to minutes
• 📊 Data Analytics: Rich insights into API usage patterns and business flows
• ⚡ Zero Intrusion: No application code changes required
• 🔒 Enterprise Ready: Production-grade security and performance
• 🏎️ High Performance & Async: Rust+WASM streaming, asynchronous HTTP capture with minimal overhead

To achieve complete session-based tracing capabilities, Softprobe Agent can be used in conjunction with @softprobe/sessify, a lightweight frontend session management library.

1. Frontend Session Tracking: @softprobe/sessify manages user sessions in the browser, automatically creating and maintaining session identifiers
2. Request Header Injection: The library injects session information into HTTP request headers as part of the tracestate field
3. Backend Correlation: Softprobe Agent captures these headers and correlates requests with the same session ID
4. Session Flow Visualization: In Softprobe Dashboard, you can view complete session flows across your service mesh

• Session-Level Tracing: Track complete user journeys across multiple services
• Enhanced Troubleshooting: Identify issues specific to user sessions or user segments
• Business Flow Analysis: Understand how users navigate through your application
• No Additional Backend Changes: The integration works seamlessly with existing Softprobe Agent deployment

Documentation

• Operating System: macOS (or Linux with docker)
• Required Tools:
  • Docker Desktop
  • Kind - brew install kind
  • kubectl - brew install kubectl
  • Istio CLI - brew install istioctl

Or install these tools all at once.

brew install kind kubectl istioctl

curl -L https://raw.githubusercontent.com/softprobe/softprobe/refs/heads/main/scripts/cluster-setup.sh | sh

# Install Softprobe Istio WASM Plugin
kubectl apply -f https://raw.githubusercontent.com/softprobe/softprobe/refs/heads/main/deploy/minimal.yaml
# Install demo app
kubectl apply -f https://raw.githubusercontent.com/softprobe/softprobe/refs/heads/main/examples/travel/apps.yaml
# Expose the demo
sleep 10 && kubectl port-forward -n istio-system svc/istio-ingressgateway 8080:80

Play with the demo travel app by open http://localhost:8080/ in browser, select a pari of cities and do a search, book and payment (fill any fake information). Then you can go to Softprobe Dashboard, check Trave View on the left navagation menu.

kind delete cluster --name sp-demo-cluster

kubectl apply -f https://raw.githubusercontent.com/softprobe/sp-istio/main/deploy/minimal.yaml

• Rust toolchain with wasm32-unknown-unknown target
• Protocol Buffers compiler (protobuf-compiler)
• kubectl and Istio (for deployment)

# Install Rust WASM target
rustup target add wasm32-unknown-unknown

# On Debian/Ubuntu
sudo apt-get install protobuf-compiler

# On macOS
brew install protobuf

make build

This will:

• Build the WASM binary for the wasm32-unknown-unknown target
• Calculate the SHA256 hash
• Show commands to update Istio configurations

make integration-test

This will:

• Validate the WASM binary
• Start a local Envoy instance
• Test the extension functionality
• Show relevant logs

Use the Makefile to spin up a local Kind cluster, build the WASM, load the Docker image directly into Kind (no registry push), deploy the plugin, and install the demo app.

# 1) Create cluster, build + load local image, deploy plugin and demo app
make dev-quickstart

# 2) In a separate terminal, expose the demo on http://localhost:8080
make forward

# 3) (Optional) Check status
make status
kubectl get wasmplugin -n istio-system

# 4) Cleanup cluster
make cluster-down

Use this workflow to iterate on the WASM plugin without deleting the Kind cluster or demo apps.

make dev-reload

What make dev-reload does:

• Builds target/wasm32-unknown-unknown/release/sp_istio_agent.wasm
• Copies it to the sp-wasm-http pod (istio-system)
• Patches WasmPlugin spec.url with a cache-busting query param so Envoy re-fetches the module (no pod restarts)

For production/global install, apply the WasmPlugin manifest under deploy/:

kubectl apply -f deploy/sp-istio-agent.yaml

To test safely with the Istio Bookinfo demo, use the scoped test manifest which targets only productpage and includes a ServiceEntry for o.softprobe.ai:

kubectl label namespace default istio-injection=enabled --overwrite
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.22/samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.22/samples/bookinfo/networking/bookinfo-gateway.yaml
kubectl apply -f deploy/test-bookinfo.yaml

Then generate traffic and verify:

export GATEWAY_URL=$(kubectl -n istio-system get svc istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
curl -sf "http://${GATEWAY_URL}/productpage" >/dev/null
kubectl get wasmplugin -A

1. Request Interception: Extension captures outgoing HTTP requests
2. Agent Lookup: Sends request data to Softprobe for agent check
3. Agent Hit: Returns agentd response if available (HTTP 200)
4. Agent Miss: Continues to upstream service (HTTP 404)
5. Response Storage: Asynchronously stores successful responses for future caching

• deploy/sp-istio-agent.yaml: Global WasmPlugin manifest
• deploy/test-bookinfo.yaml: Scoped test manifest for Bookinfo
• test/envoy.yaml: Local Envoy configuration for testing

This project includes automated GitHub Actions workflows:

• Trigger: Push to main/bill/deploy branches, Pull Requests
• Workflow: .github/workflows/integration-test.yml
• Actions:
  • Builds WASM binary
  • Runs integration tests with Softprobe backend
  • Validates end-to-end telemetry pipeline

• Trigger: Git tags with format v*.*.* (e.g., v1.2.3)
• Workflow: .github/workflows/release.yml
• Actions:
  • Updates Cargo.toml version from tag
  • Builds and tests WASM binary
  • Publishes Docker images to softprobe/softprobe and softprobe/sp-envoy
  • Creates GitHub release with WASM binary and deployment files

• DOCKERHUB_USERNAME: Docker Hub username
• DOCKERHUB_TOKEN: Docker Hub access token

git tag v1.2.3
git push origin v1.2.3

The release workflow will automatically:

1. Extract version from tag
2. Update Cargo.toml version
3. Build and test
4. Publish Docker images
5. Create GitHub release with assets

1. Check Envoy logs for WASM-related errors:

kubectl logs <pod-name> -c istio-proxy | grep -i wasm

2. Verify SHA256 hash matches between binary and configuration:

shasum -a 256 target/wasm32-unknown-unknown/release/sp_istio_agent.wasm

1. Enable debug logging, uncommend the follwoing lines in apps.yaml

        # Uncomment to enable WASM debug logging
        sidecar.istio.io/componentLogLevel: "wasm:debug"

2. Restart the app, and check extension logs for "SP" messages:

kubectl logs <pod-name> -c istio-proxy | grep "SP"

2. Verify Softprobe endpoint connectivity
3. Check request/response flow in logs

• Bodies are processed in a streaming fashion; responses are forwarded as chunks arrive (no full-body blocking)
• Optional buffering (for analytics/extraction) increases memory/CPU; prefer incremental processing and apply size caps/sampling
• Async storage keeps tail latency low; only lightweight work happens on the hot path