APISIX Etcd Hanging Bug Reproduction

A minimal reproduction setup for the APISIX etcd hanging issue where the API gateway hangs during initialization on the init_etcd phase, preventing proper route synchronization.

Hardcoded default configuration overrides custom configuration files.

APISIX ignores custom config.yaml files and uses a hardcoded default http://127.0.0.1:2379 instead of the configured etcd service host.

• File: /usr/local/apisix/apisix/cli/config.lua (inside Docker image)
• Problem: Hardcoded host = { "http://127.0.0.1:2379" } in the APISIX Docker image
• Impact: Custom config with http://etcd:2379 gets ignored completely
• Result: APISIX tries to connect to localhost instead of etcd service
• Cascading Effect: This causes BOTH hanging AND addon syncing failures

Process cleanup failure in test scripts causing intermittent failures.

During comprehensive testing, we discovered that improper process cleanup between test iterations caused nginx processes to accumulate, leading to port conflicts and configuration issues on subsequent iterations.

• Problem: Test scripts using kill $APISIX_PID instead of proper APISIX stop command
• Impact: nginx master and worker processes remain running between iterations
• Result: Port conflicts and configuration conflicts on 3rd+ iterations
• Fix: Use /usr/bin/apisix stop for proper cleanup

Issue: APISIX hangs during initialization on the init_etcd phase, preventing:

• Route synchronization with etcd
• Admin API functionality
• Proper system initialization
• Route accessibility via HTTP
• Addon/Plugin synchronization with etcd (cascading failure)

1. Primary Failure: APISIX uses hardcoded 127.0.0.1:2379 instead of etcd:2379
2. Connection Failure: Cannot connect to localhost:2379 in container environment
3. Initialization Hanging: APISIX hangs on init_etcd phase indefinitely
4. Cascading Effects:
   • ❌ Admin API never becomes accessible
   • ❌ Plugin syncer cannot initialize (depends on etcd connection)
   • ❌ Addon/plugin configuration sync fails
   • ❌ Route management fails
   • ❌ All etcd-dependent functionality breaks

Symptoms:

• APISIX process starts but never completes initialization
• Logs show: Warning! Request etcd endpoint 'http://127.0.0.1:2379/version' error, connection refused
• Admin API is not accessible
• Routes created via Admin API return 404 when accessed via HTTP
• Plugin/Addon configuration sync fails (direct result of hardcoded config)
• Plugin configuration API endpoints are not accessible (direct result of hardcoded config)
• etcd is healthy and accessible, but APISIX cannot sync with it due to wrong host

• Docker and Docker Compose
• Git
• Note: All commands require sudo privileges to access Docker daemon

git clone https://github.com/joggerjoel/apisix-etcd-hanging-bug.git
cd apisix-etcd-hanging-bug

# If Docker is not installed, run the installation script
./install_docker.sh

# After installation, log out and log back in for group changes to take effect

# Easy way: Use the run script (requires sudo)
sudo ./run_bug.sh up

# For faster restarts (no rebuild): Use the restart script
sudo ./restart_bug.sh restart

# Or manually with docker-compose
sudo docker-compose -f docker-compose.bug.yml up --build

# Quick test (15 seconds)
sudo ./run_bug.sh test

# Full demonstration (30 seconds)
sudo ./run_bug.sh demo

# Test addon sync hanging behavior
sudo ./run_bug.sh addon-sync

# Test Admin API accessibility
sudo ./run_bug.sh admin-api

# Run comprehensive test (3 iterations) - NEW!
sudo ./run_bug.sh test-all

# Start APISIX to demonstrate hanging
sudo ./run_bug.sh start

# Investigate the configuration issue
sudo ./run_bug.sh investigate

# Fix the etcd configuration issue
sudo ./run_bug.sh fix

# Compare working vs hanging scenarios
sudo ./run_bug.sh compare

# View logs to see the hanging
sudo ./run_bug.sh logs

# Access container for manual investigation
sudo ./run_bug.sh shell

apisix-etcd-hanging-bug/
├── README.md                           # This file
├── Dockerfile.bug                      # Minimal Dockerfile reproducing the issue
├── docker-compose.bug.yml              # Docker Compose setup
├── config.yaml                         # APISIX configuration
├── .env                               # Centralized environment configuration
├── run_bug.sh                          # Main script to run the bug reproduction
├── restart_bug.sh                      # Lightweight restart script (no rebuild)
├── common_functions.sh                 # Shared functions between scripts
├── install_docker.sh                   # Docker installation script
├── show_hanging.sh                     # Simple demonstration script
├── test_addon_sync.sh                  # Addon sync hanging test
├── test_admin_api.sh                   # Admin API accessibility test
├── start_apisix.sh                     # Start APISIX to demonstrate hanging
├── fix_admin_access.sh                 # Fix admin access restrictions
├── comprehensive_test.sh               # Comprehensive test script (3 iterations)
├── api_key.conf                        # API key configuration
├── update_api_key.sh                   # Script to update API key in .env
└── .gitignore                          # Git ignore file

All environment variables are centralized in .env:

# APISIX API Key - The key that APISIX generates and uses
APISIX_ADMIN_KEY=yDqxrnguAUwpHyHhgdRgRDWOPYJSHDnR

# APISIX Configuration
APISIX_ADMIN_URL=http://localhost:9180/apisix/admin
APISIX_ADMIN_PORT=9180
APISIX_GATEWAY_PORT=8080

# etcd Configuration
ETCD_HOST=etcd
ETCD_PORT=2379
ETCD_URL=http://etcd:2379

# Docker Configuration
DOCKER_COMPOSE_FILE=docker-compose.bug.yml
PROJECT_NAME=apisix-etcd-hanging-bug
APISIX_SERVICE=apisix-bug

# Test Configuration
TEST_ITERATIONS=3

This ensures consistency across all scripts and configurations. The API key is automatically passed to APISIX via environment variables.

Note: The .env file follows the standard convention for environment configuration files and is automatically ignored by git to prevent accidentally committing sensitive information.

The .env file is mounted as a volume in the Docker container to ensure the API key persists across rebuilds:

volumes:
  - ./.env:/usr/local/apisix/.env:ro

When APISIX generates a new API key (which happens on each rebuild), you can update the .env file using:

sudo ./run_bug.sh update-key

This command will:

1. Start APISIX and detect the current API key
2. Update the .env file with the correct key
3. Update all configuration files (config.yaml, Dockerfile, docker-compose.yml, README.md)
4. Ensure consistency across all components

When you run the reproduction setup, you should observe:

1. ✅ etcd starts successfully and becomes healthy
2. ❌ APISIX starts but hangs during initialization
3. ❌ Demonstration script shows:
   • APISIX process is running but not responding
   • Logs show hanging on init_etcd phase
   • etcd is reachable but APISIX cannot sync
   • Admin API is not accessible
   • Route synchronization fails
   • Plugin/Addon configuration sync fails
   • Plugin configuration API endpoints are not accessible

1. run_bug.sh - Full-featured script with build capabilities
2. restart_bug.sh - Lightweight restart script (faster, no rebuild)

• Use run_bug.sh for:

  • Initial setup and building
  • When you need to rebuild images
  • Full feature set including investigation tools
  • Development and debugging

• Use restart_bug.sh for:

  • Quick restarts (much faster)
  • Testing and validation
  • When images are already built
  • Daily development workflow

The run_bug.sh script provides convenient commands for managing the bug reproduction:

# Show all available commands
sudo ./run_bug.sh help

# Build and start services
sudo ./run_bug.sh up

# Stop services
sudo ./run_bug.sh down

# View logs from all services
sudo ./run_bug.sh logs

# View only APISIX logs
sudo ./run_bug.sh logs-apisix

# View only etcd logs
sudo ./run_bug.sh logs-etcd

# Run quick hanging test (15 seconds)
sudo ./run_bug.sh test

# Run full hanging demonstration (30 seconds)
sudo ./run_bug.sh demo

# Test addon sync hanging behavior
sudo ./run_bug.sh addon-sync

# Run comprehensive test (3 iterations) - NEW!
sudo ./run_bug.sh test-all

# Access APISIX container shell
sudo ./run_bug.sh shell

# Show service status and health checks
sudo ./run_bug.sh status

# Clean up everything (containers, images, volumes)
sudo ./run_bug.sh clean

The restart_bug.sh script provides fast restart capabilities without rebuilding:

# Show all available commands
sudo ./restart_bug.sh help

# Restart services (default command)
sudo ./restart_bug.sh
sudo ./restart_bug.sh restart

# Start services
sudo ./restart_bug.sh up

# Stop services
sudo ./restart_bug.sh down

# View logs from all services
sudo ./restart_bug.sh logs

# View only APISIX logs
sudo ./restart_bug.sh logs-apisix

# View only etcd logs
sudo ./restart_bug.sh logs-etcd

# Access APISIX container shell
sudo ./restart_bug.sh shell

# Run quick hanging test (15 seconds)
sudo ./restart_bug.sh test

# Run comprehensive test (3 iterations)
sudo ./restart_bug.sh test-all

# Test addon sync hanging behavior
sudo ./restart_bug.sh addon-sync

# Test Admin API accessibility
sudo ./restart_bug.sh admin-api

# Start APISIX to demonstrate hanging
sudo ./restart_bug.sh start

# Fix admin access configuration
sudo ./restart_bug.sh fix-admin

# Update API key in .env file
sudo ./restart_bug.sh update-key

# Show status of services
sudo ./restart_bug.sh status

# Clean up everything (containers, images, volumes)
sudo ./restart_bug.sh clean

The reproduction now includes a comprehensive test suite that runs 3 iterations to verify that all fixes work consistently:

# Run comprehensive test (3 iterations)
sudo ./run_bug.sh test-all

What the comprehensive test does:

1. Iteration 1: Tests all functionality (API key, route creation, plugin sync, etcd persistence)
2. Iteration 2: Repeats all tests to verify consistency
3. Iteration 3: Final verification that everything works reliably

Test Coverage:

• ✅ API Key Authentication: Verifies Admin API accessibility
• ✅ Plugin Configuration API: Tests plugin management endpoints
• ✅ Route Creation: Creates test routes via Admin API
• ✅ etcd Persistence: Verifies routes are stored in etcd
• ✅ Plugin Sync: Tests addon/plugin configuration synchronization
• ✅ Data Cleanup: Ensures test data is properly cleaned up
• ✅ Process Management: Verifies proper APISIX startup/shutdown

Expected Results:

🎉 ALL TESTS PASSED! (3/3)
✅ etcd sync and addon plugin functionality is working consistently
✅ APISIX configuration is stable and reliable
✅ All fixes are working correctly

During comprehensive testing, we discovered a secondary issue that caused intermittent failures:

Problem: Test scripts were using kill $APISIX_PID instead of the proper APISIX stop command, leaving nginx processes running between iterations.

Symptoms:

• Tests 1 & 2 would pass
• Test 3 would fail with HTTP 401 (wrong API key)
• nginx processes accumulating between iterations
• Port conflicts on subsequent iterations

Solution: Updated comprehensive test script to use /usr/bin/apisix stop for proper cleanup.

Impact: This discovery ensures the test suite is reliable and demonstrates that all fixes work consistently across multiple iterations.

If you want to investigate further:

# Access the running container
sudo ./run_bug.sh shell

# Or manually with docker-compose
sudo docker exec -it apisix-etcd-hanging-bug_apisix-bug_1 bash

# Check APISIX logs
tail -f /tmp/apisix.log

# Check if APISIX process is running
ps aux | grep apisix

# Test etcd connectivity
nc -z etcd 2379

# Try to access Admin API (should fail)
curl -v http://localhost:9180/apisix/admin/services \
  -H "X-API-KEY: a663d5f8d98fd490408ad230fae57312"

# Try to access APISIX HTTP endpoint (should fail)
curl -v http://localhost:8080/

The minimal configuration that reproduces the bug:

deployment:
  role: traditional
  admin:
    admin_key:
      - name: admin
        key: ${APISIX_ADMIN_KEY}
        role: admin

apisix:
  node_listen: 80
  enable_admin: true
  admin_listen:
    ip: 0.0.0.0
    port: 9180
  allow_admin:
    - 0.0.0.0/0
  etcd:
    host:
      - "http://etcd:2379"
    prefix: "/apisix"
    timeout: 30

# Minimal plugin configuration
plugins:
  - serverless-pre-function

• APISIX completes initialization in ~5-10 seconds
• Admin API responds to requests
• Routes can be created and accessed via HTTP
• etcd synchronization works properly

• APISIX hangs on init_etcd phase indefinitely
• Admin API is not accessible
• Routes cannot be synchronized with etcd
• HTTP requests to routes return 404
• Plugin/Addon configurations cannot be synchronized with etcd
• Plugin configuration API endpoints return errors

This bug has been observed with:

• APISIX versions: 3.13.0, 3.12.0, 3.9.1 (not version-specific)
• etcd versions: bitnami/etcd:latest, bitnami/etcd:3.4.15
• Configuration types: Both minimal and complex configurations
• Environment: Docker containers with proper networking

Based on investigation, the issue appears to be:

1. Not version-specific - occurs across multiple APISIX versions
2. Not configuration-specific - occurs with minimal configs
3. Not etcd-specific - etcd is healthy and accessible
4. Environment-related - likely Docker environment or system configuration issue

The hanging issue is caused by hardcoded default configuration in the APISIX Docker image that overrides custom configuration files:

• File: /usr/local/apisix/apisix/cli/config.lua (inside Docker image)
• Problem: Contains hardcoded host = { "http://127.0.0.1:2379" } in the APISIX Docker image
• Impact: APISIX tries to connect to localhost instead of the etcd service
• Result: Connection fails, causing hanging during initialization
• Cascading Effect: This single hardcoded value breaks ALL etcd-dependent functionality

The investigation script (sudo ./run_bug.sh investigate) reveals:

1. ✅ Custom config file exists with correct etcd host: http://etcd:2379
2. ✅ etcd service is reachable and responding correctly
3. ❌ Hardcoded default overrides custom config: http://127.0.0.1:2379
4. ❌ APISIX ignores custom config.yaml and uses hardcoded values
5. ❌ localhost:2379 is not reachable (as expected in container environment)

Evidence from logs:

Use customized yaml: /usr/local/apisix/conf/config.yaml
Warning! Request etcd endpoint 'http://127.0.0.1:2379/version' error, connection refused
all etcd nodes are unavailable

This proves that APISIX loads the custom config file but still uses the hardcoded default!

• Function: init_plugins_syncer() in /usr/local/apisix/apisix/plugin.lua
• Problem: Tries to create etcd instance for fetching /plugins configuration
• Root Cause: Cannot connect to etcd due to hardcoded 127.0.0.1:2379 configuration
• Impact: Plugin/addon configuration sync fails
• Result: Plugin configuration API endpoints become inaccessible
• Note: This is NOT a separate issue - it's a direct result of the hardcoded etcd configuration

# Clone and start the reproduction environment
git clone https://github.com/joggerjoel/apisix-etcd-hanging-bug.git
cd apisix-etcd-hanging-bug
sudo ./run_bug.sh up

What happens:

• etcd starts successfully on etcd:2379
• APISIX container builds with custom config pointing to http://etcd:2379
• Both services are healthy and reachable

sudo ./run_bug.sh test

What breaks:

• APISIX starts but hangs during init_etcd phase
• Logs show: request etcd endpoint 'http://127.0.0.1:2379/version' error, connection refused
• APISIX never completes initialization
• Admin API remains inaccessible

sudo ./run_bug.sh addon-sync

What breaks:

• Plugin configuration sync fails
• init_plugins_syncer() cannot create etcd instance for /plugins
• Plugin configuration API endpoints return errors
• Addon/plugin management becomes impossible

sudo ./run_bug.sh shell
# Inside container:
grep -A 5 -B 5 '127.0.0.1.*2379' /usr/local/apisix/apisix/cli/config.lua

What you'll find:

etcd = {
  host = { "http://127.0.0.1:2379" },  # ← This is the problem!
  prefix = "/apisix",
  timeout = 30,
  # ...
}

1. APISIX reads custom config file (/usr/local/apisix/conf/config.yaml)
2. Connects to etcd at http://etcd:2379
3. Completes initialization in ~5-10 seconds
4. Admin API responds to requests
5. Plugin configuration sync works
6. Routes can be created and accessed

1. APISIX ignores custom config file
2. Uses hardcoded default: http://127.0.0.1:2379
3. Hangs indefinitely trying to connect to localhost
4. Admin API never becomes accessible
5. Plugin configuration sync fails
6. Routes cannot be created or accessed

sudo ./run_bug.sh shell
# Inside container:
nc -z etcd 2379  # Should succeed
curl http://etcd:2379/version  # Should return etcd version

# Inside container:
cat /usr/local/apisix/conf/config.yaml  # Shows correct config
grep -A 10 'etcd.*host' /usr/local/apisix/apisix/cli/config.lua  # Shows hardcoded default

# Inside container:
/usr/bin/apisix start -c /usr/local/apisix/conf/config.yaml 2>&1 | head -20

Expected output:

Use customized yaml: /usr/local/apisix/conf/config.yaml
Warning! Request etcd endpoint 'http://127.0.0.1:2379/version' error, connection refused

# Inside container:
grep -A 15 'init_plugins_syncer' /usr/local/apisix/apisix/plugin.lua

What you'll see:

function init_plugins_syncer()
    local err
    plugins_conf, err = core.config.new("/plugins", {
        automatic = true,
        item_schema = core.schema.plugins,
        # This fails because etcd connection is broken
    })
end

The reproduction includes a fix that is automatically applied during the Docker build:

The Dockerfile automatically fixes the hardcoded default during build:

# Fix the hardcoded etcd default to use etcd service instead of localhost
RUN sed -i 's|"http://127.0.0.1:2379"|"http://etcd:2379"|g' /usr/local/apisix/apisix/cli/config.lua

What the fix does:

1. Modifies hardcoded config: Changes 127.0.0.1:2379 to etcd:2379 during build
2. APISIX starts correctly: Connects to etcd service instead of localhost
3. Admin API works: Becomes accessible after APISIX completes initialization
4. No more hanging: APISIX completes startup successfully

Method 1: Environment Variable Override

export APISIX_ETCD_HOST="http://etcd:2379"

Method 2: Configuration File Override

# Modify the hardcoded config file
sed -i 's|"http://127.0.0.1:2379"|"http://etcd:2379"|g' /usr/local/apisix/apisix/cli/config.lua

Method 3: Restore Original Config

# To restore the original configuration
cp /usr/local/apisix/apisix/cli/config.lua.backup /usr/local/apisix/apisix/cli/config.lua

• ✅ APISIX connects to etcd service instead of localhost
• ✅ APISIX completes initialization without hanging
• ✅ Admin API becomes accessible (proper API responses)
• ✅ Plugin Config API becomes accessible (proper API responses)
• ✅ Addon synchronization works correctly
• ✅ No more hanging behavior - APISIX starts successfully
• ✅ Comprehensive testing passes - All 3 iterations work consistently
• ✅ Process cleanup works properly - No nginx process accumulation

1. The bug is reproducible - The script consistently demonstrates the hanging behavior
2. Root cause is identified - Hardcoded default configuration overrides custom config
3. Multiple components affected - Both general initialization and plugin sync fail
4. Environment-specific - Works in some environments, fails in others
5. Fix requires configuration override - Need to ensure custom config takes precedence

The reproduction includes comprehensive Admin API testing to demonstrate when it works vs when it hangs:

sudo ./run_bug.sh admin-api

What this shows:

• ✅ etcd is healthy and accessible
• ❌ Admin API is not accessible (HTTP 000 - connection refused)
• ❌ Plugin Config API is not accessible
• 🚨 APISIX is running but Admin API is not accessible (hanging issue)

sudo ./run_bug.sh compare

What this demonstrates:

1. Scenario 1: APISIX not running (current state)

   • etcd is healthy
   • Admin API not accessible (expected)

2. Scenario 2: APISIX running but hanging (bug reproduction)

   • etcd is healthy
   • APISIX starts but hangs during initialization
   • Admin API not accessible (due to hanging)

3. Scenario 3: APISIX working correctly (theoretical)

   • etcd is healthy
   • APISIX completes initialization
   • Admin API accessible (HTTP 200)
   • Plugin Config API accessible (HTTP 200)

# Test Admin API when APISIX is working (should return HTTP 200)
curl http://localhost:9180/apisix/admin/services \
  -H "X-API-KEY: a663d5f8d98fd490408ad230fae57312"

# Test Plugin Config API when APISIX is working (should return HTTP 200)
curl http://localhost:9180/apisix/admin/plugin_configs \
  -H "X-API-KEY: a663d5f8d98fd490408ad230fae57312"

# Test Admin API when APISIX is hanging (will return HTTP 000)
curl http://localhost:9180/apisix/admin/services \
  -H "X-API-KEY: a663d5f8d98fd490408ad230fae57312"

On your base system, the Admin API works because:

1. APISIX is properly configured to connect to the correct etcd instance
2. APISIX completes initialization without hanging
3. Admin API becomes accessible on port 9180
4. Plugin configurations sync correctly with etcd

In the reproduction environment, the Admin API doesn't work because:

1. APISIX uses hardcoded config (127.0.0.1:2379) instead of custom config (etcd:2379)
2. APISIX hangs during initialization trying to connect to localhost
3. Admin API never becomes accessible because APISIX never completes startup
4. Plugin configurations fail to sync due to the same etcd connection issue

The reproduction setup includes:

• Health checks for etcd
• Debug logging for APISIX
• Connectivity tests between services
• Process monitoring to detect hanging
• Log analysis to identify hanging patterns

To clean up the reproduction environment:

# Easy way: Use the run script
sudo ./run_bug.sh clean

# Or manually
sudo docker-compose -f docker-compose.bug.yml down
sudo docker rmi apisix-etcd-hanging-bug_apisix-bug

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

This project is licensed under the MIT License - see the LICENSE file for details.

This bug reproduction is based on investigation documented in:

• APISIX GitHub Issues
• Various configuration files showing the issue across different setups

If you encounter any issues with this reproduction setup:

1. Check the Issues page
2. Create a new issue with detailed information
3. Include logs and environment details

1. Primary Issue - Hardcoded etcd Configuration:

   • Problem: APISIX ignores custom config.yaml and uses hardcoded http://127.0.0.1:2379
   • Fix: Modified /usr/local/apisix/apisix/cli/config.lua during Docker build
   • Result: APISIX connects to etcd service instead of localhost

2. Secondary Issue - Process Cleanup Failure:

   • Problem: Test scripts using kill $APISIX_PID left nginx processes running
   • Fix: Updated comprehensive test to use /usr/bin/apisix stop
   • Result: Proper cleanup between test iterations

3. API Key Management:

   • Problem: Inconsistent API key handling across scripts
   • Fix: Centralized API key in .env file with environment variable substitution
   • Result: Consistent API key usage across all components

4. Admin API Access:

   • Problem: Nginx configuration restricted Admin API access
   • Fix: Dynamic nginx configuration modification in test scripts
   • Result: Admin API accessible from host system

• ✅ All tests pass consistently (3/3 iterations)
• ✅ etcd sync and addon plugin functionality working
• ✅ APISIX configuration stable and reliable
• ✅ Process management working correctly
• ✅ Comprehensive test suite validates all fixes
• ✅ Two optimized scripts available (run_bug.sh and restart_bug.sh)
• ✅ Common functions shared between scripts for maintainability

1. Root cause was hardcoded configuration in the APISIX Docker image overriding custom config files
2. Addon syncing failure was NOT a separate issue - it was a cascading failure from the hardcoded etcd config
3. Secondary issue was process cleanup in test scripts (discovered during comprehensive testing)
4. Single hardcoded value broke ALL etcd-dependent functionality (Admin API, routes, plugins, etc.)
5. All fixes work together to create a stable, reliable system
6. Test suite validates that the solution is robust and consistent

apisix etcd hanging bug reproduction docker api-gateway initialization sync route-management comprehensive-testing process-cleanup restart-script common-functions