Can ping Pangolin site but not curl any resource #37
Description
Hi, I'm trying to setup what crowdsec calls a multi-server setup with Pangolin.
The idea is that the "local" crowdsec API server lives on one machine (the Pangolin VPS in my case), and another crowdsec instance runs on a remote machine (in my case, my NAS at home). The latter doesn't use its own local crowdsec API but instead connects to the remote one on the other machine.
The reason for doing this is that I can use my NAS as a crowdsec log processor, that would read logs from my different services running on that machine, and send them to Pangolin's crowdsec instance for the api over there to do its thing.
I want to use olm to create a tunneled connection from the NAS to the VPS and connect the NAS's crowdsec instance to the VPS's via the tunnel, and thus avoid exposing crowdsec's port 8080 publicly.
So, I installed newt on the pangolin instance to create a site on the VPS, with ACCEPT_CLIENTS=true.
I created an olm instance on the NAS, and changed the NAS crowdsec instance to go through olm's network stack.
On the NAS, running docker exec crowdsec ping [newt_vps_ip] works just fine, but docker exec crowdsec lapi register --machine NAS --url http://[newt_vps_ip]:8080 does not. docker exec crowdsec curl [newt_vps_ip]:8080 doesn't either.
I had port 8080 opened temporarily on crowdsec on the Pangolin-hosted VPS throughout my attempts, and I could verify that from the NAS's crowdsec instance, both docker exec crowdsec lapi register --machine NAS --url http://[vps_ip]:8080 and docker exec crowdsec curl [vps_ip]:8080 work.
On the Pangolin-hosted VPS, docker exec [container] curl crowdsec:8080 works from all containers.
So I'm a bit lost, I don't really understand how I can ping the VPS through the tunnel but not curl anything running there this way.
Am I missing something?
For reference, here are my compose yaml on both servers:
NAS
networks:
newt:
name: newt
ipam:
config:
- subnet: 172.20.127.0/24
ip_range: 172.20.127.0/26
gateway: 172.20.127.254
olm:
name: olm
ipam:
config:
- subnet: 172.20.100.0/24
ip_range: 172.20.100.0/26
gateway: 172.20.100.254
volumes:
crowdsec-db: null
services:
newt:
image: fosrl/newt
container_name: newt
restart: unless-stopped
networks:
- newt
environment:
- PANGOLIN_ENDPOINT=https://[my_public_pangolin_endpoint]
- NEWT_ID=redacted
- NEWT_SECRET=redacted
olm:
image: fosrl/olm
container_name: olm
restart: unless-stopped
networks:
- olm
cap_add:
- NET_ADMIN
devices:
- /dev/net/tun:/dev/net/tun
environment:
- PANGOLIN_ENDPOINT=https://[my_public_pangolin_endpoint]
- OLM_ID=redacted
- OLM_SECRET=redacted
crowdsec:
container_name: crowdsec
image: crowdsecurity/crowdsec
restart: always
network_mode: service:olm
volumes:
- crowdsec-db:/var/lib/crowdsec/data/
- /volume2/docker/appdata/crowdsec:/etc/crowdsec/VPS
name: pangolin
networks:
default:
driver: bridge
name: pangolin
volumes:
pangolin-data: null
services:
crowdsec:
command: -t
container_name: crowdsec
environment:
COLLECTIONS: crowdsecurity/traefik crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules
ENROLL_INSTANCE_NAME: pangolin-crowdsec
ENROLL_TAGS: docker
GID: "1000"
PARSERS: crowdsecurity/whitelists
healthcheck:
interval: 10s
retries: 15
test:
- CMD
- cscli
- capi
- status
timeout: 10s
image: docker.io/crowdsecurity/crowdsec:latest
labels:
- traefik.enable=false
ports:
- 6060:6060
# - 8080:8080 # wasn't commented out during my testing
restart: unless-stopped
volumes:
- ./config/crowdsec:/etc/crowdsec
- ./config/crowdsec/db:/var/lib/crowdsec/data
- ./config/traefik/logs:/var/log/traefik:ro
gerbil:
cap_add:
- NET_ADMIN
- SYS_MODULE
command:
- --reachableAt=http://gerbil:3003/
- --generateAndSaveKeyTo=/var/config/key
- --remoteConfig=http://pangolin:3001/api/v1/
container_name: gerbil
depends_on:
pangolin:
condition: service_healthy
image: docker.io/fosrl/gerbil:1.2.1
ports:
- 51820:51820/udp
- 21820:21820/udp
- 443:443
restart: unless-stopped
volumes:
- ./config/:/var/config
pangolin:
container_name: pangolin
healthcheck:
interval: 10s
retries: 15
test:
- CMD
- curl
- -f
- http://localhost:3001/api/v1/
timeout: 10s
image: docker.io/fosrl/pangolin:1.10.1
restart: unless-stopped
volumes:
- ./config:/app/config
- pangolin-data:/var/certificates
- pangolin-data:/var/dynamic
traefik:
command:
- --configFile=/etc/traefik/traefik_config.yml
container_name: traefik
depends_on:
crowdsec:
condition: service_healthy
pangolin:
condition: service_healthy
environment:
CLOUDFLARE_DNS_API_TOKEN: redacted
image: docker.io/traefik:v3.5
network_mode: service:gerbil
restart: unless-stopped
volumes:
- ./config/traefik:/etc/traefik:ro
- ./config/letsencrypt:/letsencrypt
- ./config/traefik/logs:/var/log/traefik
- pangolin-data:/var/certificates:ro
- pangolin-data:/var/dynamic:ro
newt:
image: fosrl/newt
container_name: newt
restart: unless-stopped
cap_add:
- NET_ADMIN
environment:
- PANGOLIN_ENDPOINT=https://[my_public_pangolin_endpoint]
- NEWT_ID=redacted
- NEWT_SECRET=redacted
- ACCEPT_CLIENTS=true
- USE_NATIVE_INTERFACE=true # tried with and without this