agent可观测--Langfuse阿里云自托管
上篇写了LiteLLM,网关是有了,但用起来是个黑盒,谁在调、一次agent任务烧了多少token、中间转了几手,全靠猜。决定上Langfuse,开源的LLM可观测平台,给调用链留个底。本以为docker compose一把梭的事,前后折腾了两个回合,记录一下
一.第一回合,官方全家桶,翻车
图省事,直接把官方docker-compose全家桶怼到网关机器的/opt/langfuse下,up -d之后一半容器卡在Created,纹丝不动
docker ps -a
#postgres/redis/clickhouse都healthy,minio/web/worker全是Created
netstat -tlpn
#9090被一个跑了七周的litellm_prometheus占着看compose才明白,全家桶里minio默认映射9090:9000,和机器上早就在跑的prometheus撞了,minio起不来,web/worker又配了depends_on: service_healthy强依赖,跟着全卡住。端口冲突本身是小事,改个映射就能继续,但这次翻车让我回头把文档看仔细了,官方自己说compose全家桶定位是"测试和小规模部署",postgres/clickhouse/redis/minio四件套全塞在应用机器上,和生产服务抢资源是迟早的事
总结:全家桶适合本地尝鲜,生产别偷懒。推倒重来
二.重新设计,该托管的全托管
langfuse v3 = web + worker两个应用容器,加四个基础设施:postgres存元数据,clickhouse存trace,redis做队列,S3对象存储存原始事件。官方几条硬性要求:
1.对象存储是强制依赖,不支持本地文件系统,生产用云托管
2.所有组件必须UTC时区(阿里云托管clickhouse默认Asia/Shanghai,必须改,改参数会触发集群重启)
3.多实例部署时,对象存储和数据库必须共享
最终架构:两台ECS各跑一组web+worker双活,前面挂内网SLB,基础设施全托管
内网SLB :3000
/ \
ECS-01 web+worker ECS-02 web+worker
\ /
RDS PG · 云ClickHouse · 云Redis · OSScompose关键配置,web/worker环境变量完全相同,worker端口只绑127.0.0.1:
services:
langfuse-web:
image: langfuse/langfuse:3.224.1
ports: ["3000:3000"]
environment:
DATABASE_URL: "postgresql://<user>:<pwd>@<rds>:5432/langfuse"
NEXTAUTH_SECRET: <随机32hex> #openssl rand -hex 32
SALT: <随机32hex>
ENCRYPTION_KEY: <随机64hex>
AUTH_DISABLE_SIGNUP: true #关注册,配合LANGFUSE_INIT_*无头建号
CLICKHOUSE_URL: http://<ch>:8123
CLICKHOUSE_MIGRATION_URL: "clickhouse://<ch>:9000"
REDIS_HOST: <redis>
REDIS_AUTH: <pwd>
#OSS,endpoint的写法是本文最大的坑,见下
LANGFUSE_S3_EVENT_UPLOAD_BUCKET: <bucket>
LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT: https://s3.oss-<region>-internal.aliyuncs.com
LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE: "false"
#首次初始化,幂等,只需一台配,管理员密码登录后改
LANGFUSE_INIT_PROJECT_PUBLIC_KEY: pk-lf-<uuid>
LANGFUSE_INIT_PROJECT_SECRET_KEY: sk-lf-<uuid>
langfuse-worker:
image: langfuse/langfuse-worker:3.224.1
ports: ["127.0.0.1:3030:3030"]
#环境变量与web相同,可去掉LANGFUSE_INIT_*三.第二回合,健康检查全绿,但一条trace都查不到
up -d,健康检查全绿,UI能登录,爽了没几分钟发现不对,怎么查都是空的,ingestion接口直接500,网关那头持续发事件过来,高峰30分钟丢了1811条。这里必须先记住v3的数据链路:
事件 → OSS → Redis队列 → worker → ClickHouse
OSS写失败,整条链路作废,但健康检查照样绿,health只代表进程活着,不代表链路通。翻web容器日志,Failed to upload刷屏,一共剥了三层:
1.endpoint带了bucket。配的https://<bucket>.oss-<region>-internal.aliyuncs.com,langfuse用的AWS S3 SDK,virtual-host模式下SDK会自己再拼一次bucket,实际请求成了bucket.bucket.oss-...,TLS证书直接对不上
2.没用S3兼容端点。改成https://oss-<region>-internal.aliyuncs.com,换了个报错:You have no right to access this object because of bucket acl。OSS原生端点不认AWS SigV4签名,带着签名的请求被当成了匿名访问。正确写法是https://s3.oss-<region>-internal.aliyuncs.com,注意开头的s3.
3.AK没权限。endpoint对了还不行,一查AK,复用了别的业务的RAM用户,人家的策略只授权自己的bucket。新建专用RAM用户,最小授权:
{"Version":"1","Statement":[{"Effect":"Allow",
"Action":["oss:PutObject","oss:GetObject","oss:DeleteObject","oss:ListObjects",
"oss:AbortMultipartUpload","oss:ListParts","oss:GetBucketLocation"],
"Resource":["acs:oss:*:*:<bucket>","acs:oss:*:*:<bucket>/*"]}]}三层修完,30分钟1811条的丢失归零
总结:别复用AK,别信health
四.验证四步,缺一不可
被"health绿了但其实全丢"教育之后,总结了四步验证,以后每次动完都跑一遍:
#1.健康检查,只代表进程活着
curl http://127.0.0.1:3000/api/public/health
curl http://127.0.0.1:3030/api/health
#2.API key有效性
curl -u pk-lf-xxx:sk-lf-xxx http://127.0.0.1:3000/api/public/projects
#3.写入端到端,关键,打穿OSS→队列→ClickHouse全链路
#期望successes里status 201,返回Internal Server Error就是OSS写入失败
curl -u pk-lf-xxx:sk-lf-xxx -X POST http://127.0.0.1:3000/api/public/ingestion \
-H 'Content-Type: application/json' \
-d '{"batch":[{"id":"evt-1","type":"trace-create","timestamp":"<UTC-ISO>",
"body":{"id":"trace-smoke-1","name":"smoke","timestamp":"<UTC-ISO>"}}]}'
#4.等30秒左右能查回来,验证ClickHouse读路径和时区
curl -u pk-lf-xxx:sk-lf-xxx 'http://127.0.0.1:3000/api/public/traces?limit=3'至于每个agent任务怎么在langfuse里变成一棵能搜的trace树,任务号、归树、session分组,又是一整篇的内容,未完待续