Skip to content

high etcd memory usage. #21713

Open
Open
high etcd memory usage.#21713
Labels
type/bug
@molinsoma

Description

@molinsoma
molinsoma
opened on May 6, 2026

Bug report criteria

What happened?

I have an etcd cluster with three nodes. The cluster is running with no load, and the revision is 1. The memory usage of node 1 is 114 MB, that of node 2 is 44 MB, and that of node 3 is 91 MB. Node 3 is the leader node. When the cluster is running with no load, the difference between the memory usage of node 1 and that of node 2 is 70 MB, which is unreasonable.

What did you expect to happen?

The memory usage of nodes 1, 2, and 3 is the same, and the memory usage is 44 MB.

How can we reproduce it (as minimally and precisely as possible)?

The memory usage of nodes 1, 2, and 3 is the same, which is about 44 MB. The memory usage of the leader node can be slightly higher.

Anything else we need to know?

No response

Etcd version (please run commands below)

Details 
$ etcd --version
# paste output here
v3.6.7
$ etcdctl version
# paste output here

Etcd configuration (command line flags or environment variables)

Details

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

Details 
$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

File: etcd
Build ID: eb0bedbf9c3f6d56a9846d2ba7088798683f4da0
Type: inuse_space
Time: 2026-04-29 15:49:17 CST
Showing nodes accounting for 19037.29kB, 100% of 19037.29kB total
      flat  flat%   sum%        cum   cum%
 3587.97kB 18.85% 18.85%  3587.97kB 18.85%  go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.startPeer
 3587.97kB 18.85% 37.69%  3587.97kB 18.85%  go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.(*streamWriter).closeUnlocked
 2048.75kB 10.76% 48.46%  2048.75kB 10.76%  runtime.mallocgc
 1184.27kB  6.22% 54.68%  1184.27kB  6.22%  go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.newMsgAppV2Decoder (inline)
 1184.27kB  6.22% 60.90%  1184.27kB  6.22%  go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.newMsgAppV2Encoder (inline)
 1024.47kB  5.38% 66.28%  1024.47kB  5.38%  time.goFunc
  768.26kB  4.04% 70.31%   768.26kB  4.04%  go.uber.org/zap/zapcore.newCounters (inline)
  525.43kB  2.76% 73.07%   525.43kB  2.76%  go.etcd.io/etcd/server/v3/storage/backend.newBucketBuffer (inline)
     514kB  2.70% 75.77%      514kB  2.70%  bufio.NewWriterSize (inline)
  513.50kB  2.70% 78.47%   513.50kB  2.70%  google.golang.org/grpc/reflection/grpc_reflection_v1.init
  513.31kB  2.70% 81.17%   513.31kB  2.70%  google.golang.org/protobuf/internal/impl.consumeBytesNoZero
  512.88kB  2.69% 83.86%   512.88kB  2.69%  go.etcd.io/etcd/server/v3/etcdserver/api/v2error.map.init.0
  512.09kB  2.69% 86.55%   512.09kB  2.69%  google.golang.org/protobuf/internal/filedesc.(*Enum).unmarshalFull
  512.03kB  2.69% 89.24%   512.03kB  2.69%  github.com/prometheus/client_golang/prometheus.MakeLabelPairs
  512.03kB  2.69% 91.93%   512.03kB  2.69%  github.com/prometheus/client_golang/prometheus.inlineLabelValues (inline)
  512.02kB  2.69% 94.62%  1536.09kB  8.07%  github.com/prometheus/client_golang/prometheus.(*metricMap).getOrCreateMetricWithLabelValues
  512.02kB  2.69% 97.31%   512.02kB  2.69%  github.com/prometheus/client_golang/prometheus.(*metricMap).getOrCreateMetricWithLabels
  512.01kB  2.69%   100%   512.01kB  2.69%  google.golang.org/grpc/internal/buffer.NewUnbounded (inline)
         0     0%   100%  1025.32kB  5.39%  crypto/tls.(*Conn).HandshakeContext (inline)
         0     0%   100%  1025.32kB  5.39%  crypto/tls.(*Conn).clientHandshake
         0     0%   100%  1025.32kB  5.39%  crypto/tls.(*Conn).getClientCertificate
         0     0%   100%  1025.32kB  5.39%  crypto/tls.(*Conn).handshakeContext
         0     0%   100%  1025.32kB  5.39%  crypto/tls.(*clientHandshakeStateTLS13).handshake
         0     0%   100%  1025.32kB  5.39%  crypto/tls.(*clientHandshakeStateTLS13).sendClientCertificate
         0     0%   100%   512.09kB  2.69%  github.com/golang/protobuf/proto.(*TextMarshaler).Text (inline)
         0     0%   100%   512.09kB  2.69%  github.com/golang/protobuf/proto.(*TextMarshaler).marshal
         0     0%   100%   512.09kB  2.69%  github.com/golang/protobuf/proto.(*textWriter).writeMessage
         0     0%   100%   512.09kB  2.69%  github.com/golang/protobuf/proto.CompactTextString (inline)
         0     0%   100%  1536.09kB  8.07%  github.com/grpc-ecosystem/go-grpc-middleware/providers/prometheus.(*ServerMetrics).InitializeMetrics
         0     0%   100%  1536.09kB  8.07%  github.com/grpc-ecosystem/go-grpc-middleware/providers/prometheus.(*ServerMetrics).preRegisterMethod
         0     0%   100%  1536.09kB  8.07%  github.com/prometheus/client_golang/prometheus.(*CounterVec).GetMetricWithLabelValues (inline)
         0     0%   100%   512.02kB  2.69%  github.com/prometheus/client_golang/prometheus.(*GaugeVec).GetMetricWith (inline)
         0     0%   100%   512.02kB  2.69%  github.com/prometheus/client_golang/prometheus.(*GaugeVec).With
         0     0%   100%   512.02kB  2.69%  github.com/prometheus/client_golang/prometheus.(*MetricVec).GetMetricWith
         0     0%   100%  1536.09kB  8.07%  github.com/prometheus/client_golang/prometheus.(*MetricVec).GetMetricWithLabelValues
         0     0%   100%   512.03kB  2.69%  github.com/prometheus/client_golang/prometheus.v2.NewCounterVec.func1
         0     0%   100%   512.09kB  2.69%  go.etcd.io/etcd/api/v3/etcdserverpb.(*InternalRaftRequest).String (inline)
         0     0%   100%   512.09kB  2.69%  go.etcd.io/etcd/api/v3/etcdserverpb.(*InternalRaftStringer).String
         0     0%   100%  1025.32kB  5.39%  go.etcd.io/etcd/client/pkg/v3/tlsutil.NewCert
         0     0%   100%  1025.32kB  5.39%  go.etcd.io/etcd/client/pkg/v3/transport.TLSInfo.baseConfig.func7
         0     0%   100%   512.09kB  2.69%  go.etcd.io/etcd/pkg/v3/schedule.(*fifo).executeJob
         0     0%   100%   512.09kB  2.69%  go.etcd.io/etcd/pkg/v3/schedule.(*fifo).run
         0     0%   100%   512.09kB  2.69%  go.etcd.io/etcd/pkg/v3/schedule.job.Do
         0     0%   100%   525.43kB  2.76%  go.etcd.io/etcd/server/v3/auth.NewAuthStore
         0     0%   100%   768.26kB  4.04%  go.etcd.io/etcd/server/v3/embed.(*Config).Validate
         0     0%   100%   768.26kB  4.04%  go.etcd.io/etcd/server/v3/embed.(*Config).setupLogging
         0     0%   100%  1536.09kB  8.07%  go.etcd.io/etcd/server/v3/embed.(*Etcd).serveClients.func1
         0     0%   100%  1536.09kB  8.07%  go.etcd.io/etcd/server/v3/embed.(*Etcd).startHandler.func1
         0     0%   100%   768.26kB  4.04%  go.etcd.io/etcd/server/v3/embed.(*configYAML).configFromFile
         0     0%   100%  1536.09kB  8.07%  go.etcd.io/etcd/server/v3/embed.(*serveCtx).serve
         0     0%   100%   768.26kB  4.04%  go.etcd.io/etcd/server/v3/embed.ConfigFromFile
         0     0%   100%  4625.42kB 24.30%  go.etcd.io/etcd/server/v3/embed.StartEtcd
         0     0%   100%   768.26kB  4.04%  go.etcd.io/etcd/server/v3/etcdmain.(*config).configFromFile (inline)
         0     0%   100%   768.26kB  4.04%  go.etcd.io/etcd/server/v3/etcdmain.(*config).parse
         0     0%   100%  5393.68kB 28.33%  go.etcd.io/etcd/server/v3/etcdmain.Main
         0     0%   100%  4625.42kB 24.30%  go.etcd.io/etcd/server/v3/etcdmain.startEtcd
         0     0%   100%  5393.68kB 28.33%  go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2
         0     0%   100%   512.02kB  2.69%  go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).Start
         0     0%   100%   512.09kB  2.69%  go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).apply
         0     0%   100%   512.09kB  2.69%  go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyAll
         0     0%   100%   512.09kB  2.69%  go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntries
         0     0%   100%   512.09kB  2.69%  go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntryNormal
         0     0%   100%   512.09kB  2.69%  go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyInternalRaftRequest
         0     0%   100%   512.09kB  2.69%  go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyInternalRaftRequest.func1
         0     0%   100%   512.09kB  2.69%  go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).run.func6
         0     0%   100%   512.02kB  2.69%  go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).start
         0     0%   100%  4113.40kB 21.61%  go.etcd.io/etcd/server/v3/etcdserver.NewServer
         0     0%   100%  3587.97kB 18.85%  go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.(*Transport).AddPeer
         0     0%   100%  1184.27kB  6.22%  go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.(*streamReader).decodeLoop
         0     0%   100%  1184.27kB  6.22%  go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.(*streamReader).run
         0     0%   100%  3587.97kB 18.85%  go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.(*streamWriter).close
         0     0%   100%  4772.24kB 25.07%  go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.(*streamWriter).run
         0     0%   100%   512.88kB  2.69%  go.etcd.io/etcd/server/v3/etcdserver/api/v2error.init
         0     0%   100%  1536.09kB  8.07%  go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.Server
         0     0%   100%   525.43kB  2.76%  go.etcd.io/etcd/server/v3/etcdserver/cindex.(*consistentIndex).UnsafeSave
         0     0%   100%   512.09kB  2.69%  go.etcd.io/etcd/server/v3/etcdserver/txn.WarnOfExpensiveRequest
         0     0%   100%   512.09kB  2.69%  go.etcd.io/etcd/server/v3/etcdserver/txn.warnOfExpensiveGenericRequest
         0     0%   100%   525.43kB  2.76%  go.etcd.io/etcd/server/v3/storage.(*BackendHooks).OnPreCommitUnsafe
         0     0%   100%   525.43kB  2.76%  go.etcd.io/etcd/server/v3/storage/backend.(*batchTxBuffered).Commit
         0     0%   100%   525.43kB  2.76%  go.etcd.io/etcd/server/v3/storage/backend.(*batchTxBuffered).UnsafePut
         0     0%   100%   525.43kB  2.76%  go.etcd.io/etcd/server/v3/storage/backend.(*batchTxBuffered).commit
         0     0%   100%   525.43kB  2.76%  go.etcd.io/etcd/server/v3/storage/backend.(*batchTxBuffered).unsafeCommit
         0     0%   100%   525.43kB  2.76%  go.etcd.io/etcd/server/v3/storage/backend.(*txWriteBuffer).put
         0     0%   100%   525.43kB  2.76%  go.etcd.io/etcd/server/v3/storage/backend.(*txWriteBuffer).putInternal
         0     0%   100%   525.43kB  2.76%  go.etcd.io/etcd/server/v3/storage/schema.(*authBatchTx).Unlock
         0     0%   100%   525.43kB  2.76%  go.etcd.io/etcd/server/v3/storage/schema.UnsafeUpdateConsistentIndex (inline)
         0     0%   100%   525.43kB  2.76%  go.etcd.io/etcd/server/v3/storage/schema.unsafeUpdateConsistentIndex
         0     0%   100%   768.26kB  4.04%  go.uber.org/zap.(*Logger).WithOptions
         0     0%   100%   768.26kB  4.04%  go.uber.org/zap.Config.Build
         0     0%   100%   768.26kB  4.04%  go.uber.org/zap.Config.buildOptions.WrapCore.func5
         0     0%   100%   768.26kB  4.04%  go.uber.org/zap.Config.buildOptions.func1
         0     0%   100%   768.26kB  4.04%  go.uber.org/zap.New
         0     0%   100%   768.26kB  4.04%  go.uber.org/zap.optionFunc.apply
         0     0%   100%   768.26kB  4.04%  go.uber.org/zap/zapcore.NewSamplerWithOptions
         0     0%   100%   513.31kB  2.70%  google.golang.org/grpc.(*ClientConn).Invoke
         0     0%   100%   512.01kB  2.69%  google.golang.org/grpc.(*ClientConn).initIdleStateLocked
         0     0%   100%   513.31kB  2.70%  google.golang.org/grpc.(*clientStream).RecvMsg
         0     0%   100%   513.31kB  2.70%  google.golang.org/grpc.(*clientStream).RecvMsg.func1
         0     0%   100%   513.31kB  2.70%  google.golang.org/grpc.(*clientStream).withRetry
         0     0%   100%   513.31kB  2.70%  google.golang.org/grpc.(*csAttempt).recvMsg
         0     0%   100%   512.01kB  2.69%  google.golang.org/grpc.Dial (inline)
         0     0%   100%   512.01kB  2.69%  google.golang.org/grpc.DialContext
         0     0%   100%   512.01kB  2.69%  google.golang.org/grpc.NewClient
         0     0%   100%   513.31kB  2.70%  google.golang.org/grpc.invoke
         0     0%   100%   512.01kB  2.69%  google.golang.org/grpc.newCCResolverWrapper
         0     0%   100%   513.31kB  2.70%  google.golang.org/grpc.recv
         0     0%   100%   513.31kB  2.70%  google.golang.org/grpc/encoding/proto.(*codecV2).Unmarshal
         0     0%   100%   512.01kB  2.69%  google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer
         0     0%   100%   512.09kB  2.69%  google.golang.org/protobuf/internal/filedesc.(*File).lazyInit (inline)
         0     0%   100%   512.09kB  2.69%  google.golang.org/protobuf/internal/filedesc.(*File).lazyInitOnce
         0     0%   100%   512.09kB  2.69%  google.golang.org/protobuf/internal/filedesc.(*File).lazyRawInit
         0     0%   100%   512.09kB  2.69%  google.golang.org/protobuf/internal/filedesc.(*File).unmarshalFull
         0     0%   100%   512.09kB  2.69%  google.golang.org/protobuf/internal/filedesc.(*Message).RequiredNumbers
         0     0%   100%   512.09kB  2.69%  google.golang.org/protobuf/internal/filedesc.(*Message).lazyInit (inline)
         0     0%   100%   512.09kB  2.69%  google.golang.org/protobuf/internal/filedesc.(*Message).unmarshalFull
         0     0%   100%   512.09kB  2.69%  google.golang.org/protobuf/internal/impl.(*MessageInfo).init (inline)
         0     0%   100%   512.09kB  2.69%  google.golang.org/protobuf/internal/impl.(*MessageInfo).initOnce
         0     0%   100%   512.09kB  2.69%  google.golang.org/protobuf/internal/impl.(*MessageInfo).makeCoderMethods
         0     0%   100%   513.31kB  2.70%  google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshal
         0     0%   100%   513.31kB  2.70%  google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshalPointer
         0     0%   100%   513.31kB  2.70%  google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshalPointerEager
         0     0%   100%   512.09kB  2.69%  google.golang.org/protobuf/internal/impl.(*messageReflectWrapper).Has
         0     0%   100%   512.09kB  2.69%  google.golang.org/protobuf/internal/impl.needsInitCheck
         0     0%   100%   512.09kB  2.69%  google.golang.org/protobuf/internal/impl.needsInitCheckLocked
         0     0%   100%   513.31kB  2.70%  google.golang.org/protobuf/proto.Unmarshal
         0     0%   100%   513.31kB  2.70%  google.golang.org/protobuf/proto.UnmarshalOptions.unmarshal
         0     0%   100%  5393.68kB 28.33%  main.main
         0     0%   100%      514kB  2.70%  net/http.(*conn).serve
         0     0%   100%  1025.32kB  5.39%  net/http.(*persistConn).addTLS.func2
         0     0%   100%      514kB  2.70%  net/http.newBufioWriterSize
         0     0%   100%   512.05kB  2.69%  runtime.(*scavengerState).init
         0     0%   100%  1024.47kB  5.38%  runtime.(*timer).unlockAndRun
         0     0%   100%  1024.47kB  5.38%  runtime.(*timers).check
         0     0%   100%  1024.47kB  5.38%  runtime.(*timers).run
         0     0%   100%  1024.47kB  5.38%  runtime.asyncPreempt2.func1
         0     0%   100%   512.05kB  2.69%  runtime.bgscavenge
         0     0%   100%  1026.38kB  5.39%  runtime.doInit (inline)
         0     0%   100%  1026.38kB  5.39%  runtime.doInit1
         0     0%   100%  1024.47kB  5.38%  runtime.findRunnable
         0     0%   100%  1024.47kB  5.38%  runtime.gopreempt_m (inline)
         0     0%   100%  1024.47kB  5.38%  runtime.goschedImpl
         0     0%   100%  6420.05kB 33.72%  runtime.main
         0     0%   100%  1536.70kB  8.07%  runtime.malg
         0     0%   100%  1024.47kB  5.38%  runtime.mcall
         0     0%   100%  2048.75kB 10.76%  runtime.newobject
         0     0%   100%  1536.70kB  8.07%  runtime.newproc.func1
         0     0%   100%  1536.70kB  8.07%  runtime.newproc1
         0     0%   100%  1024.47kB  5.38%  runtime.schedule
         0     0%   100%  1536.70kB  8.07%  runtime.systemstack

Metadata

Metadata

Assignees

No one assigned

    Labels

    type/bug

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions