Skip to content

Performance: lock contention and goroutine leak in prepared statement cache / LRU #7788

@pageton

Description

@pageton

Describe the feature

Reduce lock contention in the prepared statement cache and fix LRU goroutine leaks.

Motivation

1. Prepared statement write lock held during PrepareContext I/O — prepare_stmt.go:66-86

All goroutines that miss the LRU lookup contend on a single sync.RWMutex write lock. The write lock is held for the entire duration of PrepareContext — a network I/O call to the database. Under concurrent load with diverse queries, this serializes all prepare operations and blocks readers too.

Fix: Use singleflight.Group per query key. Let the first goroutine prepare the statement while others wait on the existing prepared channel. Remove the global write lock from the prepare path.

2. LRU Get() uses exclusive write lock — internal/lru/lru.go:147-160

func (c *LRU[K, V]) Get(key K) (value V, ok bool) {
    c.mu.Lock()  // exclusive write lock for a read!
    defer c.mu.Unlock()
    if ent, ok = c.items[key]; ok {
        c.evictList.MoveToFront(ent)  // mutates list order
    }
}

Every cache hit acquires an exclusive write lock. Combined with the PreparedStmtDB.Mux.RLock(), there are two levels of locking for every query. All concurrent prepared statement lookups are serialized.

Fix: Use sharded locks (lock striping), or an LRU design that allows read-mostly access (e.g., sync.Map + approximate LRU via TTL-based eviction).

3. LRU Close() commented out — goroutine leak — internal/lru/lru.go:293-303

The background deleteExpired goroutine started at line 80 runs forever — Close() is commented out. When a *sql.DB is closed and reopened, the old LRU's goroutine leaks. Every DB instance with prepared-stmt cache creates one.

Fix: Uncomment and expose Close(). Call it from PreparedStmtDB.Close().

4. deleteExpired blocks all access during eviction — internal/lru/lru.go:324-339

deleteExpired calls removeElement under the LRU mutex, which invokes onEvictgo v.Close(). This blocks all Get/Add operations during eviction. Also, v.Close() blocks on <-stmt.prepared — if PrepareContext is slow, goroutines accumulate.

Fix: Collect entries to evict under lock, release lock, then evict outside the critical section.

5. Unbounded goroutine spawn on LRU eviction — stmt_store/stmt_store.go:108-112

onEvicted := func(k string, v *Stmt) {
    if v != nil {
        go v.Close()  // unbounded goroutine spawn
    }
}

During cache pressure, this can spawn thousands of goroutines.

Fix: Use a semaphore or bounded worker pool for Close calls.

Impact

Under high-concurrency workloads with PrepareStmt: true, the lock contention in findings 1 and 2 is likely the primary throughput limiter. The goroutine leak in finding 3 causes steady memory growth in long-running services that create/destroy DB instances.

Related Issues

  • Performance architecture review of GORM codebase

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions