Tags: zillow/zkafka
Tags
cache resolved avro schema by schema ID in unmarshal (#35) ## Problem `avroSchemaRegistryFormatter.unmarshal` re-runs `GetSchema` + `avro.Parse` ×2 + `avro.NewSchemaCompatibility().Resolve` **on every message**. `Resolve` fingerprints the schema via `RecordSchema.String()` over the whole schema tree, which is extremely expensive for large schemas. Profiling a consumer of ~250–800KB `PropertyChangePayloadRecord` messages (a large, deeply-nested schema) showed the decode path allocating **~408 MB per message** — 91% of it in `hamba/avro (*RecordSchema).String` via `Fingerprint`/`SchemaCompatibility.compatible` — and burning ~100–190 ms of CPU per message, most of it GC churn from that allocation flood. ## Fix Cache the resolved (target←data) schema on the formatter, keyed by the message's 4-byte confluent schema ID. The target schema is fixed for a given formatter, and the writer schema is uniquely identified by that ID, so schema resolution now runs **once per schema ID** instead of once per message. Held via `*sync.Map` so the value-copied formatter shares one cache. ## Measured impact (200-message benchmark, same corpus) | Metric | Before | After | |---|---|---| | hot-path time / msg | ~104–194 ms | **4.5 ms** (~25–40×) | | total allocations | 81.6 GB (~408 MB/msg) | **1.13 GB** (~5.6 MB/msg, ~72×) | | consumer CPU busy | 53% | **3%** (now network-bound) | Residual `RecordSchema.String` allocation is the one-time cache fill on the first message per schema ID. Existing formatter and evolution (`test/evolution/*`) tests pass. > Draft / alpha for review — flagged for discussion before finalizing (naming, cache-key strategy, eviction if ever needed). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: stewartboyd119 <stewartb@zillowgroup.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
RMPMP-2110: add circuit breaker lifecycle hooks (#33) ## RMPMP-2110: Add circuit breaker lifecycle hooks - Add `PostCircuitBreakerOpened` and `PostCircuitBreakerClosed` to `LifecycleHooks`, with empty meta structs (`LifecyclePostCircuitBreakerOpened` / `LifecyclePostCircuitBreakerClosed`) so additional fields can be introduced without breaking changes. - Chain the new hooks in `ChainLifecycleHooks`. - Invoke the hooks from the gobreaker `OnStateChange` handler in `Work` factory wiring on transitions to `StateOpen` / `StateClosed`. --------- Co-authored-by: stewartboyd119 <stewartb@zillowgroup.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
rmpmp-2040: upgrade to go 1.26, update dependencies (#32) 1. go1.26 upgrade 2. Updated direct deps 3. Updated golangci to use version 2 (updated github action) 4. Use go fix ./... to modernize codebase (use 1.26 features) --------- Co-authored-by: stewartboyd119 <stewartb@zillowgroup.com>
Bugfix: compatible schemas not properly handled during deserialization ( #25) Forward/Backwards compatible schemas used to produce/consume messages were not being properly deserialized. `Test_SchemaRegistryReal_Avro_AutoRegisterSchemas_OldProducerCanBeConsumedByNewConsumer` and `Test_SchemaRegistryReal_Avro_AutoRegisterSchemas_NewProducerCanBeConsumedByOldConsumer` demonstrated bug. Schema registry aware avro formatter was updated to reconcile the bug. --------- Co-authored-by: stewartboyd119 <stewartb@zillowgroup.com>
Lifecycle bugfix tests added `Test_LifecycleChainedNil*Invocation` (#22) 1. Added supporting tests for bugfix submitted previously 2. Updated `TestMessage_Value` usage to use pointers of slices (testify must have changed implementation. Test complains otherwise) 3. Updated deps 4. Added changelog entry in anticipation of release --------- Co-authored-by: stewartboyd119 <stewartb@zillowgroup.com>
PreviousNext