• Improvement: The build store dashboard and OTel metrics now surface blob-exists query effectiveness. A new "Build Exists Queries" dashboard tile shows body and metadata hit/miss counts and ratios, and OTel gains per-operation build store counters for reads, writes, and hits (by blob/metadata), exists queries/hits/misses (by body/metadata), request timing, and bad requests
• Feature: zen cache drop can now drop a single DDC cache record, not just a whole namespace or bucket. Pass the record's value key with -v/--valuekey (requires -b/--bucket), or positionally as zen cache drop <namespace> <bucket> <valuekey>. This issues DELETE /z$/{namespace}/{bucket}/{hash}, a route that previously rejected DELETE with 405; the record is tombstoned in the bucket log so the deletion survives a server restart, and its now-unreferenced attachments are reclaimed by the normal reference-based GC. Only that record is removed - the bucket, namespace, and sibling records are untouched
• Feature: OpenTelemetry signals can be tunnelled through the UE Trace stream and replayed to a collector later, with no OTLP endpoint required at capture time. Capture with --trace=otel (or the granular otelspans / otellogs / otelmetrics channels) to record spans, logs, and metrics as byte-faithful OTLP into the .utrace file, then forward them with zen trace relay <file> --endpoint <uri> (--signal=traces,logs,metrics, default all). zen trace spans / zen trace logs / zen trace metrics list and decode the captured signals (add --verbose for attributes / fields / metric breakdown). Independent of the live OTLP/HTTP export and of the lossy UE-counter mirror (--trace=metrics)
• Improvement: zenserver flushes a final metrics tick on shutdown, so the last sub-interval window of metrics is exported (a closing /v1/metrics POST) and captured rather than discarded — short-lived runs now surface their metrics. Mirrors the zen CLI, which already reports once at shutdown
• Bugfix: The build store metadata hit ratio is no longer always zero. The metadata read-hit and write counters were declared but never incremented, so the dashboard's metadata hit ratio and stats stayed at zero regardless of activity
• Bugfix: Hub mode is more robust under concurrent storage-instance lifecycle operations. A storage instance whose deprovision, recovery, or obliterate stalls no longer blocks every other hub operation, and a crash the watchdog detects while an instance is being deprovisioned at the same moment no longer overwrites that operation's state and can no longer leave the instance's slot inconsistent or stuck
• Bugfix: An oplog import that is missing a chunk now fails instead of completing silently (unless missing attachments are ignored). It detects a chunk referenced by the oplog but absent from the container, a store returning fewer chunks than requested, and any needed block chunk not stored before the import is committed
• Bugfix: Garbage collection no longer reclaims a chunk referenced by an oplog the running GC pass had not yet picked up (a freshly created oplog, or the first in a project). Building it over HTTP or importing into it during a GC pass could leave the committed op referencing an already-deleted chunk, surfacing only on a later read. Referenced chunks are now pinned until a later pass picks up the oplog's on-disk state
• Bugfix: Hub dehydration no longer fails when S3 throttles the in-place metadata refresh. S3 can answer the refresh (a PUT copy) with an HTTP 200 whose body carries a SlowDown error, which slipped past the status-based retry path and failed the whole dehydration; the refresh is now retried with exponential backoff before the error is surfaced

• Improvement: The OpenTelemetry host.id resource attribute now follows the OTel semantic conventions on cloud VMs — when running on AWS, Azure, or GCP it reports the provider-assigned instance id (detected via the instance metadata service), falling back to the per-machine id (Windows MachineGuid / Linux /etc/machine-id / Darwin host UUID) on bare metal. On a detected cloud VM the exporter also emits cloud.provider (aws/azure/gcp), host.type (machine/instance type), and the host.image.id/host.image.name/host.image.version attributes that the provider's metadata exposes. Detection runs on the exporter's worker thread so it never delays startup, and a host.id supplied via OTEL_RESOURCE_ATTRIBUTES still takes precedence
• Feature: --background on zen builds download and zen oplog-import runs the transfer at a reduced worker/connection count (half the normal count, capped at 4) so it competes less for bandwidth. Slower than a normal transfer. Conflicts with --boost-workers/--boost-worker-count
• Feature: In hub mode, a managed storage instance that crashes or fails to launch now has its zenserver.log copied to <data-dir>/servers/crash-logs/<module>/ (most recent few retained per module) before the state dir is wiped or rehydrated, so the crash diagnostics survive recovery. The watchdog also caps consecutive fast crash-recovery attempts per instance: one that keeps crashing within a short uptime window is torn down instead of being respawned forever (e.g. a module too large for the host's memory), while an isolated crash after healthy uptime still recovers as before
• Feature: --register-server (Lua: server.registerserver, default true) controls whether a server registers in the host-global zen state table that backs discovery by zen status/top/ui. Pass --register-server=false to run unregistered. A hub now spawns its per-module storage children unregistered by default, so a single hub scales past the previous fixed ~1024 running-instance limit
• Feature: zenserver's Lua config can now set process environment variables via setenv(name, value) (and remove them with setenv(name, nil)); the companion getenv reads them back. The OTEL_* endpoint/sampler/kill-switch variables are resolved after the config script runs, so a script can export any OpenTelemetry SDK variable (headers, resource attributes, per-signal overrides, compression, timeouts, sampler) and have it honored as if set in the launching shell. The redundant server.tracesampler / server.tracesamplerarg Lua keys are removed in favor of --trace-sampler/--trace-sampler-arg or setenv("OTEL_TRACES_SAMPLER", ...). See docs/OpenTelemetry.md
• Bugfix: A connection reset mid-upload that curl cannot replay (curl error 65, Send failed since rewinding of the data stream failed) is now treated as a transient transport error and retried by clients that opt into retries, instead of failing as a permanent client error. Previously a single recoverable connection drop aborted an entire zen builds upload with a non-retryable exit code
• Bugfix: System memory usage on Linux no longer counts reclaimable page cache as "used", which could report a busy host as nearly out of memory and mislead memory-based autoscaling; it now uses the kernel's MemAvailable. MacOS also counts purgeable and speculative pages as available
• Bugfix: A spawned server that exits during startup (before signaling ready) is now reaped immediately when the spawn fails, instead of being left as a zombie process the hub watchdog would repeatedly report. On Linux an exited child stays a zombie until reaped, and the running-state check detects but does not reap it
• Bugfix: zen cache get now actually writes the output file when the fetched value or attachment is decompressed (the default). The write step tested the move-into-place result with an inverted condition, so decompressed in-memory data took the skip-write branch while still logging Wrote <N> to '<path>', leaving the output path empty

• Feature: zen builds download --download-spec-path (JSON form) accepts a per-file byte range, materializing only that extent of a large file (e.g. a header) while other listed files download in full. The file is written trimmed to exactly the requested [start, end) bytes. An optional outputpath writes the file to a different relative target; range and outputpath are independent and combinable, and the same path may appear in several entries to write different parts to different targets (targets must not conflict, and must stay within the download folder). Works with --append. A files entry is a path string (whole file) or an object with a path plus optional range and outputpath:

  {
      "parts": {
          "default": {
              "files": [
                  "meta/info.json",
                  { "path": "bulk/data.bin", "range": [0, 65536] },
                  { "path": "bulk/data.bin", "range": [0, 65536], "outputpath": "bulk/data0-65536.bin" }
              ]
          }
      }
  }
  

• Feature: GET /admin/logs/{name} serves the contents of a server log file as text, where name is default (the main zenserver.log), http (the per-request http.log), or cache (the cache_access.log); unknown names return 404

• Feature: Runtime lock-order (deadlock potential) detection for internal reader-writer locks, modeled on Abseil Mutex deadlock detection / FreeBSD WITNESS. Every "acquire B while holding A" records an edge in a global acquired-before graph; an acquisition that would close a cycle is reported immediately with the callstacks that established each ordering - the deadlocking thread interleaving never has to occur. Also detects same-thread re-acquisition (self deadlock). Always on in debug builds (report-only by default; modes: disabled/report/abort), opt-in for release via xmake config --mutexgraph=y. Lock scopes handed across threads (e.g. hub async provisioning) are tracked correctly via per-acquisition tokens

• Feature: Experimental Linux arm64 (aarch64) support - zenserver and the zen CLI build and run on Linux arm64 via the UE cross-compile toolchain, with mimalloc enabled and a bundled SeaweedFS arm64 binary for hub storage. Experimental and not shipped in official releases

• Improvement: OTLP metric temporality is selectable via the standard OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE (delta | cumulative); default delta preserves prior behavior. Set cumulative for Prometheus-based backends such as Grafana Mimir that reject delta series

• Improvement: Histograms export as explicit-bucket OTLP Histogram by default (was base-2 exponential), matching the OTel spec default and backends that drop OTLP exponential histograms (e.g. SigNoz, Grafana Mimir). Selectable via the standard OTEL_EXPORTER_OTLP_METRICS_DEFAULT_HISTOGRAM_AGGREGATION (explicit_bucket_histogram | base2_exponential_bucket_histogram); the zen-specific ZEN_OTEL_HISTOGRAM_FORMAT (explicit | exponential) remains as an override

• Improvement: The dashboard Telemetry tab's configuration summary now shows the resolved metric temporality (delta/cumulative) and histogram format (explicit/exponential), so the active OTLP metric export shape is visible alongside endpoint, sampler, and intervals

• Improvement: Lower peak memory and faster project store oplog load and validation. CAS log and file scan reads also skip buffer zero-initialization

• Improvement: Bump mimalloc to 3.2.8

• Improvement: mimalloc is now enabled on Arm64 targets (previously forced off); Apple Silicon validated

• Improvement: Locks that are held across HTTP, disk, or log-sink I/O in the auth manager, Horde/Nomad provisioners, upstream cache, and async log sink now use the instrumented reader-writer lock so they participate in lock-order cycle detection

• Improvement: Rejected HTTP requests (malformed body, parameters, headers, or verb) across the cache, build store, project store, object store, workspaces, sessions, and compute services now emit a structured warning carrying the request id, client address, verb, full URI, content type, and - when a body is present - its size, a hex preview, and an ASCII rendering of the leading bytes. The typed fields are exported as OTLP log attributes in addition to the rendered text line

• Bugfix: Fixed two recursive shared-lock acquisitions on the garbage collector's project-store precache path. Recursive shared acquisition of a reader-writer lock is undefined behavior and can deadlock on Windows SRWLOCK when a writer is waiting between the two acquisitions

• Bugfix: ZEN_OTEL_DUMP_DIR .txt metric dumps now contain the full per-scope metrics. The text encoder ignored the scope-organized snapshot and emitted a single empty zen scope, so dumps appeared metric-less even though the sibling .pb payload (the actual wire bytes) was complete. Binary OTLP export was unaffected

• Bugfix: HTTP PUT/POST uploads no longer fail spuriously with Send failed since rewinding of the data stream failed (curl error 65) when a request body has to be replayed (dropped reused connection, redirect, or rejected Expect: 100-continue). Affected hub S3 dehydration uploads such as the incremental metadata write, which previously forced a full re-upload on the next cycle

• Bugfix: Build store blob and metadata uploads now return 507 Insufficient Storage when disk writes are blocked, matching the cache and project store

• Bugfix: Fixed a rare hub crash (process abort) when a storage server instance was deprovisioned, recovered, or obliterated while the watchdog was concurrently inspecting it. The instance could be destroyed while a lock on it was still held by another thread, so releasing that lock then touched freed memory; on macOS this surfaced as mutex lock failed: Invalid argument thrown out of a noexcept destructor (an immediate process abort), and was undefined behavior on other platforms. Storage server instances are now reference counted, so a held lock keeps the instance alive until the lock is released

• Feature: Added zen builds status command to inspect a previously-downloaded folder's recorded build status.
  • --verify re-hashes all tracked files against the saved state; on failure prompts whether to delete .zen/current_state.cbo and reset
  • --detailed adds the per-build metadata payload, files tracked, and total size
  • -y/--yes skips the prompt and resets immediately when verify fails
• Feature: OTel hub metrics under the zen.hub scope: instance count per state, zen.hub.instance.crashes counter, instance and machine memory/disk usage and limit gauges, and zen.hub.provision.* / zen.hub.deprovision.* phase queue/duration and end-to-end totals (Histogram, ms) labelled by result. Aggregate-only, no per-module label. See docs/hub.md
• Feature: OTel hydration metrics under zen.hub.hydration.*: hydrate/dehydrate/obliterate/gc operation timings, per-phase request/byte/duration counters labelled by phase (hash, upload, touch, pack_upload, pack_touch, download, pack_download, metadata), and cache hit/miss/reuse and disk-usage gauges
• Feature: Hub dashboard "Operations" tile with Provision/Deprovision success/total and p50/p95/p99 latency, backend transfer (requests/s, throughput, per-request latency, async-client queue wait), and hydration cache hit rate and bytes reused
• Feature: Added --deployment-environment <name> (Lua: server.deploymentenvironment, zenserver only) to set the OTel deployment.environment.name resource attribute (e.g. prod, staging, dev). An explicit value wins over one supplied through OTEL_RESOURCE_ATTRIBUTES; unset omits the attribute
• Improvement: zen builds download now persists the resolved build part names and the build metadata payload into .zen/current_state.cbo
• Improvement: Hub creates the configured --hub-hydration-cache-path directory on startup if missing
• Improvement: Hub dehydration log suppresses the Cache snapshot: line when the hydration cache is disabled
• Improvement: Hub disables the hydration cache when the cross-path rename probe cannot write its probe file, rather than silently falling back to copy mode
• Improvement: Hub hydration per-file worker pool default sized to clamp(cpu/2, 8, 32) instead of clamp(cpu/8, 4, 12); a 64-core machine now gets 32 workers by default
• Improvement: Hub S3 hydration auto-derives the AsyncHttpClient shard count from --hub-hydration-async-max-concurrent-requests when asyncshards is unset (target ~32 in-flight requests per shard, capped at 32 shards). Explicit asyncshards still wins
• Improvement: Hub S3 hydration bounds per-request retry cycles to 5 minutes wall-clock so requests stuck behind a long admission queue do not surface as SigV4 RequestTimeTooSkewed 403s when the signature ages out
• Improvement: Hub S3 hydration aborts a dehydrate-upload or hydrate-download phase after 1200s of no progress instead of waiting for the watchdog. The phase fails cleanly through the existing rollback path; modules with a stuck request retry on the next operation
• Improvement: Hub S3 hydration tightens the curl mid-transfer stall guard to 1 B/s for 30s (libcurl default is 1 B/s for 90s). Tunable per call site via HttpClientSettings::LowSpeedLimit / LowSpeedTimeSec
• Improvement: Hub hydration phase summary log column "admission wait avg X/req max Y" renamed to "transport park avg X/req max Y" and now reports the time individual requests sat in AsyncHttpClient's pending-submissions queue. The old column tracked a long-removed external semaphore and was permanently zero
• Improvement: Hub hydration pack unpack writes go straight to the destination file instead of writing to a temp path and renaming, halving the per-entry metadata cost
• Improvement: Hub hydration pack blobs transit RAM end-to-end; the pack file is no longer staged under TempDir/packs/ on either side. Removes the temp-disk requirement proportional to pack size and the per-pack create/write/rename/delete metadata pass
• Improvement: Hub S3 hydration no longer holds a worker thread per in-flight request as an admission slot; requests beyond the configured concurrency cap park inside the AsyncHttpClient pending queue. Frees the per-file worker pool to do real work (unpack, hash, file write) at full capacity
• Improvement: OTel HTTP server spans are named with the low-cardinality {method} {http.route} form (e.g. GET /prj/{project}/oplog/{log}/batch) instead of {verb} {uri}; the concrete path stays on url.path. Router-less handlers (/dashboard/, /status/, the zen CLI trace viewer /trace/) set a bounded route template so their spans are named too
• Improvement: OTel url.path on the ASIO server backend is the full absolute decoded request path, matching the http.sys backend
• Improvement: zen builds download and zen builds upload keep a downloaded or staged block/chunk temp file open across its rename into place instead of closing and reopening it. Removes the brief window between close and reopen where a virus scanner or other process could lock or alter the file and fail the operation
• Bugfix: Hub no longer logs Failed to clean ... state.cbo: Access is denied after a full hydration-cache hit on Windows
• Bugfix: Hub hydration worker-side failures (e.g. one S3 GET timing out) no longer roll the module back to unprovisioned without spawning the child instance. The internal worker fast-fail flag is separated from the external abort signal, so provision-phase 2 only sees an abort when one was actually requested
• Bugfix: AsyncHttpClient shutdown no longer hangs at process exit on Windows when high-throughput curl socket teardown leaves the asio io_context with stranded outstanding_work_ (chriskohlhoff/asio#431, #869). Cleanup now pre-cancels every tracked socket's async_wait before curl_multi_cleanup and waits for the posted strand handler to complete before releasing the work guard
• Bugfix: Cache-populating operations no longer leave cache uploads running at exit. zen builds prime-cache waits for them to finish (unless aborted); zen builds download/upload and project-store oplog load cancel any still pending

• Feature: zenserver and the zen CLI export observability data - logs, traces, and metrics - to any OpenTelemetry-compatible backend via OTLP/HTTP (http/protobuf). A single endpoint configuration enables all three signals; per-signal environment variables can override routing or disable individual signals. See docs/OpenTelemetry.md for the operator's guide
  • --otlp-endpoint <uri> (Lua: server.otlpendpoint, env: OTEL_EXPORTER_OTLP_ENDPOINT, default empty = disabled). Pass none (case-insensitive) to explicitly disable and ignore the env var
  • --stats-interval-ms (Lua: server.statsintervalms, env fallback: OTEL_METRIC_EXPORT_INTERVAL, default 10000). Drives the /v1/metrics POST cadence
  • --trace-interval-ms (Lua: server.traceintervalms, default 1000; 0 disables). Optional fast tick for UE Trace counters only - decouples local trace cadence from OTLP export
  • --otel-batch-linger-ms (Lua: server.otelbatchlingerms, default 200). Worker-side wait window for batching same-path tasks into one POST; 0 disables deliberate linger while keeping opportunistic draining
  • --trace-sampler <name> / --trace-sampler-arg <value> (Lua: server.tracesampler / server.tracesamplerarg, env: OTEL_TRACES_SAMPLER / OTEL_TRACES_SAMPLER_ARG). Head-based ratio sampling per the OTel SDK spec (always_on, always_off, traceidratio, parentbased_*); default parentbased_traceidratio(1.0) preserves pre-feature behavior (every trace kept). Decisions are per-trace and deterministic across services; dropped traces still propagate traceparent with the sampled bit cleared so downstream parentbased_* samplers agree
  • Standard OTel environment variables honored: OTEL_EXPORTER_OTLP_{ENDPOINT,HEADERS,COMPRESSION,TIMEOUT,PROTOCOL} plus the per-signal _LOGS_ / _TRACES_ / _METRICS_ variants, OTEL_{LOGS,TRACES,METRICS}_EXPORTER=none, OTEL_SDK_DISABLED, OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES. OTEL_EXPORTER_OTLP_PROTOCOL other than http/protobuf disables export with a stderr warning
  • Per-signal endpoint/timeout overrides allocate a dedicated HTTP client so per-signal timeouts actually take effect; header and compression overrides without an endpoint/timeout override keep using the shared client
  • Default Resource attributes follow OTel semconv: service.name, service.namespace, service.version, service.instance.id, host.name, host.arch, host.id, os.type, os.description, process.id, process.executable.name, telemetry.sdk.{name,language,version}, plus zen-specific zen.os.runtime, zen.process.launch, zen.session.{id,parent_id}
  • W3C Trace Context (traceparent / tracestate) and Baggage (baggage) parsed on inbound HTTP across all server backends (http.sys, ASIO, plugin) and injected on outbound HTTP from sync and async curl clients; ambient log fields marked LogField::Propagate (ZEN_BAGGAGE_FIELD) ride the baggage header. Trace context auto-propagates into WorkerThreadPool tasks without caller wiring
  • The OTLP exporter is fire-and-forget: log/span/metric calls never block on the collector. Bounded internal queue (1000 records, drop-oldest), dedicated worker thread, gzip compression by default (OTEL_EXPORTER_OTLP_COMPRESSION), and a circuit breaker that opens after 5 consecutive POST failures and stays open for 30 s. Set ZEN_OTEL_DUMP_DIR=<dir> to write each batched payload to disk as .pb + best-effort .txt for collector-side debugging
  • zen CLI wraps each command invocation in an OTel root span named after the subcommand; exceptions are recorded with Error status before the mapped exit code returns. The CLI also exports a one-shot snapshot at exit so even short commands surface in the backend
• Feature: Unified Telemetry dashboard tab surfaces live OTLP exporter health (configured endpoint, per-signal enqueued/posted/failed/bytes counters, last-export/attempt timestamps), the registered metrics-provider inventory (always-on + env-gated optional providers with current state), and live host metrics. Existing ?page=metrics URLs continue to work; the tab is just relabeled
• Feature: Host network metrics on the dashboard - per-interface rx/tx bytes/packets/errors/dropped table plus an aggregate connection-count table by transport/state. Live over the existing /stats WebSocket; always available. OTLP export of system.network.* (NIC counters + socket aggregates) is opt-in via ZEN_OTEL_SYSTEM_NETWORK_METRICS due to cardinality
• Feature: OTel HTTP client metrics (http.client.request.duration histogram, http.client.active_requests UpDownCounter) keyed by http.request.method, http.response.status_code, server.address, url.scheme, network.protocol.version, and error.type. Wired through sync and async curl clients; active counter stays balanced across normal completion, retry, cancellation, and shutdown
• Feature: OTel HTTP server metrics (http.server.request.duration histogram, http.server.active_requests UpDownCounter) keyed by http.request.method and http.route (the dispatched service's BaseUri, kept low-cardinality). RAII-wrapped in all three server backends so normal and exception paths both count
• Feature: System/filesystem/service metrics providers - process memory and CPU time (process.cpu.time is double-valued ms to avoid integer truncation), system CPU%, logical CPU count, system memory used/free/limit, system and process uptime, data-root system.filesystem.usage / system.filesystem.limit with mountpoint+state attributes, per-state hub instance counts and aggregate child RSS, build store blob count and per-kind disk bytes, cache disk and memory usage
• Feature: Opt-in per-(namespace, bucket) cache counters zen.cache.bucket.{hits,misses,writes} gated by ZEN_OTEL_CACHE_BUCKET_METRICS (off by default due to cardinality)
• Feature: Added zen builds clone command to make a server side copy of a build in cloud storage.
• Feature: Added command-line option --restrict-host-headers (Lua: security.restricthostheaders) which causes zenserver to validate the HTTP Host: header against the machine's hostname, FQDN, and configured network adapter addresses (plus localhost / 127.0.0.1 / [::1]), rejecting requests with unknown or spoofed values. Matching is case-insensitive; the [addr] / [addr]:port IPv6 form is recognized
• Feature: JSON security configuration http.root now accepts a filters array of {type, config?} entries (e.g. {"type":"password","config":{...}}, {"type":"host-header"}); the previous single-filter object form is still accepted, and specifying both is an error. The same object also accepts restrict-content-types and allow-external-oidctoken-exe boolean overrides matching the existing CLI flags
• Feature: Memtrack support on Linux and macOS. zen trace analyze/zen trace serve accept --symbol-search-path "dir1;dir2;..." for cross-host symbolication; candidates are validated against the recorded .note.gnu.build-id (Linux) or Mach-O LC_UUID (macOS). Missing llvm-symbolizer degrades to address-only output; invoked with --no-debuginfod. Linux binaries link with -Wl,--build-id=sha1
• Feature: Opt-in CRT malloc interception via --mallocoverride=yes (default off) routes malloc/calloc/realloc/free/_aligned_*/_strdup/posix_memalign through GMalloc so ASIO, libfmt, and CRT-internal allocations no longer bypass the selected allocator and memtrack. Pulls Microsoft Detours v4.0.1 on Windows; ASAN forces it off
• Improvement: zenserver dashboard Network tab hides adapters whose rx/tx byte/packet/error/dropped counters are all zero by default; a "show N idle" / "hide idle" toggle reveals or re-hides them
• Improvement: Metric and attribute names align with OTel semantic conventions across zentelemetry, zenserver, and zenstore. Byte-valued gauges (zen.{cache,buildstore,hub}.*.bytes[.max]) renamed to .usage / .limit with the unit declared via instrument metadata rather than baked into the name. Event counters use the {events} unit instead of UCUM 1. OTLP export carries unit and description per-instrument
• Improvement: zencore adds a FileAccessPolicy (Normal/Sequential/Random) trailing parameter to BasicFile/TemporaryFile open. Maps to Windows FILE_FLAG_SEQUENTIAL_SCAN/FILE_FLAG_RANDOM_ACCESS at CreateFile, Linux posix_fadvise(POSIX_FADV_SEQUENTIAL|RANDOM), and macOS fcntl(F_RDAHEAD, 0) for Random (Sequential is already the macOS default). IoBufferBuilder::MakeFromFile/MakeFromTemporaryFile always open and madvise the mmap region with Random - the canonical scattered-access path for IoBuffer. Build-download hashing, scavenge chunk reads, cachedisklayer sidecar/index streams, oplog blob compaction, hub pack-file writes, and other call sites with known access patterns now pass the matching policy
• Improvement: BasicFileWriter issues posix_fadvise(DONTNEED) on the just-written byte range after each underlying write on Linux via the new HintFileRangeNotNeeded helper in filesystem.h. Caps the accumulated dirty-page footprint on sustained large writes (e.g. zen builds download chunk-unpack) and avoids triggering the Linux write-back throttle. No-op on macOS (no clean equivalent; F_NOCACHE has the wrong semantics) and Windows
• Improvement: zencore PrepareFileForScatteredWrite preallocates the destination extents on Linux via posix_fallocate64 (was a no-op outside Windows). Avoids per-write block-allocation and filesystem journal traffic on workloads with many scattered writes per file
• Improvement: per-request-class retry and timeout policies for Jupiter HTTP calls. Fixes zen builds/zen oplog aborting with ExitCode=80 on transient CURLE_PARTIAL_FILE or CURLE_OPERATION_TIMEDOUT mid-operation
  • HttpClientSettings::Retry groups count, base, cap, exponential; defaults preserve the existing 100 * (Attempt + 1) linear schedule for all current callers
  • Jupiter discovery (DiscoverJupiterEndpoints) retries 6 times with exponential backoff (500 ms base, 10 s cap), overriding caller-supplied settings
  • Overridden-host data plane retries 6 times with exponential backoff (500 ms base, 30 s cap), overriding caller-supplied settings
  • TestJupiterEndpoint bumped from 2 to 4 attempts with linear 0.5/1/1.5 s spacing; per-attempt Timeout=3 s retained so a slow host is still rejected, retries absorb a single packet drop on a VPN hop
  • TestZenCacheEndpoint tightened to ConnectTimeout=500 ms, Timeout=1.5 s, 3 attempts (linear 0.25/0.5 s); slow cache is rejected rather than retried into
  • zen builds/zen oplog/BuildsRemoteStore/JupiterRemoteProjectStore/server-side HttpProjectStore data-plane clients retry 6 times with exponential backoff and no longer set a 30-minute wall-clock Timeout; the existing 90 s no-progress stall guard catches stuck transfers regardless of payload size
• Improvement: Cache record Put that would otherwise be rejected with Conflict now inspects the existing record; if any referenced attachment is missing from the CAS the record is unservable and the put is allowed to replace it, otherwise the conflict propagates as before. Applied to direct kCbObject and kCbPackage PUT to /z$/<ns>/<bucket>/<key> and to the /$rpc PutCacheRecords per-record handler; triggers a ZEN_DEBUG PUTCACHERECORD - OVERWRITE log on the cache channel with bucket, key, and missing-attachment count
• Improvement: zen oplog-export pre-sizes the rewritten oplog buffer up front from the in-memory oplog metadata, avoiding repeated reallocations and large transient memory peaks during the per-op serialization loop on projects with millions of ops
• Bugfix: zen builds upload per-part discovery log now reports the correct block count and find-blocks time instead of 0 blocks and 0ms; summary find-block chunk and byte counts no longer multiply with the number of parts in a build
• Bugfix: zenserver no longer emits spurious SetFileInformationByHandle(DeleteOnClose) failed warnings during cas chunk replacement; the replace path now renames the existing chunk file aside and removes it via the standard cross-platform delete helper
• Bugfix: zen oplog-import no longer fails with dechunk errors when GC runs concurrently with an in-progress import; chunks downloaded before the new oplog is persisted are now retained across the import scope and survive a concurrent GC pass
• Bugfix: zen oplog-export no longer aborts with an improbably large request allocator assert when serializing oplogs whose uncompressed op section approaches 1.5 GiB; the compact-binary writer's backing buffer now grows in capped increments past 256 MiB and has no hard 2 GiB size ceiling
• Bugfix: zen oplog-import no longer aborts with a chunk-store assertion when a virus scanner or filesystem filter briefly holds the dechunked temp file open after close; the dechunk worker hands its open handle directly to the chunk store instead of reopening the path
• Bugfix: zen trace serve /trace (no trailing slash) now redirects to /trace/ so the viewer's relative asset and API fetches resolve

• Bugfix: zenserver no longer stack-overflows when sentry-native logs a LoadLibraryExW failure on a Windows hotpatch DLL during event capture; a thread-local guard breaks the cross-logger recursion loop back into SentrySink

• Improvement: zen builds download emits a No content in build matches input filters warning when include/exclude filters or the download spec leave no parts to download
• Bugfix: Windows: zen no longer hangs at shutdown when stdout is piped to a parent that drains only after the child exits (e.g. UE's GetZenVersion); FlushFileBuffers is skipped on pipe and console handles
• Bugfix: zen builds download --download-spec now skips build parts not listed in the spec; previously the first build part was always processed regardless of the spec
• Bugfix: zen builds download --download-spec validates the spec path exists up front and fails with a clear error message
• Bugfix: GET /prj/<project>/oplog/<oplog>/chunk/<cid> no longer asserts when the stored chunk decompresses to a zero-byte payload; the server now returns 204 No Content

• Feature: Hub adds an optional stable Allocated instance state that reserves a slot and port without spawning a process or creating data; counts toward the configured instance limit. Use only when a stable port must be known before deciding to provision; existing callers can continue to go straight from Unprovisioned to Provisioning. New endpoint POST /hub/modules/{moduleId}/allocate claims the slot; a subsequent Provision against the same module transitions Allocated to Provisioning and reuses the reserved port; Deprovision and Obliterate from Allocated release the slot directly without entering Deprovisioning or Obliterating; the watchdog reaps idle Allocated slots after a configurable timeout
  • --hub-watchdog-allocated-inactivity-timeout-seconds (Lua: hub.watchdog.allocatedinactivitytimeoutseconds, default 10 min (600 s))
• Feature: Hub watchdog activity probe runs at a fixed configurable cadence per Provisioned instance plus edge probes when the instance first crosses the inactivity margin window and immediately before a deadline-driven deprovision; first-pass probe also fires on the first watchdog cycle after entering Provisioned
  • --hub-watchdog-activity-probe-interval-seconds (Lua: hub.watchdog.activityprobeintervalseconds, default 5 min (300 s))
• Feature: Hub adds an optional persistent cache of hydrated module state. Subsequent Hydrate skips backend downloads on a full manifest match, or transfers matching files by content hash on partial match. Obliterate clears the cache alongside backend and local state.
  • Cache path may share a volume with the server data dir (rename mode, O(1) atomic transfers) or live on a different volume (copy mode, byte copy); the hub resolves the mode at startup via a cross-path rename probe and logs the result
  • Cache path must be a distinct directory from the data dir (hub refuses to start if they resolve to the same directory) and must not be shared between hub instances (no inter-process locking on the cache layout)
  • Cache transfers are single-shot best-effort with no retry
  • Cache GC: oldest orphan modules are evicted until used disk drops to 87.5% of the cap. Torn cache entries (a partial snapshot with no state.cbo) and non-conforming module directory names are wiped on the next GC pass
  • --hub-hydration-cache-path <path> (Lua: hub.hydration.cachepath, default empty = disabled). Must be an absolute path
  • --hub-hydration-cache-disk-limit-bytes <bytes> (Lua: hub.hydration.cachedisklimitbytes, default 0). Absolute cap on used disk space of the cache volume
  • --hub-hydration-cache-disk-limit-percent <0..100> (Lua: hub.hydration.cachedisklimitpercent, default 90). Cap as percent of cache-volume total capacity. When both bytes and percent are set the tighter cap wins, matching the existing --hub-provision-disk-limit-* semantics. Both zero disables GC
• Feature: Hub adds POST /hub/modules/{moduleId}/abort and a matching zen hub abort <moduleid> CLI to request cancellation of an in-flight async operation (Provision, Deprovision, Obliterate, Hibernate, Wake, Recovery) on a module. Returns 404 Not Found when the module is unknown, 202 Accepted when the module is in a transitioning state, or 409 Conflict when the module is in a stable state (body names the current state). The running operation observes the request at its next inter-phase gate, including mid-stream during S3 downloads, and exits through its failure path
• Feature: Hub watchdog auto-aborts a module that has been sitting in a transitioning state (Provisioning, Hibernating, Waking, Deprovisioning, Recovering, Obliterating) for longer than the configured timeout; emits a WARN each watchdog cycle while the slot remains stuck
  • --hub-watchdog-stuck-transient-timeout-seconds (Lua: hub.watchdog.stucktransienttimeoutseconds, default 45 min (2700 s); 0 disables)
• Improvement: AsyncHttpClient::AsyncStream takes an AsyncHttpStreamHandlers struct (OnAttemptStart, OnData) so callers can reset destination state on retry; only 2xx body bytes reach OnData and non-2xx bodies (1xx/3xx/4xx/5xx) surface in Response.ResponsePayload (capped at 64 KiB with a [truncated] marker on overflow)
• Improvement: /hub/stats response includes instanceCountByState with per-state counts (sum equals the configured instance limit)
• Improvement: Hub dashboard Instances tile shows a fixed used (count and percent) / free split with a per-state breakdown row underneath; tile no longer resizes as instances move between states
• Improvement: Hub dashboard per-row action button drives Provision when the row is Allocated and Wake when Hibernated, with a matching icon and tooltip; toolbar adds an Allocate button (placed before Provision) that opens a module-id modal
• Improvement: Hub watchdog log lines now carry a [watchdog] prefix so they can be grepped distinct from request-driven hub events
• Improvement: Hub watchdog logs the child-process exit code (decimal and hex) when an instance is detected dead and transitioned to Crashed
• Improvement: /hub/status per-module response includes last_activity_time; hub dashboard module foldout shows "Last activity", "Idle for", "Deprovision in", and "Deprovision at" alongside the state-since/state-for rows
• Improvement: Hub dashboard Status cell shows <dot> <state> <duration> (deprov in <X>) for stateful instances; Module ID, Status, and Port column headers are sortable; inline durations render to whole seconds
• Improvement: /hub/status response includes a top-level watchdog object reporting the configured provisioned_inactivity_timeout_ms, hibernated_inactivity_timeout_ms, allocated_inactivity_timeout_ms, inactivity_check_margin_ms, and activity_probe_interval_ms
• Improvement: Hub hydration completion log adds a Cache: row reporting hit or miss, files reused, and bytes reused (with a (copy) qualifier when the cache lives on a different volume from the server data dir); row is suppressed when no cache path is configured. Hub dehydration completion log adds a matching Cache snapshot: row reporting whether the post-dehydrate cache snapshot was written, skipped, or failed
• Improvement: zen HTTP clients (sync and async) abort a stalled curl transfer when no body bytes flow for 90 seconds; catches connections that go silent mid-stream after leaving the idle pool, beyond what TCP keepalive can detect
• Improvement: AsyncHttpClient exposes a stall-diagnostics snapshot of in-flight transfers, retrying transfers, cached sockets, handle pool size, and oldest in-flight transfer age
• Improvement: Hub hydration phase Wait() emits a WARN when the latch's pending count has not changed for the phase's threshold (120s for local-IO phases, 300s for backend-IO phases); slow-but-progressing operations do not trip the log
• Improvement: Hub hydration cleanup-after-failure WARN reports the first unremovable entry, the count of failed entries, and the total entries seen, instead of one WARN per file
• Improvement: Hub hydrate completion summary Clean row shows (FAILED) when entries remained after cleanup, mirroring the existing dehydrate behavior
• Improvement: Hub hydration stall WARN includes STALE in the backend-diagnostics field when the async HTTP client's io strand fails to service a stats snapshot within a bounded timeout, so a wedged strand surfaces in the stall log instead of blocking it
• Improvement: Hub S3 async transfer failures caused by allocation pressure or unexpected callback throws now surface the actual cause via the operation's error rather than masking it as "destroyed without Complete/Fail"; multipart per-part / per-range completion accounting is guaranteed even when the callback body throws
• Improvement: Hub S3 hydration async path supports sharding via the asyncshards value in the S3 target settings (default 1, capped at 32); each shard is an independent AsyncHttpClient (own io thread + curl_multi) so completion work spreads across cores. --hub-hydration-async-max-concurrent-requests is split evenly across shards (remainder added to the first shards) so aggregate in-flight cap and ephemeral-port usage are unchanged at asyncshards=1
• Improvement: Hub instance pool defaults retuned for the documented 1000-instance limit. --hub-instance-provision-threads default changed from clamp(cpu/8, 4, 16) to clamp(min(cpu/2, cpu/4 + 16), 8, 64); --hub-instance-spawn-threads default changed from clamp(cpu/8, 4, 16) to clamp(min(cpu/2, cpu/8 + 16), 4, 32). Prevents Phase 2 deprovision from queuing minutes behind in-flight hydrations on large hubs
• Improvement: Hub S3 multipart upload streams each part from disk via the AsyncHttpClient source callback instead of materialising the part in heap, reducing hub peak RSS during dehydration of large modules
• Improvement: Hub hydration unpack phase reads each entry directly from the open pack file instead of holding the full pack content in heap, reducing hub peak RSS during hydration of modules with packed entries
• Improvement: Hub S3 async upload no longer closes source files on the curl io thread; the BasicFile close runs on a worker pool so the io thread is not stalled by file close syscalls when a transfer completes
• Improvement: AsyncHttpClient surfaces the underlying exception text when a request fails during setup (curl handle alloc, header build, access-token provider throw); previously these surfaced as a fixed "setup threw before dispatch" message
• Bugfix: Hub S3 hydration no longer corrupts the destination file when an upstream attempt returns 5xx with an error body or drops mid-stream; the streaming GET filters non-2xx body bytes from caller delivery and resets per-attempt accumulators on retry
• Bugfix: Hub S3 dehydration LIST no longer fails with 403 SignatureDoesNotMatch on continuation pages once a module's object count exceeds one page (~1000 keys)
• Bugfix: Hub no longer leaves the local server-state directory on disk when a module's dehydration fails during deprovision; the directory is wiped after the warning is logged, since the backend retains the prior state
• Bugfix: zen HTTP clients (sync and async) now enable OS-level TCP keepalive and bound idle reuse below 60s so cloud load balancer silent idle timeouts (AWS NLB/ALB default) no longer cause the next reuse to burn a socket and accumulate Windows TIME_WAIT entries; fixes ephemeral port exhaustion observed under bulk hub hydration to S3
• Bugfix: zen <cmd> --help now lists positional arguments in the option body, not only in the usage line
• Bugfix: zen HTTP clients (sync and async) now retry on transient HTTP/2 and HTTP/3 transport failures (CURLE_HTTP2, CURLE_HTTP2_STREAM, CURLE_HTTP3) and on CURLE_GOT_NOTHING (empty reply, typically a load-balancer-cycled connection); previously a server-sent RST_STREAM (e.g. HTTP/2 stream reset by server (error 0x1 PROTOCOL_ERROR)) surfaced as a hard failure on the first attempt
• Bugfix: Hub Hydrate / Dehydrate / Obliterate cleanup after a worker failure no longer leaves files behind in the server-state and temp directories
• Bugfix: Hub child storage server instance stdout no longer interleaves into the hub log; child processes are launched with --noconsole
• Bugfix: Hub deprovision GC poll tolerates a single transient /admin/gc HTTP failure instead of abandoning the wait; the captured error is surfaced in the did not complete warning
• Bugfix: Hub /admin/gc no longer returns 500 when a project storage file is rotated between an existence check and the size query; storage-size accounting treats a vanished file as zero
• Bugfix: Linux build of zenserver against the UE clang toolchain via ~/.ue-toolchain auto-detect now links libc++/libc++abi statically; previously any target with exception handling failed to link
• Bugfix: zen directory-clean (CleanDirectory) with excluded subdirectories no longer reports the surviving parent directories as failed removals; suppresses spurious "Clean failed to remove files" and "Some files in ... could not be removed" warnings on cooked-build wipe paths
• Bugfix: Counter and region trace event timestamps on Linux now use the same clock as the trace's advertised CycleFrequency; previously counter and region cycles were emitted at a 10x scale mismatch so analyzers placed every counter sample and region marker at t=0 in the timeline

• Feature: AsyncHttpClient adds cancellable request tokens, streaming GET to a file (AsyncDownload), zero-copy chunk-callback GET (AsyncStream), pull-mode body source for streaming AsyncPut, retry layer mirroring the synchronous client, and a submit-side in-flight cap (HttpClientSettings::MaxConcurrentRequests) so hub-scale fanout against a single host cannot stall queued handles into curl's connect-timeout window
• Feature: Hub hydration can route S3 transfers through a non-blocking AsyncHttpClient (curl_multi + asio) backed by a single io thread; hydrate and dehydrate now pipeline requests instead of blocking worker threads
  • --hub-hydration-async-enabled (Lua: hub.hydration.async.enabled, default true)
  • --hub-hydration-async-max-concurrent-requests (Lua: hub.hydration.async.maxconcurrentrequests, default clamp(cpu*4, 128, 512))
• Feature: Hub provision/deprovision/obliterate now run as two phases on separate worker pools so per-module hydration cannot starve child-process spawn/despawn (and vice versa)
  • New --hub-instance-spawn-threads (Lua: hub.instance.spawnthreads, default clamp(cpu/8, 4, 16)) drives child-process spawn/despawn
  • --hub-instance-provision-threads (Lua: hub.instance.provisionthreads) now drives per-module hydrate/dehydrate scheduling only; default changed from max(cpu/4, 2) to clamp(cpu/8, 4, 12)
  • --hub-hydration-threads (Lua: hub.hydration.threads) now controls per-file workers inside a single hydrate/dehydrate; default changed from max(cpu/4, 2) to clamp(cpu/8, 4, 12)
• Feature: Hub::Configuration C++ struct fields renamed (OptionalProvisionWorkerPool/OptionalHydrationWorkerPool -> OptionalProvisionPool/OptionalSpawnPool/OptionalHydrationPool). Embedders constructing Hub directly must update field names; provision and spawn pools must both be set or both null (asserted at construction).
• Feature: Hub hydration excludes list is now a top-level option; the legacy HydrationOptions excludes JSON field is no longer read. Built-in defaults are unchanged; an explicit empty value opts out of defaults.
  • --hub-hydration-excludes <pattern[,pattern...]> (Lua: hub.hydration.excludes)
• Feature: zen builds and zen oplog import/zen oplog download accept --zen-cache-host-env to discover the zen builds cache host from an environment variable when --zen-cache-host is not supplied. Default name is UE-ZenSharedBuildCacheHost on Windows and UE_ZenSharedBuildCacheHost on Linux/macOS; pass an empty value to disable the env-var fallback. An explicit --zen-cache-host always wins.
• Improvement: Hub pools HTTP connections to managed instances so provision/deprovision churn no longer exhausts Windows ephemeral ports
• Improvement: zen consolidates the project-* commands into a single zen project <sub> command tree (project create, project drop, project info, project op-details, project stats). The legacy project-create/project-drop/etc. names remain as hidden deprecated aliases that forward to the new dispatcher, so existing scripts keep working.
• Improvement: zen consolidates the oplog-* commands into a single zen oplog <sub> command tree (oplog create, oplog export, oplog import, oplog snapshot, oplog mirror, oplog validate, oplog download). The legacy oplog-create/oplog-export/etc. names remain as hidden deprecated aliases that forward to the new dispatcher, so existing scripts keep working.
• Improvement: zencore C++ APIs (filesystem.h, basicfile.h, iobuffer.h) now use the opaque NativeFileHandle type for native file handles instead of bare void*. Construct via NativeFileHandle::FromHandle(HANDLE) on Windows or NativeFileHandle::FromFd(int) on Linux/macOS; default-constructed values represent an invalid handle (INVALID_HANDLE_VALUE / fd -1)
• Improvement: zen tests now use SeaweedFS as the embedded S3 emulator (replacing MinIO) for more stable macOS process lifecycle in CI
• Improvement: zen test runner shares a single SeaweedFS S3 mock across all tests in an xmake test invocation, cutting server-suite cold-start; tests can also point at an external mock via ZEN_TEST_S3_ENDPOINT
• Improvement: S3 client streams SHA-256 over file-backed request bodies in PutObject and UploadPart instead of mmaping the full file
• Improvement: Hub retries module provision once after wiping the on-disk state dir when zenserver fails to start, recovering from corrupt state
• Improvement: libcurl on Windows now links against OpenSSL (HTTP/2 + Schannel TLS 1.3 corrupts post-handshake framing) and pulls trust roots from the Windows certificate store via CURLSSLOPT_NATIVE_CA; zen's server-side TLS / crypto paths still use BCrypt (ZEN_USE_OPENSSL=0)
• Bugfix: S3Client signing-key cache no longer returns stale signatures after IMDS-rotated credentials change AccessKeyId; cache is now keyed on (DateStamp, AccessKeyId)
• Bugfix: zen builds download no longer injects the default build part when --download-spec is given; the spec drives part selection
• Bugfix: Hub Consul deregister no longer fails with "Unknown service ID" when the module ID contains /; the service ID is percent-encoded in the deregister URL path
• Bugfix: Hide oidc token auth console window during authentication for oplog import/export

• Feature: Hub hydration packs small files into raw CAS pack blobs to reduce request count for modules dominated by tiny metadata files
  • --hub-hydration-enable-pack (Lua: hub.hydration.enablepack, default true)
  • --hub-hydration-pack-threshold-bytes (Lua: hub.hydration.packthresholdbytes, default 256 KiB)
  • --hub-hydration-max-pack-bytes (Lua: hub.hydration.maxpackbytes, default 4 MiB)
• Feature: Hub hydration and dehydration can be disabled per direction
  • --hub-enable-hydration (Lua: hub.enablehydration, default true)
  • --hub-enable-dehydration (Lua: hub.enabledehydration, default true)
• Feature: Hub hydration accepts a configurable file exclude list via HydrationOptions excludes (array of wildcards). Built-in defaults skip transient runtime files (.lock, .sentry-native/*, state_marker, *.bak, gc/reserve.gc, auth/*) so they no longer participate in dehydrate scans. Override semantics: a present field replaces the default outright; explicit [] opts out of all defaults.
• Improvement: Hub hydration completion logs now report per-request average and max latency, peak in-flight workers, queue wait, and hash-cache hit percentage; loose and pack-blob transfers are reported separately
• Improvement: Hub hydration pre-creates unique parent directories before scheduling parallel writes
• Improvement: S3 hydration retries transient HTTP failures (timeouts, 429 throttling, 5xx server errors, connection errors) up to 3 times via the HTTP client retry layer
• Improvement: S3 hydration multipart chunk size is persisted in state.cbo per module so hydrate replays the partitioning used at dehydrate; default raised to 64 MiB (was 32 MiB)
• Improvement: Hub hydration Obliterate retries backend delete once before falling back to local cleanup
• Improvement: zen builds --exclude-folders and --exclude-extensions values now match paths case-insensitively and tolerate surrounding whitespace between separators
• Bugfix: zen builds ls restored (for backwards compatibility) to past behavior of listing all parts if none are specified, zen builds download, and zen builds prime-cache retain the new standard behavior of only operating on the default part if none are specified.
• Bugfix: Trace Misc.RegionBeginWithId event name and category fields are now declared as AnsiString so UE Insights analyzers can parse Region names from zen-emitted .utrace files

• Bugfix: zen builds download no longer crashes intermittently when scavenging sequences or copying local chunks

• Improvement: zen builds zen-folder handling is now consistent per subcommand
  • list-namespaces, list, list-blocks, ls: no local scratch folder is created; responses stay in memory
  • upload, fetch-blob, prime-cache, validate-part: default to <cwd>/.zen (no change)
  • download: default to <local-path>/.zen (no change)
• Improvement: Dashboard banner displays the zenserver version next to the wordmark
• Improvement: Zen log messages are forwarded to trace as typed ZenLog.* events that preserve structured fmt args end-to-end, so the zen trace analyzer and other downstream consumers can re-render messages with full formatter support (nested widths, chrono specs, etc.). --trace=log now routes to the zen log channel (the upstream UE log channel is unused since zen no longer emits Logging.* events)
• Improvement: Hub Consul client HTTP timeout defaults raised to 1s connect / 2s total so transient latency to a slow Consul agent no longer fails registration calls
• Improvement: Hub hydration and dehydration completion logs now include per-phase wall time, bytes transferred, bits/s throughput, number of unique worker threads used, and the storage source/target URI
• Improvement: Hub storage server instance lifecycle logs now report elapsed time for spawn and shutdown
• Improvement: Hub deprovisioning now logs GC completion status and elapsed time; a GC that does not complete within the 5s deadline is logged as a warning and shutdown proceeds anyway
• Bugfix: zen builds ls no longer fails against cloud build storage (--host/--url) when --storage-path is not supplied
• Bugfix: NamedEvent construction on Linux/macOS no longer races against a concurrent destructor's unlink() of the backing file; the IPC key is now derived via fstat() on the open fd instead of ftok() re-stat'ing the path
• Bugfix: Hub provision requests now return 202 Accepted when the module is Recovering or Waking instead of rejecting

• Feature: zen trace subcommand suite for Unreal Insights .utrace files (and emitted by zenserver/zen via --trace options)
  • zen trace analyze - offline analysis with CPU scope, memory allocation, allocation churn, and callstack reports; symbolication via PDB, DbgHelp, and Microsoft Symbol Server backends; optional HTML output
  • zen trace serve - embedded HTTP server hosting a browser-based trace viewer (timeline with threads/scopes/regions/bookmarks/log events, memory analysis, stats) for interactive analysis
  • zen trace inspect - print event schemas from a trace file
  • zen trace trim - trim a trace file to a time range
• Feature: zen down new targeting modes
  • --pid <pid> shuts down a specific zen server process by process id
  • --executable <name-or-path> shuts down all zen server processes whose executable matches; a bare filename matches by name only, a path compares in full
• Feature: Per-user invocation history for zen and zenserver; each startup appends a record to a JSONL file capped at the most recent 100 entries. Location: %LOCALAPPDATA%\Epic\Zen\History\invocations.jsonl on Windows, ~/.zen/History/invocations.jsonl on POSIX
  • zen history opens an interactive picker; selecting a zen row re-runs it inline and forwards the exit code, selecting a zenserver row spawns it detached
  • zen history --list (-l) prints the table to stdout instead of showing the picker
  • zen history --filter zen|zenserver restricts the listing to one executable
  • zen history --print prints the reconstructed command line of the selected row instead of launching it
  • --enable-execution-history global option on both binaries (default true) to opt out per invocation
  • The history file is attached to Sentry crash reports (alongside the existing zenserver log)
• Improvement: zen auth machine key is now wrapped with OS-protected storage so the on-disk blob cannot be unwrapped off-machine or from a backup without also stealing the user's OS master key: Windows DPAPI at per-user scope, macOS Keychain Services (GenericPassword, kSecAttrAccessibleAfterFirstUnlockThisDeviceOnly), and Linux libsecret (opt-in via --zenlibsecret=yes, default off because headless servers typically have no Secret Service daemon). On platforms/configurations without wrapping available, the file falls back to raw bytes with 0600 permissions.
• Improvement: tourist trace-analysis library and raw_pdb vendored under thirdparty/; tourist made cross-platform (Windows/Linux/macOS)
• Improvement: Trace viewer HTML/JS frontend bundled via zen frontend zip build rule; ZipFs moved from zenserver into zenhttp for reuse
• Improvement: Replaced OperationLogOutput with ProgressBase in zenutil; logging and progress reporting are now separate concerns. Operation classes receive a LoggerRef for logging and a ProgressBase& for progress bars
• Improvement: New ZEN_SCOPED_LOG(Expr) macro routes ZEN_INFO/ZEN_WARN/ZEN_DEBUG in the enclosing block through the given logger expression instead of the default
• Improvement: BuildContainer, SaveOplog, and LoadOplogContext now take a caller-provided LoggerRef so diagnostic messages route through the caller's logger
• Improvement: zen CLI progress bar pins to the bottom of the terminal so log output can scroll above it; --verbose no longer forces plain progress output
• Improvement: zen down --data-dir probes lock-file liveness instead of presence; stale lock files are cleaned up and the command reports "nothing to do" instead of failing
• Improvement: zen attach --data-dir rejects stale lock files with a clearer error message
• Improvement: zen up, zen down, and zen attach normalize their path arguments (--base-dir, --data-dir, --executable), so relative paths and long paths work consistently
• Improvement: Process lookup by executable is now case-insensitive on Windows
• Improvement: Incremental hydration refreshes the modification time of existing S3 CAS entries during dehydrate so lifecycle-expiration policies do not evict objects still referenced by the module
• Improvement: Sensitive values in command line arguments (credentials, tokens, passwords, connection strings) are scrubbed before being written to logs or sent to Sentry
• Improvement: IMDS credential refresh log message no longer exposes the first 8 characters of the AWS AccessKeyId
• Improvement: zen builds download, zen builds ls, and zen builds prime-cache now default to the part named default when no --build-part-id/--build-part-name is given (previously all parts were selected). Pass --build-part-name=* to select all parts.
• Improvement: Hub Consul service registration and deregistration are now dispatched on a dedicated background thread so instance state transitions no longer stall when the Consul agent is slow or unreachable
• Improvement: ZenCacheStore namespace lookups use transparent string hashing so hot-path Get/Put calls no longer allocate a std::string per request
• Improvement: File copy, scan, clone, and move operations now report the underlying OS error in failure messages
• Improvement: Hub shares a single S3 client and IMDS credential provider across all modules, reducing IMDS load and surviving transient IMDS blips during bulk provisioning
• Improvement: Hub validates hydration config at startup; bad --hub-hydration-target-spec or --hub-hydration-target-config now fails zen hub at boot instead of per-module at first hydrate
• Improvement: S3 hydration multipart chunk size configurable via settings.chunksize (default 32 MiB)
• Improvement: S3 client extracts <Error><Code> and <Message> from XML error bodies (previously logged as <unhandled content format>)
• Improvement: S3 client fails fast with a "no credentials available" error when AWS credentials are missing, instead of sending an unsigned request that S3 rejects with a generic 400
• Improvement: IMDS credential provider retries transient connection failures (up to 3 attempts with backoff)
• Improvement: HTTP clients with RetryCount > 0 also retry on CURLE_COULDNT_CONNECT
• Improvement: zen cache subcommand supersedes old flat style cache commands; old names remain as hidden deprecated aliases
  • zen cache info replaces zen cache-info
  • zen cache stats replaces zen cache-stats
  • zen cache details replaces zen cache-details
  • zen cache gen replaces zen cache-gen
  • zen cache get replaces zen cache-get
  • zen cache drop replaces zen drop
  • zen cache record <path> replaces zen rpc-record-start
  • zen cache record stop replaces zen rpc-record-stop
  • zen cache replay replaces zen rpc-record-replay
• Bugfix: builds download partial-block fetch decisions now account for build storage host latency
• Bugfix: Transfer rate displays in builds commands now smooth correctly
• Bugfix: Structured cache PUT errors with a detail body no longer write the HTTP response twice
• Bugfix: Structured cache endpoints now return 405 Method Not Allowed for unsupported verbs instead of hanging the client
• Bugfix: exec$/replay-recording now clamps thread_count to at most max(HW concurrency, 16) so a bogus value cannot spawn an unbounded thread pool
• Bugfix: zen builds download --download-spec-path <file> now resolves the manifest path relative to --local-path when relative; previously the computed absolute path was discarded and the file was looked up relative to the current working directory
• Bugfix: zen project oplog-export --builds-metadata-path <file> now fails with a descriptive error when the metadata JSON is malformed; previously a JSON parse failure was silently ignored and a truncated compact-binary payload was shipped to the server
• Bugfix: zen project oplog-mirror now rejects filenames from the remote oplog that could escape the mirror root (absolute paths, drive letters, UNC / device path prefixes, or .. components); previously a malicious or compromised remote oplog could cause writes to arbitrary filesystem locations
• Bugfix: zen project oplog-mirror now handles Ctrl-C cleanly by signalling the worker pool to abort; previously SIGINT killed the process mid-transfer
• Bugfix: Malformed --build-id / --build-part-id values are now rejected with an OptionParseException; previously the Oid parser aborted the process on non-hex or wrong-length input
• Bugfix: zen cache get now requires --output-path unless --as-text is set; previously an empty path silently wrote a file named after the value key / attachment hash into the process's current working directory
• Bugfix: zen project oplog-export and oplog-import "[builds] ..." description messages now print the actual builds URL; previously they showed the (often empty) cloud URL due to a copy-paste error
• Bugfix: zen auth storage now uses a machine-specific random AES key/IV persisted to <system-root>/auth/machinekey.dat when --encryption-aes-key/--encryption-aes-iv are not supplied; previously a hardcoded constant visible in the source was used, letting anyone with source access decrypt persisted OIDC refresh tokens. Supplying only one of the two flags is now rejected rather than silently substituting the hardcoded default for the missing half. Existing auth state files will fail to decrypt once after upgrade and re-auth automatically.
• Bugfix: Fix Windows service shutdown signalling so it works as intended again

• Improvement: Session service and endpoint moved from storage server to base server class; now available in hub, compute, and proxy modes
• Improvement: HTTP client Content-Type in additional headers now overrides the payload content type
• Improvement: Dashboard copy-to-clipboard buttons on hub module IDs and ports, compute worker/action IDs and queue tokens, orchestrator client IDs, cache namespace names and directories, project names and directories, and workspace IDs
• Improvement: --buildstore-disksizelimit-percent - Max percentage of total drive capacity (of --data-dir drive) for build storage. When combined with --buildstore-disksizelimit, the lower value wins.
• Bugfix: OAuth client credentials token request now sends correct application/x-www-form-urlencoded content type
• Bugfix: builds download with cache upload enabled no longer holds downloaded blobs in memory when boost-worker-memory is active; blobs are written to disk before upload
• Bugfix: Removed obsolete --cache-prime-only flag from builds download
• Bugfix: Hub provision requests for already-provisioned instances now reset the inactivity timer
• Bugfix: Fix 404 returned from ProjectStore url requests: details$, details$/{project}, details$/{project}/{log}, details$/{project}/{log}/{chunk}

• Feature: Hub bulk deprovision endpoint (POST /hub/deprovision) tears down all provisioned and hibernated modules in a single request
• Feature: Hub instance forwarding options for memory allocator and tracing
  • --hub-instance-malloc selects the memory allocator for child instances
  • --hub-instance-trace sets trace channels for child instances
  • --hub-instance-tracehost sets the trace streaming host for child instances
  • --hub-instance-tracefile sets the trace output file for child instances; supports {moduleid} and {port} placeholders
• Improvement: Replaced archived http_parser library with llhttp for HTTP request parsing
• Improvement: HTTP range responses (RFC 7233) are now fully compliant across the object store and build store
  • 206 Partial Content responses now include a Content-Range header; previously absent for single-range requests, which broke HttpClient::GetRanges()
  • 416 Range Not Satisfiable responses now include Content-Range: bytes */N as required by RFC 7233
  • Out-of-bounds range requests return 416 Range Not Satisfiable (was 400 Bad Request)
  • Single-byte ranges (bytes=N-N) are now correctly accepted (were previously rejected)
  • Range byte positions widened from 32-bit to 64-bit; RFC 7233 imposes no size limit on byte range values
  • Build store binary GET requests with a Range header now return 206 Partial Content with Content-Range (previously returned 200 OK without it)
• Improvement: Updated rpmalloc to develop branch commit 262c698d7019 (2026-04-10), which fixes memory ordering on weak architectures and avoids assert on mmap failure with callback
• Improvement: Increased rpmalloc page decommit thresholds to reduce commit/decommit churn under high allocation turnover
• Improvement: Disk full error message for builds download now shows human-readable sizes and available free space
• Improvement: Dashboard paginated lists now include a search input that jumps to the page containing the first match and highlights the row
• Improvement: Dashboard paginated lists show a loading indicator while fetching data
• Improvement: Hub dashboard navigates to and highlights newly provisioned instances
• Improvement: --consul-register-hub option to disable hub parent service Consul registration while keeping instance registration active
• Improvement: Log formatting updated with elapsed timestamps, short level labels, and consistent field order
• Improvement: Dashboard service stats panes are now always visible on service pages (projects, cache, builds, workspaces) rather than hidden until data arrives
• Improvement: Updated vendored MinIO test server from RELEASE.2025-07-23 to RELEASE.2025-09-07 (HTTP listener bugfixes, ListObject fix)
• Improvement: HTTP client retries on CURLE_PARTIAL_FILE (truncated response) and logs the raw curl error code on retry
• Improvement: Default memory allocator changed back to mimalloc from rpmalloc
• Improvement: Hub proxy returns graceful responses when an instance is unavailable instead of a generic bad gateway error
• Bugfix: Dashboard stats tiles no longer flicker on page load when WebSocket updates arrive before all stats are available
• Bugfix: Build storage operations now silence HTTP exceptions on aborted requests instead of logging spurious errors
• Bugfix: Avoid use of O_CLOEXEC when operating on shared memory on Mac and Linux as it was reported to cause issues for MacOS 26.4 and later
• Bugfix: Added logic to shared memory instance state management to ensure unclean shutdown followed by restart with identical pid doesn't lead to errors. Particularly likely to happen when running on k8s

• Feature: Incremental CAS-based hydration/dehydration replacing the previous full-copy approach
• Feature: S3 hydration backend with multipart upload/download support
• Feature: Configurable thread pools for hub instance provisioning and hydration --hub-instance-provision-threads defaults to max(cpu_count / 4, 2). Set to 0 for synchronous operation. --hub-hydration-threads defaults to max(cpu_count / 4, 2). Set to 0 for synchronous operation.
• Feature: Hub obliterate operation deletes all local and backend hydration data for a module
• Improvement: Hub triggers GC on instance before deprovisioning to compact storage before dehydration
• Improvement: GC status now reports pending triggers as running
• Improvement: S3 client debug logging gated behind verbose mode to reduce log noise at default verbosity
• Improvement: Hub dashboard Resources tile now shows total memory
• Improvement: filesystemutils moved from zenremotestore to zenutil for broader reuse
• Improvement: Hub uses separate provision and hydration worker pools to avoid deadlocks
• Improvement: Hibernate/wake/deprovision on non-existent or already-in-target-state modules are idempotent
• Improvement: ScopedTemporaryDirectory with empty path now creates a temporary directory instead of asserting
• Improvement: File modification tick on Linux/macOS now uses sub-second precision (100ns), matching Windows FILETIME granularity
• Improvement: Dashboard front page now paginates Cooked Projects and Cache namespace lists
• Improvement: Cache and Projects pages now paginate and sort lists consistently with the front page
• Improvement: Hub dashboard adds obliterate button for individual, bulk, and by-name module deletion
• Improvement: Consul service registration for fully provisioned hub instances now sets initial check status to passing
• Bugfix: Fixed hub Consul health check registering with /hub/health endpoint which does not exist; now uses /health
• Bugfix: Fixed oplog export crash when a temp attachment file already exists in the pending directory
• Bugfix: Hub dashboard Resources tile was missing total disk space
• Bugfix: Fixed oplog export losing small attachments when block reuse is active; reused blocks were omitted from the container
• Bugfix: Object store now falls back to sending full payloads and 200 when receiving multi-range requests instead of responding with bad request

• Feature: Hub dashboard proxy - instance dashboards are accessible through the hub server at /hub/proxy/{port}/ without requiring direct port access
• Feature: Hub dashboard provision button allows creating new module instances from the web UI
• Improvement: Consul token is now re-read from the environment variable on every request, allowing token rotation without restarting the service
• Improvement: Hub child process spawning on macOS now uses posix_spawn in line with Apple recommendations
• Bugfix: Handle zero length file attachments without assert
• Bugfix: Hub child process spawning on Linux now uses vfork instead of fork, preventing ENOMEM failures on systems with strict memory overcommit (vm.overcommit_memory=2)
• Bugfix: Fixed process group management on POSIX; child processes were not placed into the correct process group, breaking group-wide signal delivery
• Bugfix: Fixed IMDS token request using wrong Put overload, causing EC2 credential fetching to fail
• Bugfix: Fixed Consul health check URL using empty address instead of localhost when no address is configured
• Bugfix: Fixed Consul service deregister failing when the service was registered via a different agent; deregister now falls back to catalog deregister
• Bugfix: Fixed hub dashboard stats endpoint path in compute hub page

• Feature: Hub dashboard now shows a Resources tile with disk and memory usage against configured limits
• Feature: Hub module listing now shows state-change timestamps and duration for each instance
• Feature: Added --hub-hydration-target-config option to specify the hydration target via a JSON config file (mutually exclusive with --hub-hydration-target-spec); supports file and s3 types with structured settings

  {
    "type": "file",
    "settings": {
      "path": "/path/to/hydration/storage"
    }
  }

  {
    "type": "s3",
    "settings": {
      "uri": "s3://bucket[/prefix]",
      "region": "us-east-1",
      "endpoint": "http://localhost:9000",
      "path-style": true
    }
  }

• Improvement: Hub provisioning rejects new instances when disk or memory usage exceeds configurable thresholds; limits are disabled by default (0 = no limit)
  • --hub-provision-disk-limit-bytes - Reject provisioning when used disk exceeds this many bytes
  • --hub-provision-disk-limit-percent - Reject provisioning when used disk exceeds this percentage of total disk
  • --hub-provision-memory-limit-bytes - Reject provisioning when used memory exceeds this many bytes
  • --hub-provision-memory-limit-percent - Reject provisioning when used memory exceeds this percentage of total RAM
• Improvement: Hub process metrics are now tracked atomically per active instance slot, eliminating per-query process handle lookups
• Improvement: Hub, Build Store, and Workspaces service stats sections in the dashboard are now collapsible
• Improvement: Hub hydration dehydration skips the .sentry-native directory
• Improvement: builds ls structured output now includes rawHash for each file entry
• Improvement: Hub server now supports Lua config file for all hub-specific options
  • hub.upstreamnotification.* - upstream notification endpoint and instance ID
  • hub.consul.* - service registration endpoint, token, health interval, deregister timeout
  • hub.instance.* - base port, HTTP class, thread count, core limit, config path
  • hub.instance.limits.* - instance count cap, disk and memory usage limits
  • hub.hydration.* - hydration target spec and config path
  • hub.watchdog.* - cycle timing, inactivity timeouts, and activity check timeouts
• Improvement: Added --hub-instance-base-port-number as an alias for --hub-base-port-number, and --upstream-notification-instance-id as an alias for --instance-id
• Improvement: Added hub mode documentation at docs/hub.md
• Bugfix: Fix an issue when a file goes from being internally stored to externally referenced and it could get into an inconsistent state
• Bugfix: Hub watchdog loop did not check m_ShutdownFlag, causing it to spin indefinitely on shutdown
• Bugfix: Fixed MakeSafeAbsolutePathInPlace when a UNC prefix is present but path uses mixed delimiters
• Bugfix: Fixed concurrency issue in JupiterBuildStorage when updating stats

• Feature: Hub watchdog automatically deprovisions inactive provisioned and hibernated instances
• Feature: Added stats/activity_counters endpoint to measure server activity
• Feature: Added configuration options for hub watchdog
  • --hub-watchdog-provisioned-inactivity-timeout-seconds Inactivity timeout before a provisioned instance is deprovisioned
  • --hub-watchdog-hibernated-inactivity-timeout-seconds Inactivity timeout before a hibernated instance is deprovisioned
  • --hub-watchdog-inactivity-check-margin-seconds Margin before timeout at which an activity check is issued
  • --hub-watchdog-cycle-interval-ms Watchdog poll interval in milliseconds
  • --hub-watchdog-cycle-processing-budget-ms Maximum time budget per watchdog cycle in milliseconds
  • --hub-watchdog-instance-check-throttle-ms Minimum delay between checks on a single instance
  • --hub-watchdog-activity-check-connect-timeout-ms Connect timeout for activity check requests
  • --hub-watchdog-activity-check-request-timeout-ms Request timeout for activity check requests
• Feature: Added sessions service - servers register themselves with a heartbeat and remote servers can announce themselves via --sessions-url; session state (active/ended) is visible in the dashboard with live WebSocket updates, metadata panel, and paging
• Feature: Added per-session log ingestion - server log output is forwarded to the session record and viewable in the sessions dashboard with incremental updates, follow/newest-first toggles, and text filtering
• Feature: Added Object Store dashboard page showing bucket summary stats and per-bucket object listings
• Feature: Added Network dashboard page with live HTTP stats, per-service request table, proxy connections, and distinct client IP / session counts via HyperLogLog
• Feature: Added Storage dashboard page with per-volume disk usage breakdown, GC status (next-run countdown, last-run stats), and paginated GC history with expandable detail panels
• Feature: Added in-dashboard Documentation page with sidebar navigation, markdown rendering, Mermaid architecture diagrams, collapsible sections, and text filtering
• Feature: Added Workspaces dashboard page with HTTP request stats and per-workspace metrics
• Feature: Added Build Storage dashboard page with service-specific HTTP request stats
• Feature: Local IP addresses are now included in the /status/status and /health/info responses and displayed in the dashboard banner
• Feature: Added S3 hydration backend for hub mode (--hub-hydration-target-spec s3://<bucket>[/<prefix>])
  • Credentials resolved from AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY env vars, falling back to EC2 instance profile via IMDS
  • Each dehydration uploads to a new timestamped folder and commits a current-state.json pointer on success, so a failed upload never invalidates the previous state
  • Hydration downloads to a temp directory first and only replaces the server state on full success; failures leave the existing state intact
• Feature: Added zen bench disk <path> command to gather basic disk performance numbers for diagnostics. Measures sequential/random/clone and disk operation (create,delete) timings
• Feature: Added transparent TCP proxy mode (zenserver proxy) for inspecting and monitoring HTTP/1.x traffic between clients and upstream Zen servers in real time. Useful for observing multi-server/client interactions during development. Defaults to port 8118 with its own data directory to avoid collisions with normal instances.
• Feature: Added proxy dashboard showing live connection stats, per-target request counts, active connections, bytes transferred, and client IP/session ID rollups
• Feature: Dashboard banner now shows the running server mode (Zen Proxy, Zen Compute, etc.)
• Feature: Log levels can now be set per-category using wildcard patterns (e.g. --log-debug=proxy.*)
• Feature: Added zen down --all flag to shut down all running zenserver instances
• Feature: Added xmake kill task to terminate all running zenserver instances
• Feature: Support ZEN_MALLOC environment variable for default allocator selection; default allocator switched to rpmalloc
• Feature: Added Dockerfile and extended GHA workflows to produce zenserver Docker images, primarily intended for compute nodes
• Feature: Added a lightweight crash handler installed before Sentry initializes. It remains active for the full process lifetime when Sentry is disabled
• Feature: Added --allow-port-probing option to control whether zenserver searches for a free port on startup (default: true, automatically false when --dedicated is set)
• Feature: Added support for consul token passed via environment variable for zen hub, default env var name is CONSUL_HTTP_TOKEN, change with --consul-token-env option.
• Feature: Added zen hub command for managing a hub server and its provisioned module instances:
  • zen hub up - Start a hub server (equivalent to zen up in hub mode)
  • zen hub down - Shut down a hub server
  • zen hub provision <moduleid> - Provision a storage server instance for a module
  • zen hub deprovision <moduleid> - Deprovision a storage server instance
  • zen hub hibernate <moduleid> - Hibernate a provisioned instance (shut down, data preserved)
  • zen hub wake <moduleid> - Wake a hibernated instance
  • zen hub status [moduleid] - Show state of all instances or a specific module
• Feature: Added new hub HTTP endpoints for instance lifecycle management:
  • POST /hub/modules/{moduleid}/hibernate - Hibernate the instance for the given module
  • POST /hub/modules/{moduleid}/wake - Wake a hibernated instance for the given module
• Feature: Added new hub options for controlling provisioned storage server instances:
  • --hub-instance-http HTTP server implementation for instances (asio/httpsys)
  • --hub-instance-http-threads Number of HTTP connection threads per instance
  • --hub-instance-corelimit Limit CPU concurrency per instance
  • --hub-instance-config Path to Lua config file for instances
  • --hub-hydration-target-spec Specification for the hydration target (e.g. file://<path>); defaults to <data-dir>/servers/hydration_storage
• Improvement: Dashboard navigation unified into a zen-nav web component shared across all pages
• Improvement: Embedded frontend zip is now deflate-compressed, reducing size from ~3.4 MB to ~950 KB
• Improvement: BroadcastSink introduced for shared dynamic log sink management; adding/removing a sink is now O(1) and immediately visible to all loggers
• Improvement: Console logging is now faster thanks to improved buffering and coloration logic
• Improvement: Added support for CoW block-cloning (used by build download) on Linux (tested with: btrfs/ XFS)
• Improvement: Added full-file CoW copying on macOS (APFS)
• Improvement: Updated libcurl to 8.19.0 to solve macOS trust issues (curl/curl#20435)
• Improvement: Updated asio to 1.38.0
• Improvement: Updated fmt to 1.12.1
• Improvement: Updated Oodle to 2.9.15
• Improvement: Updated mimalloc to v2.2.7
• Improvement: Server now logs the active memory allocator at startup
• Improvement: Fixed issue where oplog upload could create blocks larger than the max limit (64Mb)
• Improvement: Add easy access options for sanitizers with xmake config and xmake test as options
  • --msan=[y|n] Enable MemorySanitizer (Linux only, requires all deps instrumented)
  • --asan=[y|n] Enable AddressSanitizer (disables mimalloc and sentry)
  • --tsan=[y|n] Enable ThreadSanitizer (Linux/Mac only)
• Improvement: Hub now manages a deterministic port pool for provisioned instances allowing reuse of unused ports
• Improvement: Hub module listing now includes per-instance process metrics (memory, CPU time, working set, pagefile usage)
• Improvement: Hub now monitors provisioned instance health in the background, refreshes process metrics periodically, and automatically recovers crashed instances
• Improvement: Hub no longer exposes raw StorageServerInstance pointers to callers; instance state is returned as value snapshots (Hub::InstanceInfo)
• Improvement: Hub instance access is now guarded by RAII per-instance locks (SharedLockedPtr/ExclusiveLockedPtr), preventing concurrent modifications during provisioning and deprovisioning
• Improvement: Hub instance lifecycle is now tracked as a HubInstanceState enum covering transitional and stable states; exposed as a string in the HTTP API and dashboard
• Improvement: xmake sln now accepts an optional positional argument to select the IDE target (e.g. xmake sln vs2026); defaults to vs2022 on Windows, xcode on macOS, vscode on Linux. The --open flag can now appear in any position (e.g. xmake sln vs2026 --open)
• Improvement: zen up refactored to use shared StartupZenServer/ShutdownZenServer helpers (also used by zen hub up/zen hub down)
• Improvement: Hub dashboard module list improved
  • Animated state dots, with flashing transitions for hibernating, waking, provisioning, deprovisioning, and recovering; static red dot for crashed
  • Per-row port used by the provisioned instance
  • Per-row hibernate, wake, and deprovision actions with confirmation for destructive operations
  • Per-row button to open the instance dashboard in a new window
  • Multi-select with bulk hibernate/wake/deprovision and select-all
  • Pagination at 50 lines per page
  • Inline fold-out panel per row with human-readable process metrics
• Improvement: Hub provision, deprovision, hibernate, and wake operations are now async. HTTP requests returns 202 Accepted while the operation completes in the background
• Improvement: Hub returns 202 Accepted (instead of 409 Conflict) when the same async operation is already in progress for a module
• Improvement: Hub returns 200 OK when a requested state transition is already satisfied
• Improvement: Provisioning a hibernated instance now automatically wakes it instead of requiring an explicit wake call first
• Improvement: Deprovisioning now accepts instances in Crashed or Hibernated states, not just Provisioned
• Improvement: Added --consul-health-interval-seconds and --consul-deregister-after-seconds options to control Consul health check behavior (defaults: 10s and 30s)
• Improvement: Lazy initialize CpuSampler - reduces startup time by ~600 ms when --no-sentry is enabled
• Improvement: Consul registration now occurs when provisioning starts; health check intervals are applied once provisioning completes
• Improvement: Front page now shows Hub and Object Store activity tiles; HTTP panel is fixed above the tiles grid
• Improvement: HTTP stats tiles now include 5m/15m rates and p999/max latency across all service pages
• Bugfix: Fixed sentry-native build to allow LTO on Windows
• Bugfix: Fixed zenserver exiting on startup when encountering a stale shared-memory entry from a previous instance
• Bugfix: Minor test stability fixes (flaky hash collisions, per-thread RNG seeds)
• Bugfix: Handle long paths in the project store
• Bugfix: Authentication callbacks are not thread safe, ensured call sites does single threaded calls
• Bugfix: Retry OIDC token refresh once on failure before propagating the error
• Bugfix: Handle HTTP 501 (Not Implemented) from Jupiter as a signal to fall back from multi-range to single-range requests
• Bugfix: Fixed shutdown event not being cleared after the server process exits in ZenServerInstance::Shutdown(), which could cause stale state on reuse
• Bugfix: Don't try to wipe .sentry-native folder at missing manifest - sentry is already running. Reduces startup time by ~450 ms when data folder is empty

• Improvement: OIDC token expiry parsing now validates the UTC timezone marker and handles sub-second precision gracefully
• Bugfix: Fixed crash at startup of zen storage server when build store was not enabled (but structured cache was enabled)

• Improvement: Updated libcurl to 8.19.0 to solve macOS trust issues (curl/curl#20435)
• Bugfix: Authentication callbacks are not thread safe, ensured call sites does single threaded calls
• Bugfix: Retry OIDC token refresh once on failure before propagating the error
• Bugfix: Don't do unattended OIDC token refresh unless explicitly requested
• Bugfix: Don't hide the oidctoken process when fetching tokens in httpprojectstore
• Bugfix: Handle HTTP 501 (Not Implemented) from Jupiter as a signal to fall back from multi-range to single-range requests

• Bugfix: Crash at startup if a log message was emitted before logging is properly initialized

• Feature: Add --allow-partial-block-requests to zen oplog-import
• Feature: The integrated dashboard has been updated with streamlined more interactive interface with streaming statistics where available. The dashboard can now also be reached without qualification. I.e http://localhost:8558/ is sufficient.
• Feature: Added 404 page to dashboard, to make it easier to find your way back to a valid URL
• Feature: zen ui can be used to open dashboards for local instances
• Feature: HttpClient now properly handles multi-part request/response
• Feature: Added POST variant to /builds/{namespace}/{bucket}/{buildid}/blobs/{hash} endpoint that accepts a CbObject payload with 1 or more ranges
• Feature: OidcToken helper executable is now bundled with the zen/zenserver binaries
• Feature: Added support for Unix domain sockets. You can now launch zenserver with --unix-socket=<path> and it will listen on the specified socket. It will still listen on a network socket when using this option. The HTTP client and Websockets clients also support domain sockets.
• Feature: Added HTTPS support to zenserver when using http.sys as well as the asio http path. HTTPS requires specific setup for certificates and the configuration differs between the two back-ends. Please review the command line help for details.
• Feature: Added --oidctoken-exe-path command-line option for use when zenserver is not installed beside an OidcToken executable
• Feature: Added --allow-external-oidctoken-exe command-line option to control whether or not requests are allowed to specify an arbitrary OidcToken executable
• Feature: Added --restrict-content-types command-line option to zenserver to disallow requests with a content-type that is potentially vulnerable to cross-site request forgery
• Feature: Basic consul integration for zenserver hub mode, restricted to host local consul agent and register/deregister of services
• Feature: Added new options to zenserver hub mode
  • consul-endpoint - Consul endpoint URL for service registration (empty = disabled)
  • hub-base-port-number - Base port number for provisioned instances
  • hub-instance-limit - Maximum number of provisioned instances for this hub
  • hub-use-job-object - Enable the use of a Windows Job Object for child process management (Windows only)
• Improvement: zen builds download now uses multi-range requests for blocks to reduce download size
• Improvement: zen oplog-import now uses partial block with multi-range requests for blocks to reduce download size
• Improvement: Improved feedback in log/console during zen oplog-import
• Improvement: --allow-partial-block-requests now defaults to true for zen builds download and zen oplog-import (was mixed)
• Improvement: Improved range merging analysis when downloading partial blocks
• Improvement: IAX's lane tracing
• Improvement: Eliminated spurious sentry logging during startup and allocated sentry-sdk logging channel defaulting to WARN. The logging level can be changed via --log-debug=sentry-sdk or --log-info=sentry-sdk etc Though it should be noted that since Sentry is initialized very early there is a small period where the default will be WARN no matter what you pass on the command line.
• Improvement: Output more information about auth options for zen command
• Improvement: Better progress reporting during oplog import
• Improvement: ETA estimate is now shown during oplog and builds operations when using --plain-progress and --log-progress
• Bugfix: --plain-progress style progress bar should now show elapsed time correctly
• Bugfix: Time spent indexing local and remote state during zen builds download now show the correct time
• Bugfix: ObjectStore failed to correctly parse urls with sub folder paths causing 404 to be returned
• Bugfix: Total attachment size when querying for bucket sizes could end up incorrect
• Bugfix: Scavenge of other downloads is now disabled for old state files due to problems with consistency

• Feature: Added --security-config-path option to zenserver to configure security settings

  • Expects a path to a .json file

  • Default is an empty path resulting in no extra security settings and legacy behavior

  • Current support is a top level filter of incoming http requests restricted to the password type

  • password type will check the Authorization header and match it to the selected authorization strategy

  • Currently the security settings is very basic and configured to a fixed username+password at startup

    {
        "http" {
            "root": {
                "filter": {
                    "type": "password",
                    "config": {
                          "password": {
                              "username": "<username>",
                              "password": "<password>"
                          },
                        "protect-machine-local-requests": false,
                          "unprotected-uris": [
                              "/health/",
                              "/health/info",
                              "/health/version"
                          ]
                    }
                }
            }
        }
    }
    

• Improvement: Reduced time project and project oplogs are locked during GC and Validation

• Improvement: zen now supports additional configuration of logging options, such as --log-warn=... for configuring log levels, etc (see zen --help)

• Improvement: Web UI shows both hard and soft package dependencies

• Improvement: Web UI presents ops with only files without resorting to json representation

• Improvement: Web UI offers a cook artifacts view to present cook dependencies

• Bugfix: If a corrupted block (or partial block) is downloaded, handle it gracefully and end the download instead of causing an assert

• Improvement: When validating cache records read from disk we now do a limited validation of the payload to reduce overhead
• Improvement: Reduce maximum size per chunk to read to reduce disk contention
• Improvement: Reduce default size for block store chunk read window when iterating chunks
• Improvement: Increase timeout before warning on slow shut down of zenserver
• Improvement: On Windows, TransmitFile is used to transfer data directly from files when possible
• Improvement: Revise logic for enabling work backlog during zen builds download
• Improvement: Added --maxtimeslice option to zen scrub command to control how long a scrub operation may run
• Improvement: Increased the default scrub timeslice from 1 min 40 sec to 5 min.
• Improvement: Reduce lock contention when performing a scrub operation
• Improvement: Replaced http routing regex matching with faster matching lambdas for admin, buildstore, projectstore, objectstore and workspaces services
• Improvement: Ported optimizations of MeasureVarUInt from UE (CL50464402)
• Bugfix: Restore /health/log and /health/info endpoint functionality
• Bugfix: Fixed 32-bit truncation of transmission chunk sizes when using the asio http path
• Bugfix: zen builds download could leave files from previous downloads in the target folder
• Bugfix: Fixed formatting error on dashboard stat page

• Feature: zen builds upload now support structure manifest input for --manifest-path when the path has a .json extension enabling multi-part upload

  • The unstructured format expects one line per file relative to the root with '/' as a path delimiter

  • The structured format uses JSon format and the --manifest-path must have extension .json to enable structured input

    {
        "parts": {
            "default" : {
                "partId": "f939f3939939fff3f3202", # optional - used to control the id of each part
                "files": [
                    "foo/bar",
                    "baz.exe"
                ]
            },
            "symbols": {
                "files": [
                    "baz.pdb"
                ]
            }
        }
    }
    

• Feature: zen builds download now supports --download-spec-path to determine what content to download from a build

  • The unstructured format expects one line per file relative to the root with '/' as a path delimiter

  • The structured format uses JSon format and the --download-spec-path must have extension .json to enable structured input

    {
        "parts": {
            "default" : {
                "files": [
                    "foo/bar",
                    "baz.exe"
                ]
            },
            "symbols": {
                "files": [
                    "baz.pdb"
                ]
            }
        }
    }
    

• Feature: Added experimental zenserver "hub" mode which is used to manage a set of zenserver instances on a host

• Feature: Added --chunking-cache-path option to zen builds upload and zen builds diff

  • Path to cache for chunking information of scanned files. Default is empty resulting in no caching

• Improvement: For zen builds download, reworked scheduling writes of downloaded data to reduce memory usage

• Improvement: For zen builds download, reworked copy from local data buffering to reduce memory usage

• Bugfix: Avoid conversion from JSon to compact binary when querying for builds to avoid integer vs float conversion issues

• Bugfix: Fix oplog navigation elements that were showing up as HTML escape codes instead of the intended characters

• Bugfix: Make sure zenserver can handle non-ascii characters in command line options
• Bugfix: Default exclusion folders was lost causing --clean option for zen builds download to wipe the .zen working folder

• Temporarily disabled asio TransmitFile path due to issues with large transfers

• Feature: Added --exclude-folders to zen upload, zen download and zen diff to extend the default exclude folders. Each folder name is separated with ';' or ','
  • The default folders that are always excluded are:
    • .unsync
    • .zen
    • .ugs
    • .zen-tmp
• Feature: Added --exclude-extensions to zen upload and zen diff to set the file extensions to exclude. Each extension name is separated with ';' or ','
• Feature: Added --result-path option to zen builds ls to output structured to a file
• Improvement: On Windows, the asio HTTP back-end now uses TransmitFile to send data directly from files instead of using memory mapping.
• Improvement: Optimized scavenge lookup performance
• Improvement: Enable limit-overwrite behavior by default
• Improvement: Validate chunk hashes when dechunking files in oplog import
• Improvement: Use stream decompression when dechunking files
• Improvement: When assembling blocks for oplog export, make sure we keep under/at block size limit
• Improvement: Make cancelling of oplog import more responsive
• Improvement: Use decompress to composite to avoid allocating a new memory buffer for uncompressed chunks during oplog import
• Improvement: Reduce memory buffer size and allocate it on demand when writing multiple chunks to block store
• Improvement: Reduce lock contention when fetching/checking existence of chunks in block store
• Improvement: If we fail to create the server mutex, gracefully report the problem and exit without sending error to Sentry
• Improvement: Excluded folder names are now matched by folder name in subfolders in addition to root level folders
• Improvement: If the path given in --oidctoken-exe-path for zen commands is invalid, exit with error instead of ignoring the argument
• Bugfix: Upstream propagation of Put operations would not retain the overwrite cache policy if it was used
• Bugfix: zen oplog-download failed to initialize build part id causing it to fail to download the build part
• Bugfix: Windows: Set up UTF8 as current locale to properly handle non-ascii characters

• Feature: zen oplog-export, zen oplog-import and zen oplog-download now has options to boost workers
  • --boost-worker-count - Increase the number of worker threads - may cause computer to be less responsive
  • --boost-worker-memory - Increase the limit where we write downloaded data to temporary storage to conserve space - may cause computer to be less responsive due to high memory usage
  • --boost-workers - Enables both 'boost-worker-count' and 'boost-worker-memory' - may cause computer to be less responsive
• Improvement: At end of zen builds download and zen oplog-import we now show information over the source of the data - cache, cloud storage, s3/azure, replication
• Improvement: Use the improved block reuse selection function from zen builds upload in zen oplog-export to reduce oplog download size
• Improvement: Remove duplicate download of chunks from different blocks in zen oplog-export
• Improvement: Changed some logging output to have them not get caught by Hordes error/warning log parsing
• Improvement: Refactored boost options for zen builds operations upload, download, diff, prime-cache, fetch-blob and validate-part
  • --boost-worker-count - Increase the number of worker threads - may cause computer to be less responsive
  • --boost-worker-memory - Increase the limit where we write downloaded data to temporary storage to conserve space - may cause computer to be less responsive due to high memory usage
  • --boost-workers - Enables both 'boost-worker-count' and 'boost-worker-memory' - may cause computer to be less responsive
• Bugfix: Upload of oplogs could reference multiple blocks for the same chunk causing redundant downloads of blocks
• Bugfix: Fixed warning due to calling std::filesystem::equivalent on a non-existing path (again)

• Improvement: asio http server now supports the --dedicated option which prevents port remapping
• Improvement: If a server http port is already in use and --dedicated is enabled we do more retries when opening the http port
• Improvement: Automatic scaling up of http server threads when using --dedicated is only done if --http-threads is not specified
• Improvement: Refactored cache batch fetch operations to not do complex execution in destructor of PutBatch and GetBatch classes
• Improvement: Stop unhandled exceptions from async work in windows threadpool
• Improvement: Upload of data to zen cache during oplog import is now disabled by default
• Bugfix: Fixed warning due to calling std::filesystem::equivalent on a non-existing path
• Bugfix: Resume download in http client no longer tries to read from a stale header iterator
• Bugfix: Zen builds cache background upload work no longer keeps memory mapped buffers alive in backlog queue
• Bugfix: If no chunks in a batch put operation needs to be written we no longer call the write operation which resulted in an ASSERT
• Bugfix: Compressed buffers with zero size payload failed to decompress with "none" stream decompression
• Bugfix: Changed output that was hardcoded to log to console to follow intended log output rules

• Feature: Added --append option to zen builds download
  • --append=false removes local content and adds downloaded content
    • If wildcards are given, removes local state and all the downloaded content that matches the wildcards will be added
    • If no wildcards are given, removes local state and all the downloaded content will be added
    • This is the default behavior
  • --append=true add downloaded content to the local state
    • If wildcards are given, all local state that matches the wildcards will be removed and all downloaded content that matches the wildcards will be added
    • If no wildcards are given, all downloaded content will be added/overwrite local state files
• Improvement: Increased timeout and add retry when testing builds host connectivity
• Improvement: Eliminated redundant file flushing on log files. Now only flushes if something was actually written
• Improvement: Deeper validation of data when scrub is activated (cas/cache/project)
• Improvement: Enabled more multi threading when running scrub operations
• Improvement: Added means to force a scrub operation at startup with a new release using ZEN_DATA_FORCE_SCRUB_VERSION variable in xmake.lua
• Bugfix: If oplog container save fails the error code is now propagated out as a return code for the command
• Bugfix: GetCacheChunk range requests now follows UE DDC rules for ranges outside of chunk size
• Bugfix: Stream decompression of none-encoded data on disk would read wrong data if chunk did not start at beginning of file

• Bugfix: Block store GC would write the wrong chunk data for chunks larger than 256KB
• Bugfix: Remove read-only flags for files copied as part of a full service install operation

• Improvement: Drop-all action added to dashboard for projects and z$ namespaces
• Bugfix: Fixed ASSERT when using zen builds download with wildcard filtering due to needed loose chunks being incorrectly filtered out

• Improvement: Sentry is now disabled by default when building locally, to avoid uploading crashes due to local changes. CI builds still enable it just like before
• Improvement: Include the server version on the dashboard's start page
• Bugfix: Wildcard matching could cause out of bound read error
• Bugfix: Fixed buffering of written blocks during GC which could lead to chunks returned as not found until restart

• Include all changes from 5.7.7
• Improvement: The build process no longer relies on vcpkg for external packages. Instead we now use xmake native packages, with package definitions in-tree (in repo/), along with some in-tree snapshots in thirdparty/
  • Updates libcurl to 8.11.0
  • Bumps xmake version used in CI to 2.9.9 to fix some issues when building xmake packages for Windows
  • Enables use of ASAN on all platforms, TSAN/UBSAN/MSAN/LSAN on Linux/MacOS (see comments in xmake.lua)
  • Crashpad handler on MacOS is now supported also on arm64 as multi-architecture binary
  • Updates openssl to 3.6.0
• Feature: Added option to zen oplog-download to download attachments
  • --attachments use to provide a comma separated list of attachments in IoHash form to download
  • --attachments-path the output folder for the attachments to download
• Feature: Added option to zen oplog-download command
• Feature: Added local caching of project and oplog data to zen oplog-download to improve multiple calls to download attachments for the same oplog
  • --force disables local cache
• Feature: Added support for zen cache host to zen oplog-download
  • --zen-cache-host is used to override any automatically resolved zen cache host
• Feature: Added output option for zen oplog-download, default output for progress is "pretty"
  • --quiet , --verbose,--plain-progress and --log-progress
• Feature: Added option to boost worker count for zen oplog-download command
  • --boost-workers increases both network and disk/io worker thread count, disabled by default
• Feature: Added option --boost-worker-memory for zen builds command. Use with caution as it can cause zen.exe to use over 40 GB of memory during a download
• Feature: Added --zen-cache-upload option to zen builds upload, zen build download and zen oplog-download command to control upload to zen cache host. Defaults to true, disable with --zen-cache-upload=false
• Improvement: zen builds download now verifies that there is enough free space on the target disk
• Improvement: Hide compress/download rates when uploading build when complete
• Improvement: Compress partial file chunks inside blocks if appropriate
• Bugfix: If we fail to finalize a build part during zen builds upload due to request of upload of pre-existing blobs, fail the upload after retry attempts
• Bugfix: Don't display "Validating" progress title during download if validation is disabled
• Bugfix: zen builds wildcard options with leading local directory prefixes ./ or .\ will be trimmed to exclude the local directory prefix

• Rollback to 5.7.6
• Updated: Bump vcpkg version to 2024.06.15
• Improvement: Exclude .sym and .psym files from chunking and compress them as individual chunks
• Improvement: Exclude .ogg and .jpg files from chunking and compress them as individual chunks
• Improvement: Exclude various source code file extensions from chunking and compress them as individual chunks
• Improvement: Self-hosted dashboard's summary of stats if more robust to future changes
• Improvement: Change the logic for number of network and disk threads for zen builds commands
• Improvement: Optimized file size check if currently written CAS block file
• Improvement: Flushing of CAS block store files are now done outside of synchronization lock
• Bugfix: Add quotes around messages when using --log-progress with zen builds commands
• Bugfix: Fix ASSERT after running zen workspace info on a non-existing workspace
• Bugfix: Fixed ASSERT when using zen workspace create command
• Bugfix: Storage stats of builds on dashboard start page fixed
• Bugfix: Fixed progress reporting when scanning for changes in local folder state
• Bugfix: Fixed issue on MacOS where trace options would not stick due to IterateCommandlineArgs bug
• Bugfix: Fixed race condition in Latch. Sometimes the Wait() could early out before the completion event had been set
• Bugfix: Flushing of CAS block store files are now done after the block is fully written
• Bugfix: Fixed ASSERT when using zen builds download with include/exclude wildcards

• Feature: Added zen builds prime-cache command to download all blocks and build blobs for a build and upload to a zen cache host
  • This replaces the --cache-prime-only option for the zen builds download command
  • If no zen cache host is specified or resolved it will only download the blobs, this can be used to measure download speed from Cloud Storage
  • If --zen-folder-path is not given it defaults to {current directory}/.zen. Temporary downloaded files are stored in that path
• Improvement: On Windows file systems that allow block reference counting we use it where possible to speed up copy of data during zen builds download operations
  • Enabled by default, disable with --allow-file-clone=false
• Improvement: Zen server concurrency can now be limited via the --corelimit option
• Improvement: Better progress reporting for zen builds upload and zen builds download when using --log-progress
• Improvement: Validation of files is now part of the "Writing Chunk" progress avoiding stall of feedback at end of the phase
• Improvement: Cleaned up http response error logging
• Improvement: Exclude .sym and .psym files from chunking and compress them as individual chunks
• Improvement: Exclude .ogg and .jpg files from chunking and compress them as individual chunks
• Improvement: Exclude various source code file extensions from chunking and compress them as individual chunks
• Improvement: Make cancelling of zen builds operations more responsive
• Improvement: zen builds now split large files that are compressed only into 64 MB chunks to avoiding very large files in Cloud Storage
• Improvement: zen builds now treats .msixvc files as non-compressable
• Improvement: Self-hosted dashboard's summary of stats is more robust to future changes
• Improvement: Validation of downloaded data in zen builds download in now fully opt-in
• Improvement: Change the logic for number of network and disk threads for zen builds commands
• Improvement: zen oplog-import command now does cloud storage host and Zenserver cache discovery
  • --builds-override-host option can be used instead of or in addtition to --builds to specify host to override cloud storage host discovery
  • --zen-cache-host specifies a Zenserver cache host that overrides any host discovery
• Improvement: If gc disk usage log file is corrupt, remove and restart the log
• Improvement: Optimized file size check if currently written CAS block file
• Improvement: Remove speculative RemoveFile call when saving large blobs in CAS store
• Improvement: Don't rebuild content lookup when verifying a folder downloaded with zen builds download
• Improvement: Flushing of CAS block store files are now done outside of synchronization lock
• Bugfix: Add quotes around messages when using --log-progress with zen builds commands
• Bugfix: Fix ASSERT after running zen workspace info on a non-existing workspace. UE-349934
• Bugfix: Fixed issue where GC could start after shutdown of GC had been requested causing a race
• Bugfix: Storage stats of builds on dashboard start page fixed
• Bugfix: Fixed ASSERT when zen builds download --allow-redirect was enabled and downloading partial block ranges
• Bugfix: If disk space is below low water mark or above high water mark - only run GC if we find any data we can remove
• Bugfix: If we can't run GC due to low disk space, skip GC and wait until next monitor interval to check again
• Bugfix: Bumped vcpkg version to 2024.06.15 to work around issues during vcpkg install
• Bugfix: Fixed progress reporting when scanning for changes in local folder state
• Bugfix: Make sure GC status disk load graph includes the latest measurement
• Bugfix: Flushing of CAS block store files are now done after the block is fully written

• Improvement: Oidc token executable is now launched hidden when launch from inside zenserver (oplog import/export)
• Improvement: Helper scripts get_ue_toolchain.h and ue_build.sh can now be used without passing an argument to them. If you don't specify a toolchain directory then the default of .tmp-ue-toolchain is used
• Bugfix: Fixed bug in block store GC where a chunk could be lost due to writing blocks larger than max block size

• Bugfix: Parsing of zen oplog-import --oidctoken-exe-path option fixed
• Bugfix: Fixed race condition when checking cas store block size causing chunks to be incorrectly reported as missing

• Bugfix: Parsing of zen builds --log-progress option fixed
• Bugfix: Rebuild State phase of zen builds download could read out of bound for target path array
• Bugfix: Don't include oplogs pending delete when scanning for oplogs
• Bugfix: Fixed issue where large ops in oplog would fail to read properly

• Feature: zen builds download and zen builds ls now allows multiple wildcards for --wildcard and --exclude-wildcard separated by semicolons (;)
• Improvement: Gracefully handle missing chunks when exporting an oplog
• Improvement: Add additional validations when reading disk cache records to get references in GC
• Bugfix: Fixed invalid namespace+bucket regexes in BuildStore (only fired with new MSVC compiler)
• Bugfix: GcScheduler could delay shutdown in some situations
• Bugfix: On exit, trace shutdown can happen before all threads completed their TLS cleanup
• Bugfix: When requesting a value via the GetCacheChunks rpc method, if the request specified a raw hash, only return a hit if the raw hash matches what was in the cache
• Bugfix: Flush of blockstore file could sometimes cause an error due to a race condition
• Bugfix: Add object validation when replaying oplog to avoid crash on malformed data

• Feature: Added --force option to zen builds download command that forces download of all content ignoring any local state
• Feature: Added zen oplog-download command to download the oplog body of a cooked output stored in Cloud DDC
  • Oplog source is specified using one of the following options
    • --cloud-url Cloud artifact URL for oplog
    • --host Base host to resolve download host from
    • --override-host Specific host to use without resolve
    • --assume-http2 assume that the builds endpoint is a HTTP/2 endpoint skipping HTTP/1.1 upgrade handshake
    • --namespace Builds Storage namespace
    • --bucket Builds Storage bucket
      • --build-id an Oid in hex form for the source identifier to use
    • --yes suppress conformation query when doing output of a very large oplog to console
    • --quiet suppress all non-essential console output
    • --output-path path to oplog output, extension .json or .cb (compact binary). Default is output to console
    • --system-dir override default system root path
    • Authentication options
      • Auth token
        • --access-token http auth Cloud Storage access token
        • --access-token-env name of environment variable that holds the Http auth Cloud Storage access token
        • --access-token-path path to json file that holds the Http auth Cloud Storage access token
        • --oidctoken-exe-path path to OidcToken executable
      • OpenId authentication
        • --openid-provider-name Open ID provider name
        • --openid-provider-url Open ID provider url
        • --openid-client-idOpen ID client id
        • --openid-refresh-token Open ID refresh token
        • --encryption-aes-key 256 bit AES encryption key for storing OpenID credentials
        • --encryption-aes-iv 128 bit AES encryption initialization vector for storing OpenID credentials
      • OAuth authentication
        • --oauth-url OAuth provier url
        • --oauth-clientid OAuth client id
        • --oauth-clientsecret OAuth client secret
• Feature: Added --oidctoken-exe-unattended tozen builds and zen oplog-download command to use unattended mode when launching oidc-token.exe
• Feature: New /prj/{project}/{oplog}/rpc endpoint method appendops to send an array of oplog ops and receiving a list of need for attachments not present
• Feature: Added usingtmpfiles boolean field to /prj/{project}/{oplog}/rpc method putchunks to be explicit about allowing move of temp attachment files
• Improvement: Change BadAlloc exceptions in GC to warnings
• Improvement: Add explict ASSERT exception catch in http plugin request processing
• Improvement: Make exceptions handled in http request processing to warnings
• Improvement: Revised project oplog in-memory representation which reduces load times and memory usage
• Improvement: Updated README.md to state the required version vcpkg
• Improvement: Updated rpmalloc to latest from 'develop' stream, now also compiles on Linux/Mac
• Improvement: Automatically remove enclosing quotes for zen builds --wildcard and --exclude-wildcard command line options
• Improvement: Change the responses from batch put (record and value) operations to allow more detailed reporting by clients of non-determinism issues
• Improvement: Change the responses from get record operations to allow clients to report when partial records are incomplete
• Improvement: zen command help now uses available horizontal space for output
• Improvement: Partial block request analisys has been improved with reduced download size at lower number of requests
• Improvement: Validate that --zen-cache-host exists and is responsive before begin used
• Improvement: Faster oplog import due to chunk existance check improvement
• Improvement: Cancelling oplog import is now more responsive during initial phase
• Improvement: Chunk block generation for zen oplog-export and zen builds upload command are now limited to max 4000 chunks per block
• Improvement: Added additional validation of compact binary objects when reading from disk/receiving from client
• Improvement: If blocks metadata is missing for a download --verbose will enable output of all the missing chunk block hashes
• Improvement: Make sure we call the previous terminate handle if present when we intercept terminate calls
• Improvement: Avoid allocating memory for call stack in terminate handle and assert callback
• Improvement: Warnings are now logged when iterating of chunks in CAS store and payloads are missing
• Feature: Options --allow-partial-block-requests for zen builds download command has been augmented with two new modes, zencacheonly and mixed. Defaults to mixed.
  • false only full block requests allowed
  • mixed multiple partial block ranges requests per block allowed to zen cache, single partial block range request per block to host
  • zencacheonly multiple partial block ranges requests per block allowed to zen cache, only full block requests allowed to host
  • true multiple partial block ranges requests per block allowed to zen cache and host
• Improvement: Windows: Added fallback code to use standard MoveFile api when rename via SetFileInformationByHandle fails in MoveToFile (used by filecas)
• Bugfix: Self-hosted dashboard; sorting oplog by size in the tree view would raise an error
• Bugfix: zen print command now properly outputs very large compact binary objects as json to console

• Improvement: Oplogs that have not been touched for 15 min are unloaded from memory during GC pass, oplogs are reloaded on demand

• Improvement: Oplogs read for GC/Validation operations are loaded using a lightweight mode to reduce memory usage and load times

• Improvement: Oplog GET operation now use a lightweight mode to reduce memory usage and load times when oplog is not already loaded into memory

• Improvement: Reduce memory usage for oplog Op address lookup

• Improvement: Cleaned up logging for oplogs/projects to reduce clutter

• Improvement: Allow large headers support in asio server implementation

• Improvement: Use consistent language for command line argument parsing errors

• Improvement: Changed zen command parsing errors to output help first and error last to make it easier to spot the error

• Improvement: Refactor zen command return codes to conform to valid Linux range (0-255)

    kSuccess     = 0,
    kOtherError  = 1,
    kBadInput    = 2,
  
    kOutOfMemory = 16,
    kOutOfDisk   = 17,
  
    kAssertError = 70,
  
    kHttpOtherClientError = 80,
    kHttpCantConnectError = 81,
    kHttpNotFound         = 66,	 // NotFound(404)
    kHttpUnauthorized     = 77,	 // Unauthorized(401),
    kHttpSLLError         = 82,
    kHttpForbidden        = 83,  // Forbidden(403)
    kHttpTimeout          = 84,  // RequestTimeout(408)
    kHttpConflict         = 85,  // Conflict(409)
    kHttpNoHost           = 86,
  
    kHttpOtherServerError	   = 90,
    kHttpInternalServerError = 91,	// InternalServerError(500)
    kHttpServiceUnavailable  = 69,	// ServiceUnavailable(503)
    kHttpBadGateway          = 92,	// BadGateway(502)
    kHttpGatewayTimeout      = 93,	// GatewayTimeout(504)
  

• Improvement: Faster project store /entries endpoint, 10-15% faster when using a reference set to limit entries

• Improvement: Add a new mode to worker thread pools to avoid starvation of workers which could cause long stalls due to other work begin queued up. UE-305498

• Bugfix: Linux only, set ownership of installed files to specified user when using zen service install --full --user

• Bugfix: Make sure all services are initialized and ready to accept requests before responding "OK" on health checks

• EXPERIMENTAL Feature: Added command line tool zen service to do maintenance operations for running zenserver as a local service
  • zen service status - Check the status of zenservice as local service, indidicating if it is installed, running and the options it was installed with
  • zen service install - Install zenserver as a local service, this requires admin/sudo priviliges to execute properly
    • --executable - Path to zenserver executable. Defaults to same path as zen command line executable that was issued but with filename replaced.
    • --name - Name of the installed service. Defaults to ZenServer
    • --full - If necessary, will uninstall a currently running service, then copy updated Zen binaries and debugging information to the location specified by --install-path and start the service. Requires --install-path to be specified.
    • --install-path - Path to install service binaries. Only necessary in conjunction with --full.
    • --display-name - Windows only, the user friendly display name of the service. Defaults to Unreal Zen Storage Service
    • --description - Windows only, the user friendly description of the service
    • --allow-elevation - Windows only, if command is not run with elevated priviliges, the command will attempt to relaunch itself with elevated priviliges
    • --user - Linux only, username to install service under. Will attempt to determine user via SUDO_USER environment variable if not set.
    • -- - All options given after the double dash will be added as command line arguments to the zenserver executable when starting the service
  • zen service uninstall - Uninstall zenserver as a local service, this requires admin/sudo priviliges to execute properly
    • --name - Name of the installed service. Defaults to ZenServer
    • --allow-elevation - Windows only, if command is not run with elevated priviliges, the command will attempt to relaunch itself with elevated priviliges
  • zen service start - Start the installed zenserver local service, this requires admin/sudo priviliges to execute properly
    • --name - Name of the installed service. Defaults to ZenServer
    • --allow-elevation - Windows only, if command is not run with elevated priviliges, the command will attempt to relaunch itself with elevated priviliges
  • zen service stop - Stops the running installed zenserver local service, this requires admin/sudo priviliges to execute properly
    • --name - Name of the installed service. Defaults to ZenServer
    • --allow-elevation - Windows only, if command is not run with elevated priviliges, the command will attempt to relaunch itself with elevated priviliges
• Feature: zen print now has a --show-type-info option to add type information to output of compact binary content
• Improvement: zen print now allows output of compact binary content even if they are in non-optimal format (Unifom vs Non-Uniform arrays and objects)
• Improvement: Switch to bits/sec for logged/output transfer rate when doing oplog import/export
• Improvement: Include http client session id in log/output when doing oplog import/export
• Improvement: Changed zenserver --quiet option to suppress INFO level messages and below to surface startup and runtime errors
• Feature: Added --noconsole option that suppresses all output to standard out, this works as the --quiet option used to work
• Bugfix: Stats information for Build Store (Zen Store Cache) no longer throws exception and outputs invalid state information
• Bugfix: Properly convert elapsed time for http operations when doing oplog import/export to Builds remote store

• Bugfix: Don't skip oplogs without package data in zen dashboard
• Bugfix: Finalize build (in addition to build part) when exporting oplogs

• Feature: Added zen build ls option to list the content of a build part(s)

  • Build source is specified using one of the following options
    • --cloud-url cloud artifact URL to build
    • --host or --override-host, --namespace, --bucket and --buildid
    • --filestorage, --namespace, --bucket and --buildid
  • --build-part-name to specify a particular build part(s) in the build
  • --wildcard windows style wildcard (using * and ?) to match file paths to include
  • --exclude-wildcard windows style wildcard (using * and ?) to match file paths to exclude. Applied after --wildcard include filter

• Feature: Added wildcard options for zen build download

  • --wildcard windows style wildcard (using * and ?) to match file paths to include
  • --exclude-wildcard windows style wildcard (using * and ?) to match file paths to exclude. Applied after --wildcard include filter

• Feature: Added global zenserver option --cache-bucket-limit-overwrites controlling Whether to require policy flag pattern before allowing overwrites or not. Default false = overwrites always allowed

• Feature: Add per bucket cache configuration option limitoverwrites (Lua options file only)

    cache = {
        bucket = {  -- This is the default for all namespaces
            limitoverwrites = true
        },
        buckets = { -- Here you can add matching per bucket name (matches accross namespaces)
            iostorecompression = {
              limitoverwrites = false
            },
        },
    }
  

• Improvement: Added --quiet option to zen builds commands to suppress non-essential output

• Improvement: Remove early wipe of target folder for zen download to allow for scavenging useful data

• Improvement: Refactored jupiter oplog export code to reuse builds jupiter wrapper classes

• Improvement: If zen builds, zen oplog-import or zen oplog-import command fails due to a http error, the return code for the program will be set to the error/status code

• Improvement: Add validation of cache bucket metadata manifest when reading legacy format
• Improvement: Don't set m_DispatchComplete in ParallelWork until after pending work countdown succeeds
• Improvement: Safeguard FormatCallstack to not throw exceptions when building the callstack string
• Improvement: Limit thread name length when setting it for debugger use
• Improvement: Don't allow assert callbacks to throw exception
• Improvement: When formatting log output for malformed attachments in a package message, allow the string buffer to grow instead of throwing exception
• Improvement: Refactored build store cache to use existing CidStore implementation instead of implementation specific blob storage
  • CAUTION This will clear any existing cache when updating as the manifest version and storage strategy has changed
• Improvement: If cloud-ddc requests upload of a blob it earlier reported as good to reuse we treat it as a transient error and attempt to retry
• Bugfix: Parents were not notified when successfully attaching to an existing server instance
• Bugfix: BuildStorage cache return "true" for metadata existance for all blobs that had payloads regardless of actual existance for metadata
• Bugfix: Builds download - don't query for block metadata if no blocks are required
• Bugfix: Add the referenced attachments correctly when storing inline cache bucket records using batch mode

• Improvement: If zen builds upload fails to upload metadata for a block with a 404 response (due to race condition from hitting different server) we save and retry metadata upload at end of upload
• Improvement: Updated Oodle libs to 2.9.14

• Feature: Added --sentry-environment to zen and zenserver
• Feature: Added --sentry-debug to zen and zenserver
• Feature: Added environment variable parsing for the following options:
  • UE_ZEN_SENTRY_ENABLED: --no-sentry (inverted)
  • UE_ZEN_SENTRY_DEBUG: --sentry-debug
  • UE_ZEN_SENTRY_ALLOWPERSONALINFO: --sentry-allow-personal-info
  • UE_ZEN_SENTRY_DSN: --sentry-dsn
  • UE_ZEN_SENTRY_ENVIRONMENT: --sentry-environment
• Feature: Added --output-path option to zen version command to save version information to a file
• Improvement: zen build --cloud-url option now accepts URLs without the api/v2/builds/ part in urls
• Bugfix: Range requests for build blobs that reached end of blob now works correctly
• Bugfix: Gracefully handle a malformed response when querying list of blocks
• Bugfix: Make sure we unregister cache namespaces and buckets when dropping them

• Bugfix: Don't require --namespace option when using zen list-namespaces command
• Bugfix: Crash in upload of blobs to Cloud DDC due to buffer range error
• Improvement: Zcache namespace and bucket information now shown on self-hosted dashboard.

• Bugfix: Reinstate zen builds --url option to old behaviour, use zen builds --cloud-url for "Cloud Artifact URL"

• Feature: zen builds list now allows --bucket to be skipped to allow for multi-bucket search, bucketRegex field must be included in the search query
• Feature: --url option for zen builds command has been reworked to accept a "Cloud Artifact URL", removing the need to specify "host", "namespace" and "bucket" separately
• Feature: zen builds pause, zen builds resume and zen builds abort commands to control a running zen builds command
  • --process-id the process id to control, if omitted it tries to find a running process using the same executable as itself
• Feature: New --sentry-dsn option for zen command line and zenserver to control Sentry reporting endpoint
• Improvement: Use fixed size block chunking for know encrypted/compressed file types
• Improvement: Skip trying to compress chunks that are sourced from files that are known to be encrypted/compressed
• Improvement: Add global open file cache for written files increasing throughput during download by reducing overhead of open/close of file by 80%
• Improvement: Multithreaded scavenge pass for zen builds download
• Improvement: Optimized check for modified files when verifying state of scavenged paths
• Improvement: Process report now indicates if it is pausing or aborting
• Improvement: Don't report OOD or OOM errors to Sentry when running zen builds commands
• Improvement: Use unique temporary file name when calling OidcToken executable to generate auth token
• Improvement: Moved sentry database path to temporary directory for zen commandline
• Bugfix: Zen CLI now initializes Sentry after command line options parsing, so that the options can be propetly taken into account during init
• Bugfix: Revert: zen builds upload now use the system temp directory for temporary files leaving the source folder untouched
• Bugfix: Use selected subcommand when displaying help for failed command line options in zen builds
• Bugfix: Make sure we recreate the log file in CAS/cache bucket when creating snapshot at startup causing lost changes. UE-291196

• Bugfix: Remove long running exclusive namespace wide locks when dropping buckets or namespaces
• Bugfix: Flush the last block before closing the last new block written to during blockstore compact. UE-291196
• Bugfix: Fix stats for memcached entries in disk cache buckets
• Feature: Drop unreachable CAS data during GC pass. UE-291196
• Improvement: Check available disk space more frequently and trigger GC operation if low water mark is reached
• Improvement: Faster oplog validate to reduce GC wall time and disk I/O pressure
• Improvement: zen builds upload now use the system temp directory for temporary files leaving the source folder untouched
• Improvement: NoneDecoder::DecompressToStream and NoneDecoder::CompressToStream now uses direct disk I/O
• Improvement: Add streaming upload from HttpClient to reduce I/O caused by excessive MMap usage

• Feature: Add per bucket cache configuration (Lua options file only)

    cache = {
        bucket = {  -- This is the default for all namespaces
                maxblocksize = 1073741824,
                payloadalignment = 4,
                memlayer = {
                    sizethreshold = 1024,
                },
                largeobjectthreshold = 2097152,
        },
        buckets = { -- Here you can add matching per bucket name (matches accross namespaces)
            iostorecompression = {
                maxblocksize = 1073741824,
                payloadalignment = 4,
                memlayer = {
                    sizethreshold = 1024,
                },
                largeobjectthreshold = 2097152,
            },
        },
    }
  

• Feature: zen oplog-import and zen oplog-export now supports --oidctoken-exe-path option

• Feature: Added --use-sparse-files option to zen builds command improving write performance of large files. Enabled by default.

• Bugfix: Handle invalid plugin config file without terminating

• Bugfix: Gracefully handle errors while running oplog-mirror

• Bugfix: Wait for async threads if dispatching of work using ParallellWork throws exception

• Bugfix: Fix crash when batch-fetching chunks and an exception was thrown in non-async code

• Bugfix: Implemented compact binary validation of custom fields

• Bugfix: Fix oplog creation during zen oplog-import

• Improvement: --cache-memlayer-sizethreshold is now deprecated and has a new name: --cache-bucket-memlayer-sizethreshold to line up with per cache bucket configuration

• Improvement: Remove CAS log files at startup when we write full state index

• Improvement: Optimize memory-buffered chunk iteration sizes

• Improvement: Skip command line arguments that are empty or just a single space for zen command line tool on Mac/Linux to handle "triple space" problem. UE-273411

• Improvement: Extend log information when httpsys response sending fails

• Improvement: Log warning when port remappng occurs due to desired port is already in use

• Improvement: Custom CopyFile in zen builds command increasing throughput by 50% on Windows and give better progress update

• Feature: Support for --log-progress to output UE style @progress log messages has been added to
  • zen builds upload
  • zen builds download
  • zen builds diff
  • zen builds validate-part
  • zen wipe
• Feature: zen builds upload and zen builds download not accepts --allow-redirect for Cloud hosts allowing direct upload to/download from backend store (such as S3)
• Bugfix: Add explicit lambda capture in CasContainer::IterateChunks to avoid accessing state data references
• Bugfix: Validate compact binary objects stored in cache before attempting to parse them to avoid crash when data is corrupt
• Bugfix: Add explicit lambda capture in CasContainer::IterateChunks to avoid accessing state data references
• Bugfix: Validate compact binary objects stored in cache before attempting to parse them to avoid crash when data is corrupt
• Bugfix: Don't report redundant error if disk is full when writing GC state or GC log
• Bugfix: Properly handle empty directories when using zen wipe command
• Improvement: If garbage collection fails due to out of disk or out of memory we issue a warning instead of reporting an error to Sentry
• Improvement: Improved cache bucket state index write time with ~15% and cache bucket side-car write time with ~20%
• Improvement: Optimize block compact reducing memcpy operations
• Improvement: Handle padding of block store blocks when compacting to avoid excessive flusing of write buffer
• Improvement: Handle padding when writing oplog index snapshot to avoid unnecessary flushing of write buffer
• Improvement: Reduce CPU and lock contention during non-critical phases of GC

• Improvement: Made metadata presentation for cooked output entries more generic
• Improvement: builds diff command uses state file if available reducing need to hash content of files
• Improvement: Retry http client requests if we get an unspecified internal error or a bad gateway response
• Improvement: Added Sentry integration to zen command line
  • --no-sentry disables Sentry integration, defaults to false (Sentry is enabled)
  • --sentry-allow-personal-info to allow personally identifiable information in Sentry reports, disabled by default
• Bugfix: Use proper FindClose call when using fallback when getting file attributes on windows
• Bugfix: Fixed race condition at final chunks when downloading multipart blobs which could lead to corruption and/or crash
• Bugfix: Fixed BigInt conversion error affecting the tree view in the web UI
• Bugfix: Limit retry count when finalizing build part to avoid getting stuck in an infinite loop
• Bugfix: Fixed lua config naming for zenserver --buildstore-disksizelimit option
• Feature: zen builds list-namespaces subcommand allows listing namespace and bucket information from jupiter builds API.
• Feature: New zen wipe command for fast cleaning of directories, it will not remove the directory itself, only the content
  • --directory - path to directory to wipe, if the directory does not exist or is empty, no action will be taken
  • --keep-readonly - skip removal of read-only files found in directory, defaults to true, set to false to remove read-only files
  • --quiet - reduce output to console, defaults to false
  • --dryrun - simulate the wipe without removing anything, defaults to false
  • --yes - skips prompt to confirm wipe of directory
  • --plain-progress - show progress using plain output
  • --verbose - enable verbose console output
  • --boost-workers - increase the number of worker threads, may cause computer to be less responsive, defaults to false
• Feature: EXPERIMENTAL New --plugins-config option to load plugins based on transport-sdk.
  • It accepts a json of format [{"name": "%path_to_dll%", "%opt1%": "%val1%", ...}, ].

• Bugfix: zen builds multipart download of large chunks could result in crash
• Bugfix: Fixed invalid hash check on messages from Jupiter when doing range http requests
• Bugfix: Handle temporary locked files more gracefully (Crowdstrike workaround)
• Bugfix: Avoid integer overflow in web UI causing numbers to show up as negative

• Improvement: zen builds now scavenges previous download locations for data to reduce download size, enabled by default, disable with --enable-scavenge=false
• Improvement: zen builds upload Fixed and improved layout of end of run stats output
• Improvement: zen builds Reduced clutter and added timing to some logging entries when --verbose is enabled
• Bugfix: If both --host and --override-host was given to a zen builds command, -override-host no host would be resolved
• Bugfix: Failing to rename a file during download sometimes reported an error when it succeeded when retrying

• Feature: zen oplog-export, zen oplog-import for --url (cloud) and --builds (builds) option now has --oidctoken-exe-path to let zen run the OidcToken executable to get and refresh authentication token
• Feature: zenserver option --buildstore-disksizelimit to set an soft upper limit for build storage data. Defaults to 1TB.
• Bugfix: Restore Mac minver back to 12.5
• Bugfix: Fixing bug where some imports were shown as a blank string in the web UI
• Improvement: Hide progress ETA number until at least 5% of work is done to reduce misleading processing times. UE-256121
• Improvement: Dynamically adjust console progress output based on console width. UE-256126
• Improvement: Usability improvements to oplog search in web UI. Case insensitve, short string search, reset search.

• Bugfix: Changed Mac minver from 12.5 to 14.0, and removed _LIBCPP_DISABLE_AVAILABILITY as a define on Mac due to executable incompatibility issues
• Bugfix: Fixed missing trailing quote when converting binary data from compact binary to json
• EXPERIMENTAL zen builds
  • Feature: zen builds auth option --oidctoken-exe-path to let zen run the OidcToken executable to get and refresh authentication token
  • Bugfix: Fixed upload failure on Mac/Linux if --zen-folder-path was not given. UE-265170
  • Feature: zen builds upload command has new option --find-max-block-count to control how many blocks we search for reuse.
  • Improvement: Bumped the default number of blocks to search during upload to 10000 (from 5000).

• Bugfix: GetModificationTickFromPath and CopyFile now works correctly on Windows/Mac
• Bugfix: Handling of quotes and quotes with leading backslash for command line parsing - UE-231677
• Improvement: When logging with a epoch time prefix, the milliseconds/fraction is now correct. We now also set the epoch to the process spawn time rather than the time when the logger is created
• Improvement: Reduce concurrent disk I/O during GC for cache buckets
• EXPERIMENTAL zen builds
  • Feature: zen builds list command has new options
    • --query-path - path to a .json (json format) or .cbo (compact binary object format) with the search query to use
    • --result-path - path to a .json (json format) or .cbo (compact binary object format) to write output result to, if omitted json format will be output to console
  • Feature: New /builds endpoint for caching build blobs and blob metadata
    • /builds/{namespace}/{bucket}/{buildid}/blobs/{hash} GET and PUT method for storing and fetching blobs
    • /builds/{namespace}/{bucket}/{buildid}/blobs/putBlobMetadata POST method for storing metadata about blobs
    • /builds/{namespace}/{bucket}/{buildid}/blobs/getBlobMetadata POST method for fetching metadata about blobs
    • /builds/{namespace}/{bucket}/{buildid}/blobs/exists POST method for checking existance of blobs
  • Feature: zen: --zen-cache-host option for upload and download operations to use a zenserver host /builds endpoint for storing build blob and blob metadata
  • Feature: zenserver: Add command line option --gc-buildstore-duration-seconds to control GC life time of build store data
  • Feature: zen --boost-workers option to builds upload, download and validate-part that will increase the number of worker threads, may cause computer to be less responsive
  • Feature: zen --cache-prime-only that uploads referenced data from a part to --zen-cache-host if it is not already present. Target folder will be untouched.
  • Feature: zen: --zen-folder-path added to builds command, list, upload, download, fetch-blob, validate-part to control where .zen folder is placed and named
  • Feature: Added --host option to use Jupiters list of cloud host and zen servers to resolve best hosts
  • Feature: Use local zenserver as builds cache if it has the builds service enabled and --cloud-discovery-host is provided and no remote zenserver cache hosts can be found
  • Improvement: Do partial requests of blocks if not all of the block is needed
  • Improvement: Better progress/statistics on upload and download
  • Improvement: Scavenge .zen temp folders for existing data (downloaded, decompressed or written) from previous failed run
  • Improvement: Faster abort during stream compression
  • Improvement: Try to move downloaded blobs with rename if possible avoiding an extra disk write
  • Improvement: Only clean temp folders on successful or cancelled build - keep it if download fails
  • Improvement: Do put/get build and find blocks while scanning local folder when uploading
  • Improvement: Don't chunk up .mp4 files as they generally won't benefit from deduplication or partial in-place-updates
  • Improvement: Emit build name to console output when downloading a build
  • Improvement: Reduced memory usage during upload and part upload validation
  • Improvement: Reduced I/O usage during upload and download
  • Improvement: Faster block regeneration when uploading in response to PutBuild/FinalizeBuild
  • Improvement: More trace scopes for build upload operations
  • Improvement: Progress bar automatically switches to plain mode when stdout is not a console
  • Improvement: Progress bar is much more efficient on Windows (switched away from printf)
  • Improvement: Improved stats output at end of upload and download operations
  • Improvement: Reduced disk I/O when writing out chunk blocks during download
  • Improvement: Collapse consecutive ranges when reading/writing data from local cache state
  • Improvement: If a chunk or block write operation results in more than one completed chunk sequence, do the additional verifications as async work
  • Improvement: Improved error reporting when async tasks fail
  • Improvement: At end of build upload we post statistics to the Jupiter build stats endpoint:
    • totalSize
    • reusedRatio
    • reusedBlockCount
    • reusedBlockByteCount
    • newBlockCount
    • newBlockByteCount
    • uploadedCount
    • uploadedByteCount
    • elapsedTimeSec
    • uploadedBytesPerSec
  • Improvement: Allow cook metadata to be browsed in the web UI
  • Improvement: ELF and MachO executable files are no longer chunked
  • Improvement: Compress chunks in blocks that encloses a full file (such as small executables)
  • Improvement: Only check known files from remote state when downloading to a target folder with no local state file
  • Improvement: Don't move existing local to cache and back if they are untouched
  • Improvement: Faster cleaning of directories
  • Improvement: Faster initial scanning of local state
  • Improvement: Added --override-host option as a replacement for --url (--url still works, but --override-host is preferred)
  • Improvement: Output Build and Build Part information to console during builds download
  • Improvement: Output zen executable path and version at start of builds commands
  • Bugfix: Strip path delimiter at end of string in StringToPath
  • Bugfix: Ensure that temporary folder for Jupiter downloads exists during verify phase
  • Bugfix: Fixed crash during download when trying to write outside a file range
  • Bugfix: MacOS / Linux zen build download now works correctly
  • Bugfix: Env auth parsing blocked parsing OAuth and OpenId options
  • Bugfix: Long file paths now works correctly on Windows
  • Bugfix: Validate that we can read input files correctly for files that are not chunked

• Feature: Added support for --trace, --tracehost and --tracefile options to zen CLI command
• Improvement: When logging HTTP responses, the body is now sanity checked to ensure it is human readable, and the length of the output is capped to prevent inadvertent log bloat
• Improvement: Instrumented zen builds download command code so we get more useful Insights output
• EXPERIMENTAL zen builds
  • Improvement: Better error reporting and faster exit on error
  • Improvement: Downloads without --clean now preserves untracked files in target folder
  • Improvement: Reduce write operations for duplicate files in dowloads
  • Improvement: Added on the fly validation zen builds download of files built from smaller chunks as each file is completed
  • Improvement: Do streaming decompression of loose blobs to improve memory and I/O performance
  • Improvement: Validate hash of decompressed data inline with streaming decompression
  • Improvement: Do streaming compression of large blobs to improve memory and I/O performance
  • Feature: Added --verify option to zen builds upload to verify all uploaded data once entire upload is complete
  • Feature: Added --verify option to zen builds download to verify all files in target folder once entire download is complete

• Bugfix: Fix bug causing crash if large file was exact multiple of 256KB when using zen builds command
• Bugfix: Fix authentication error affecting Jupiter client that manifests as 'bad function call' error UE-252123

• Feature: EXPERIMENTAL New zen builds command to list, upload and download folders to Cloud Build API
  • builds list list available builds (INCOMPLETE - FILTERING MISSING)
  • builds upload upload a folder to Cloud Build API
    • --local-path source folder to upload
    • --create-build creates a new parent build object (using the object id), if omitted a parent build must exist and --build-id must be given
    • --build-id an Oid in hex form for the Build identifier to use - omit to have the id auto generated
    • --build-part-id and Oid in hex form for the Build Part identifier for the folder - omit to have the id auto generated
    • --build-part-name name of the build part - if omitted the name of the leaf folder name give in --local-path
    • --metadata-path path to a json formatted file with meta data information about the build. Meta-data must be provided if --create-build is set
    • --metadata key-value pairs separated by ';' with build meta data for the build. (key1=value1;key2=value2). Meta-data must be provided if --create-build is set
    • --clean ignore any existing blocks of chunk data and upload a fresh set of blocks
    • --allow-multipart enable usage of multi-part http upload requests
    • --manifest-path path to text file listing files to include in upload. Exclude to upload everything in --local-path
  • builds download download a folder from Cloud Build API (INCOMPLETE - WILL WIPE UNTRACKED DATA FROM TARGET FOLDER)
    • --local-path target folder to download to
    • --build-id an Oid in hex form for the Build identifier to use
    • --build-part-id a comma separated list of Oid in hex for the build part identifier(s) to download - mutually exclusive to --build-part-name
    • --build-part-name a comma separated list of names for the build part(s) to download - if omitted the name of the leaf folder name give in --local-path
    • --clean deletes all data in target folder before downloading (NON-CLEAN IS NOT IMPLEMENTED YET)
    • --allow-multipart enable usage of multi-part http download reqeusts
  • builds diff download a folder from Cloud Build API
    • --local-path target folder to download to
    • --compare-path folder to compare target with
    • --only-chunked compare only files that would be chunked
  • builds fetch-blob fetch and validate a blob from remote store
    • --build-id an Oid in hex form for the Build identifier to use
    • --blob-hash an IoHash in hex form identifying the blob to download
  • builds validate part fetch a build part and validate all referenced attachments
    • --build-id an Oid in hex form for the Build identifier to use
    • --build-part-id an Oid in hex for the build part identifier to validate - mutually exclusive to --build-part-name
    • --build-part-name a name for the build part to validate - mutually exclusive to --build-part-id
  • builds test a series of operation that uploads, downloads and test various aspects of incremental operations
    • --local-path source folder to upload
  • Options for Cloud Build API remote store (list, upload, download, fetch-blob, validate-part)
    • --url Cloud Builds URL
    • --assume-http2 assume that the builds endpoint is a HTTP/2 endpoint skipping HTTP/1.1 upgrade handshake
    • --namespace Builds Storage namespace
    • --bucket Builds Storage bucket
    • Authentication options for Cloud Build API
      • Auth token
        • --access-token http auth Cloud Storage access token
        • --access-token-env name of environment variable that holds the Http auth Cloud Storage access token
        • --access-token-path path to json file that holds the Http auth Cloud Storage access token
      • OpenId authentication
        • --openid-provider-name Open ID provider name
        • --openid-provider-url Open ID provider url
        • --openid-client-idOpen ID client id
        • --openid-refresh-token Open ID refresh token
        • --encryption-aes-key 256 bit AES encryption key for storing OpenID credentials
        • --encryption-aes-iv 128 bit AES encryption initialization vector for storing OpenID credentials
      • OAuth authentication
        • --oauth-url OAuth provier url
        • --oauth-clientid OAuth client id
        • --oauth-clientsecret OAuth client secret
  • Options for file based remote store used for for testing purposes (list, upload, download, fetch-blob, validate-part, test)
    • --storage-path path to folder to store builds data
    • --json-metadata enable json output in store for all compact binary objects (off by default)
  • Output options for all builds commands
    • --plain-progress use plain line-by-line progress output
    • --verbose

• Bugfix: Fix parsing of workspace options in Lua config
• Bugfix: Add missing Lua option for option --gc-projectstore-duration-seconds
• Bugfix: Add missing Lua mapping option to --statsd command line option
• Bugfix: Verify that chunking is allowed before chunking loose files duriung oplog export
• Bugfix: Fix oplog target url for oplog export to remote zenserver
• Bugfix: Handle workspace share paths enclosed in quotes and ending with backslash UE-231677
• Bugfix: Strip leading path separator when creating workspace shares

• Improvement: Batch fetch record attachments when appropriate
• Improvement: Reduce memory buffer allocation in BlockStore::IterateBlock
• Improvement: Tweaked BlockStore::IterateBlock logic when to use threaded work (at least 4 chunks requested)
• Improvement: Remove overhead of verifying oplog presence on disk for "getchunks" rpc call in project store
• Improvement: Increase limit where we switch from simple read of IoBuffer file to mmap
• Bugfix: CasContainerStrategy::IterateChunks could give wrong payload/index when requesting 1 or 2 chunks
• Bugfix: Suppress progress report callback if oplog import detects oplog with zero ops

• Feature: Project store "getchunks" rpc call /prj/{project}/oplog/{log}/rpc extended to accept both CAS (RawHash) and Id (Oid) identifiers as well as partial ranges
  • Legacy call still has and chunks array in the request body with IoHash entries providing CAS data for whole chunks only
  • New call has a top level Request object in request body with the following elements:
    • SkipData: bool, optional, default false
      • If SkipData is set to true we will not include the payload of the chunk, just the information requested for the chunk
    • Chunks: array of objects for the requests
      • RawHash: IoHash
        • Indicates we want a Cid chunk which is stored in CAS and identified by the RawHash of the chunk.
        • Mutually exclusive with Oid.
        • The value of the field will be used as the Id field in the chunk response.
      • Oid: IoChunkId
        • Indicates we want a data chunk via its Id. May be stored in CAS or be a reference to a loose file on disk.
        • Mutually exclusive with RawHash.
        • The value of the field will be used as the Id field in the chunk response.
      • Offset: uint64, optional
        • Partial request offset into the requested chunk. Default is from start of chunk (0).
      • Size: uint64, optional
        • Partial request size of the requested chunk. Default is to end of chunk (-1).
      • ModTag: uint64, optional
        • A tag indicating the modification version of the chunk.
        • If SkipData is set to false and the ModTag matches the current modification version in zenserver the chunk response will only contain an Id field.
        • If SkipData is set to false and the ModTag does not match the current modification version in zenserver the response will contains an Id field, current ModTag and the chunk payload.
        • If SkipData is set to true and the ModTag matches the current modification version in zenserver the chunk response will only contain an Id field.
        • If SkipData is set to true and the ModTag does not match the current modification version in zenserver the response will contains an Id field and the current ModTag.
  • New call responds with a zen Http CbPackage (with the header magic 0xaa77aacc) with the following elements in the included package object:
    • Chunks: array of objects for the response
      • Id: IoHash or IoChunkId
        • This is the identifier used for the chunk - if the requested chunk can not be found by zenserver the chunk will be omitted from the reponse Chunks array.
      • Error: string
        • If the chunk was found but there was an error processing the chunk to fulfill the request the reason will be added as a descriptive string.
        • Mutually exclusive with ModTag, Hash, RawHash and FragmentHash.
      • ModTag: uint64
        • This field is set if the ModTag for the chunk was not present in the request or the current ModTag for the chunk does not match the ModTag in the request.
        • Mutually exclusive with Error.
      • Hash: IoHash
        • If the response payload was found and SkipData is false and ModTag in the request does not match the current ModTag this is the identifier for the attachment data.
        • A Hash field indicates that the response for the chunk is attached as uncompressed data.
        • If a range is given for the chunk the attached data is limited to the requested range.
        • Mutually exclusive with RawHash, FragmentHash amd Error.
      • RawHash: IoHash
        • If the response payload was found and SkipData is false and ModTag in the request does not match the current ModTag this is the identifier for the attachment data.
        • A RawHash field indicates that the response for the chunk is attached as compressed data.
        • If a range is given for the chunk the this field will be replaced with a FragmentHash field.
        • Mutually exclusive with Hash, FragmentHash and Error.
      • FragmentHash: IoHash
        • If the response payload was found and SkipData is false and ModTag in the request does not match the current ModTag this is the identifier for the attachment data.
        • A FragmentHash field indicates that the response for the chunk is a partial chunk as compressed data. Compressed data ranges will always be sent as full compressed blocks covering the requested range.
        • If FragmentHash is present, a FragmentOffset field will also be present.
        • Mutually exclusive with Hash, RawHash and Error.
      • FragmentOffset: uint64
        • Indicates at which offset the partial chunk of compressed data starts. Compressed data is compressed into blocks and ranges for compressed data will always send full compressed blocks.
        • FragmentOffset will be less or equal to the requested Offset.
        • FragmentOffset plus the decompressed attachment size will always be more or equal to the requested Offset + Size.
        • Only present if FragmentHash is present.
      • Size: uint64
        • If the uncompressed chunk response is not the full chunk the Size field will contain the chunks full size
        • Mutually exclusive with RawHash and Error.
      • RawSize: uint64
        • If the compressed chunk response is not the full chunk the RawSize field will contain the chunk full decompressed size
        • Mutually exclusive with RawHash, Hash and Error.
• Feature: zen command oplog-export and oplog-import now supports --builds remote target using the Jupiter builds API
  • oplog-export:
    • builds Jupiter builds api service endpoint to export to
    • namespace Name of namespace to store data to
    • bucket Name of bucket to store data to
    • builds-id Key to the stored oplog container as an Object-Id in hex string format, optional - if omitted a new Object Id will be generated
    • builds-metadata-path Path to json file that holds the metadata for the build
    • builds-metadata Key-value pairs separated by ';' with build meta data. (key1=value1;key2=value2)
    • openid-provider Optional name of openid provider used to authenticate with, requires that the zen server instance has been provided with a oids refresh token for the provider name
    • access-token Optional JWT access token to authenticate with
    • access-token-env Optional name of environment variable that holds an JWT access token to authenticate with
    • access-token-path Read cloud access token from json file produced by OidcToken.exe
    • disableblocks Disable block creation and save all attachments individually
    • disabletempblocks Disable temp block creation and upload blocks without waiting for oplog container to be uploaded
  • oplog-import:
    • builds Jupiter builds api service endpoint to export import from
    • namespace Name of namespace to read data from
    • bucket Name of bucket to read data from
    • builds-id Key to the stored oplog container as an Object-Id in hex string format
    • openid-provider Optional name of openid provider used to authenticate with, requires that the zen server instance has been provided with a oids refresh token for the provider name
    • access-token Optional JWT access token to authenticate with
    • access-token-env - Optional name of environment variable that holds an JWT access token to authenticate with
    • access-token-path Read cloud access token from json file produced by OidcToken.exe
• Improvement: Build release binaries with LTO on Windows/Mac
• Improvement: Release binaries now build with "faster" instead of "smaller" optimization flags
• Improvement: Improved payload validation in HttpClient and oplog import UE-230990
• Improvement: Self-hosted frontend;
  • Renamed "view" action to "list" in project's oplog list
  • Downloaded items are suffixed with the oplog entry's file name
  • Better styling of tables with a single column
• Improvement: Fixed threading and lifetime issues in AuthMgr
• Improvement: Adding an existing OpenId provider now preplaces any existing one if the parameters differ
• Bugfix: Reduce background job description name for oplog import/export FORT-829408
• Bugfix: Properly handle trailing path separators in zen workspace and zen workspace-share commands UE-231677
• Bugfix: Long oplog entry keys could cause infinite loops in xxhash due to API mis-use. A workaround is in place which retains the old key mapping to maintain backwards compatibility with on-disk state

• Bugfix: Fix returned content type when requesting a project store chunk with non-compressed accept type
• Bugfix: Fix path parsing using stale memory pointers when reading oplog index file UE-231759
• Bugfix: Ensure we don't throw exceptions from worker threads when doing oplog upload/download operations UE-225400
• Bugfix: Don't add workspace to config.json if the provided file system path does not exist UE-231801
• Improvement: Increase the smallest chunk size in block store to do batch reading for
• Improvement: Block compact of oplogs from performance sensitive calls

• Feature: Added --malloc option for selecting which memory allocator should be used. Currently the available options are --malloc=mimalloc, --malloc=rpmalloc, --malloc=stomp or --malloc=ansi. Some of these options are currently not available on all targets, but all are available on Windows. rpmalloc is currently not supported on Linux or Mac due to toolchain limitations.
• Feature: Added support for generating Unreal Insights-compatible traces for memory usage analysis. Currently only supported for Windows. Activate memory tracing by passing --trace=memory on the command line, alongside one of --tracehost=<ip> or --tracefile=<path> to enable tracing over network or to a file.
• Feature: Self-hosted dashboard displays oplog entry package data sizes in tree and entry view pages
• Bugfix: Don't add RawSize and Size in ProjectStore::GetProjectFiles response if we can't get the payload
• Bugfix: Use validation of payload size/existance in all chunk fetch operations in file cas
• Bugfix: Fixed deadlock in oplog snapshot
• Bugifx: Validate that we can actually fetch the payload when checking attachments in response to a /prep call
• Bugfix: Add missing exclusive ShardLock during compact in filecas
• Improvement: In project store oplog validate, make sure we can reach all the payloads
• Improvement: Add threading to oplog validate request
• Improvement: For /chunkinfos and /files requests we will not fill in the size or rawsize in the response for chunks that are not found
• Improvement: For /chunkinfos and /files requests we will not fill in the size or rawsize in the response for chunks that are are invalid compressed buffers when rawsize is requested
• Improvement: Tweaked how small chunks are batch-read info memory for different use cases. Project store /chunkinfos call execution time is 2x to 3x times faster
• Improvement: Add warning log when filecas fails to open a file it expects to be present
• Improvement: Httpclient: Add validation of response payloads that are compressed binary - attempt retry if applicable
• Improvement: Httpclient: Add validation of response payloads where response contains a X-Jupiter-IoHash header - attempt retry if applicable

• Bugfix: Fix inconsistencies in filecas due to failing to remove payload file during GC causing "Missing Chunk" errors
• Bugfix: Fixed crash on corrupt attachment block when doing oplog import
• Bugfix: Fixed off-by-one in GetPidStatus (Linux) which could cause spurious errors

• Feature: Added option gc-validation to zenserver that does a check for missing references in all oplog post full GC. Enabled by default.
• Feature: Added option gc-validation to zen gc command to control reference validation. Enabled by default.
• Feature: Self-hosted dashboard:
  • Searchable oplog
  • Oplog tree view
  • Added links to an oplog entry's dependencies
• Improvement: Added more details in post GC log.
• Bugfix: Fixed race condition in oplog writes which could cause used attachments to be incorrectly removed by GC
• Bugfix: fixed issue with ZenServerInstance::SpawnServer. Previously it would assign a child identifier twice, which could lead to confusing log output when running tests
• Bugfix: Fixed batch request not handling missing chunks correctly
• Bugfix: Fixed CorrelationId in oplog batch chunk fetch
• Bugfix: Fixed issue which could lead to inconsistent timestamp output within the first second of execution

• Bugfix: Fixed memory leak in cache (would leak GetBatchHandle instances)

• Improvement: Provide shorter project store name in list when not conflicting with other project names
• Improvement: zen project-details is now zen project-op-details and uses the standard resolving logic for project id and oplog id
• Improvement: If there is only one project in the project store, auto-resolve to that if no project id is given
• Improvement: Nicer progress bar during oplog import/export
• Improvement: Added progress bar for zen oplog-mirror command
• Improvement: Verify that oplog has not been deleted from disk behind our back
• Bugfix: If zenserver fails to pick up a request for a sponsor process, make sure to clear the slot
• Bugfix: Fix potential hole where an chunk could be lost during GC if the package referenced a chunk that already existed in store
• Bugfix: Make op key and file path matching in zen oplog-mirror case insensitive

• Feature: Added command zen cache-get to fetch a cache value/record or an attachment from a cache record
• Feature: Added interpretation of ReferencedSet op written by the cooker.
  • The ReferencedSet specifies which ops in the oplog were reachable by the most recent incremental cook and should be staged.
• Feature: Added options --bucketsize and --bucketsizes to zen cache-info to get data sizes in cache buckets and attachments
• Improvement: Self-hosted dashboard
  • Oplog entry view is more complete
  • Separate page for inspecting server stats
• Improvement: When batch-fetching cache values/attachments, only use block memory-buffering for items that may be memcached to reduce disk I/O
• Improvement: zen project-drop command defaults to dry-run to prevent accidental drop and provides information on how to actually perform the drop
• Improvement: Use a smaller thread pool during pre-cache phase of GC to reduce memory pressure
• Improvement: Reworked workspace shares to be more secure. Workspaces and workspace shares can only be created using the zen workspace command, the http endpoint is disabled unless zenserver is started with the --workspaces-allow-changes option enabled.
  • Each workspace are now configured via a zenworkspaceconfig.json file in the root of each workspace
  • A workspace can allow shares to be created via the http interface if the workspace is created with the --allow-share-create-from-http option enabled
  • A new http endpoint at /ws - issuing a Get operation will get you a list of workspaces
  • A new http endpoint at /ws/refresh - issuing a Get will make zenserver scan for edits in workspaces and workspace shares
• Bugfix: Parse filenames as utf8 string when mirroring files inside an oplog using zen oplog-mirror and zen vfs
• Bugfix: Properly initialize stats for file/jupiter/zen remote project store - fixes stats log output when exporting and importing oplogs
• Bugfix: Properly calculate month for last gc times
• Cleanup: Removed GCv1 implementation

• Feature: Added option gc-attachment-passes to zenserver
  • Limit the range of unreferenced attachments included in GC check by breaking it into passes. Default is one pass which includes all the attachments.
• Feature: Added options to zen gc to filter which attachments get checked for GC
  • --reference-low attachments with a hash higher than this will not be considered for GCv2
  • --reference-high attachments with a hash lower than this will not be considered for GCv2
• Feature: Added zen cache-gen command to generate large amount of cache data for testing
• Feature: Initial release of a self-hosted HTML dashboard
• Feature: Added --cache-attachment-store and --projectstore-attachment-store options to zen gc command to override settings for attachment meta store during GCv2
• Improvement: Refactored GCv2 to reduce time we block write requests, trading for longer overall GCv2 time
  • Reduces time writes are blocked by ~8 times, going from 6s to 0.7s on a large data set
  • Increases time for GCv2 to execute (without blocking reads/writes) by ~1.7 times, going from 10.4s to 17.6s on a large data set
• Improvement: Cleaned up GCv2 start and stop logs and added identifier to easily find matching start and end of a GCv2 pass in log file
• Improvement: Optimizations to GCv2 leading to 50-100% performance increase when handling large data sets
• Improvement: Added a namespace-qualified RPC endpoint for z$ at /z$/<namespace>/$rpc which may be used to validate RPC requests by URL inspection
• Improvement: Faster memcache trimming
  • Reduce calculations while holding bucket lock for memcache trim analysis to reduce contention
  • When trimming memcache, evict 25% more than required to reduce frequency of trimming
  • When trimming memcache, don't repack memcache data vector, defer that to regular garbage collection
  • When trimming memcache, deallocate memcache buffers when not holding exclusive lock in bucket
• Improvement: Optimized startup performance - reduced startup time by ~30% for large data sets
• Improvement: During block store compact, merge small blocks even if no new unused data is found
• Improvement: Self-hosted dashboard
• Bugfix: Symbol resolution could fail on Windows depending on the process' current working directory
• Bugfix: Don't block disk cache buckets from writing when scanning records for attachment references
• Bugfix: Make sure we clear active write block of block store if we get exception when writing to disk

• Bugfix: Fix race condition in zenserver during batched fetch of small cache chunks (<=1024b) resulting in missing rawhash/rawsize in the response. UE-223703
• Improvement: Don't add batch overhead if we are only going to put once cache value in a request
• Improvement: Removed redundant commands project-delete and oplog-delete. Use already existing project-drop instead.
• Improvement: zen oplog commands project-drop, project-info, oplog-create, oplog-import, oplog-mirror can now help resolve partial project and oplog identifiers
• Improvement: zen oplog-mirror command now has new filter options to control which files are realized to disk: --key for op key, --file for file path matching and --chunk for chunk id matching
• Improvement: Validate oplog log file and wipe state if it is corrupt before attempting to open
• Improvement: project-drop command defaults to --dry-run=true and will only delete the target if --dry-run=false is added to the command line to avoid accidental delete
• Improvement: Gracefully handle oplog being deleted during early phase of GC
• Improvement: Optional paging when iterating oplog entries

• Bugfix: Make sure noexcept functions does not leak exceptions via ASSERT statements which causes crash via abort
• Bugfix: If we fail to move a temporary file into place, try to re-open the file so we clean it up
• Improvement: Added CPU trace scope "Z$::Bucket::PutBatched"
• Improvement: Cleaned up catch of exception in noexcept functions
• Improvement: Removed --owner-pid command line option for zen up - send is as a pass through option after -- instead if needed
• Improvement: Added --show-console option to zen up to control if zenserver should open a visible console window - default is false
• Improvement: Added --show-log option to zen up to force output of zenserver log on launch success - default is false
• Improvmenet: Made zen down more robust
• Improvement: Added detection of zombie processes on Mac/Linux when waiting for process to exit
• Improvement: Don't keep all oplogs open after GC, close them when references are fetched unless they are open by client
• Improvement: Move GC logging in callback functions into "gc" context
• Improvement: Clean up cache bucket log files at startup as we store the matching information in the index snapshot for the bucket
• Feature: Added option --gc-cache-attachment-store which caches referenced attachments in cache records on disk for faster GC - default is false
• Feature: Added option --gc-projectstore-attachment-store which caches referenced attachments in project store oplogs on disk for faster GC - default is false
• Feature: Added project store oplog index snapshots for faster opening of oplog - opening oplogs are roughly 10x faster

• Improvement: Log reasons for failing to read cache bucket sidecar file
• Improvement: If we fail to read access times for a project it now issues a warning instead of error
• Improvement: If a zenserver instance is already using our named mutex - exit with error code instead of reporting error to Sentry
• Improvement: Gracefully handle scenarios where sponsor processes running status can not be checked
• Improvement: Split worker pools into "burst" and "background" to prevent background job from starving client requests that uses thread pools
• Improvement: Add zenserver session id to Sentry context
• Bugfix: If we fail to get information on a cache chunk for a partial chunk request - treat it as a cache miss
• Bugfix: Set last GC time when we skip GC due to low disk space to avoid spam-running GC
• Bugfix: Make sure we lock project and verify directory exists before trying to iterate to find oplogs
• Bugfix: Failure to get size or if size mismatch of a file in filecas now results in overwrite without error report
• Bugfix: Handle "path not found" error when trying to traverse a directory
• Bugfix: Remove bad ASSERT in cache batch get

• Feature: Added new option to zenserver for GC V2
  • --gc-single-threaded GCV2 - force GC operation to run in single-threaded mode to help with debugging, default is off
• Feature: Added new options to zen gc command for GC V2
  • --single-threaded GCV2 - force GC operation to run in single-threaded mode to help with debugging, default is off
• Feature: Project store now compacts oplog container at first open when 50% of storage is unused and during GCv2 when more than 25% is unused
• Feature: Added --detach option to zenserver which controls the call to setsid() on Mac/Linux for better ctrl-c behaviour when running it using xmake run
• Bugfix: Absolute paths and paths going outside the root path for workspace shares are now blocked
• Bugfix: Fix ASSERT that would trigger in GC under certain conditions if source block was empty
• Bugfix: Skip and report invalid configurations for workspaces instead of crashing
• Bugfix: Report back error to http caller if removal of oplog fails
• Bugfix: If a cache bucket value fails validation - don't try to memcache the empty buffer
• Bugfix: Skip chunk in block stores when iterating a block if the location is out of range
• Bugfix: Make sure we use the full max block size when compacting block store and not create a new block for each input block
• Improvement: xmake run zenserver ... now passes in --detach=false to allow subprocesses to be terminated using Ctrl-C (previously, zenserver instances would linger)
• Improvement: zen workspace-share create now resolves relative root paths to absolute paths
• Improvement: Add better output/logging when failing to initialize shared mutex
• Improvement: Validate data when gathering attachments in GCv2
• Improvement: Add file and size of file when reading a iobuffer into memory via ReadFromFileMaybe
• Improvement: Add hardening to gracefully handle malformed oplogs in project store
• Improvement: Catch exceptions in threaded work to avoid uncaught exception errors
• Improvement: Make oplog/project removal more robust
• Improvement: Made LSN number for project store oplog an unsigned 32 bit value at the top layer to increase range
• Improvement: A request for an out of range project store chunk will now result in a "not found" status result
• Improvement: Callstack logs on assert/error is now indented for better readability
• Improvement: Full command line is now logged on startup and added as context information in Sentry reports
• Improvement: Retry writing GC state if it fails to handle transient problems
• Improvement: If GC operation fails we report an error, after that errors are demoted to warnings until we manage to do a successful GC run. This is to reduce spam on Sentry.
• Improvement: Removed stats for CId entries when doing GCv2 without cleaning up CId references
• Improvement: Enabled mimalloc on arm64
• Improvement: Enabled Sentry support on arm64

• Feature: New 'workspaces' service which allows a user to share a local folder via zenserver. A workspace can have mulitple workspace shares and they provie an HTTP API that is compatible with the project oplog HTTP API. Workspaces and shares are preserved between runs. Workspaces feature is disabled by default - enable with --workspaces-enabled option when launching zenserver.
  • New http service endpoint /ws
    • /ws/{workspace_id} - manage workspaces. {workspace_id} is a Object Id
      • PUT - add a workspace. Set {workspace_id} to 000000000000000000000000 to automatically generate id from root_path
        • Parameter: root_path the root local file path of the workspace.
      • GET - get information about a workspace
      • DELETE - removes a workspace (does not affect the files on disk). Remove all shares inside the workspace
    • /ws/{workspace_id}/{share_id} - manage folder shares within a workspaces. {share_id} is a Object Id
      • PUT - add a workspace. Set {share_id} to 000000000000000000000000 to automatically generate id from share_path
        • Parameter: share_path the relative local file path to the workspace root path.
        • Parameter: alias a names alias for the share to be used with the /ws/share/{alias} url
      • GET - get information about a workspace share
      • DELETE -removes a workspace share (does not affect the files on disk)
    • /ws/{workspace_id}/{share_id}/files
      • GET - get the files in the workspace share
        • Parameter: fieldnames is a parameter to filter which fields to include in the response. Default fields are: id, clientpath and serverpath (translates to ?fieldnames=id,clientpath,serverpath). Use ?fieldnames=* to get all the fields.
        • Parameter: filter only applies if fieldnames is not given as a parameter. Use ?filter=client to exclude the serverpath
    • /ws/{workspace_id}/{share_id}/entries
      • GET - get share as oplog entries
        • Parameter: fieldfilter is a parameter to filter which fields to include in the response. Default fields are: id, clientpath, serverpath (translates to ?fieldnames=id,clientpath,serverpath). Use ?fieldfilter=* to get all the fields
        • Parameter: filter only applies if fieldfilter is not given as a parameter. Use ?filter=client to exclude the serverpath
        • Parameter: opkey limits which ops to include, for a workspace share the only valid value for this parameter is file_manifest
    • /ws/{workspace_id}/{share_id}/{chunk}
      • GET - get the content of a file in a file share
        • Parameter: offset the start of the range of the chunk - Defaults to zero
        • Parameter: size the size of the range of the chunk - Defaults to entire size
    • /ws/{workspace_id}/{share_id}/{chunk}/info
      • GET - get properties of the chunk. Currently only includes the size of the chunk.
    • /ws/{workspace_id}/{share_id}/batch
      • POST - do a batch request for multiple chunks in a workspace share
    • /ws/share/{alias}/ - a shortcut to /ws/{workspace_id}/{share_id}/ based endpoints using the alias for a workspace share.
  • New zen workspace {subcommand} command to manage workspaces
    • create - creates a new workspace
      • --root-path - the root local file path of the workspace
      • --workspace - the object id identity of the workspace, omit to automatically generate an id from the --root-path parameter
    • info - gets information about a workspace
      • --workspace - the object id identity of the workspace
    • remove - removes an existing workspace
      • --workspace - the object id identity of the workspace
  • New zen workspace-share {subcommand} command to manage workspace shares
    • create - creates a new workspace
      • --workspace - the object id identity of the workspace to add the share to
      • --root-path - the root local file path of the workspace - if given it will automatically create the workspace before creating the share. If --workspace is omitted, an id will be generated from the --root-path parameter
      • --share-path - the relative local file path to the workspace root path
      • --share - the object id identity of the workspace share, omit to automatically generate an id from the --share-path parameter
      • --alias - a name for the workspace share that can be accessed via the /ws/share/<alias> endpoint
    • info - gets information about a workspace share
      • --workspace - the object id identity of the workspace to add the share to
      • --share - the object id identity of the workspace share
      • --alias - alias name of the share - replaces --workspace and --share options
    • remove - removes an existing workspace share
      • --workspace - the object id identity of the workspace to add the share to
      • --share - the object id identity of the workspace share
      • --alias - alias name of the share - replaces --workspace and --share options
    • files - get a list of files in a workspace share
      • --workspace - the object id identity of the workspace to add the share to
      • --share - the object id identity of the workspace share
      • --alias - alias name of the share - replaces --workspace and --share options
      • --filter - a list of comma separated fields to include in the response
      • --refresh - for a refresh by re-reading the content of the local folder for the share
    • entries - get ops for a workspace share
      • --workspace - the object id identity of the workspace to add the share to
      • --share - the object id identity of the workspace share
      • --alias - alias name of the share - replaces --workspace and --share options
      • --opkey - filter the entries using an opkey
      • --filter - a list of comma separated fields to include in the response
      • --refresh - for a refresh by re-reading the content of the local folder for the share
    • get - get a chunk from a workspace share
      • --workspace - the object id identity of the workspace to add the share to
      • --share - the object id identity of the workspace share
      • --alias - alias name of the share - replaces --workspace and --share options
      • --chunk - the chunk id for the chunk or the share local path for the chunk
      • --offset the start of the range of the chunk - Defaults to zero
      • --size the size of the range of the chunk - Defaults to entire size
    • batch - get multiple chunks from a workspace share
      • --workspace - the object id identity of the workspace to add the share to
      • --share - the object id identity of the workspace share
      • --alias - alias name of the share - replaces --workspace and --share options
      • --chunks - the chunk ids for the chunk or the share local paths for the chunk
• Bugfix: Removed test data added at current folder when running test
• Bugfix: Make sure we monitor and include new project/oplogs created during GCv2
• Bugfix: Make sure we monitor and include new namespaces/cache buckets created during GCv2
• Improvement: Various minor optimizations in cache package formatting
• Improvement: Add batch fetch of cache values in the GetCacheValues request
• Improvement: Use a smaller thread pool for network operations when doing oplog import to reduce risk of NIC/router failure
• Improvement: Medium worker pool now uses a minimum of 2 threads (up from 1)
• Improvement: Don't try to cache process handles on Windows if we don't have a session id

• Bugfix: Don't try to read bytes to validate a compact binary object that is empty
• Bugfix: Don't free fake memory buffer pointer when materializing a zero size file
• Bugfix: Fix crash when iterating chunks in project store causing crash due threading issues
• Bugfix: If we get a request for a partial chunk that can not be fulfilled we warn and treat it as a miss
• Bugfix: Correctly calculate memory view size from Mid/MidInline function if size is not given
• Improvement: Asserts gives an immediate ERROR log entry with callstack and reason
• Improvement: Asserts flushes the log before sending error report to Sentry
• Improvement: Refactored IterateChunks to allow reuse in diskcachelayer and hide public GetBlockFile() function in BlockStore
• Improvement: Don't use "error:" in log messages unless there is an error as Horde CI will pick up that log line and interpret it as an error

• Bugfix: If we get a request for a partial chunk that can not be fulfilled we warn and treat it as a miss
• Bugfix: Correctly calculate memory view size from Mid/MidInline function if size is not given

• Bugfix: Remove extra loop causing GetProjectFiles for project store to find all chunks once for each chunk found
• Bugfix: Don't capture ChunkIndex variable in CasImpl::IterateChunks by reference as it causes crash
• Bugfix: Don't try to respond with zero size partial cache value when partial size is zero
• Improvement: Make FileCasStrategy::IterateChunks (optionally) multithreaded (improves GetProjectFiles performance)
• Improvement: Add batch scope for adding multiple cache values from single request efficently
• Improvement: Use temp file write and move into place for manifest/state files to avoid partial incomplete file writes
• Improvement: Added more validation of data read from cache / cas
• Improvement: We now detect if we are running on Wine and change the default http path to asio since Wine does not implement http.sys properly

• Change: GCv2 is now the default option, use --gc-v2=false to fall back to GCv1
• Bugfix: Correctly calculate size freed/data moved from blocks in GCv2
• Bugfix: Only disable oplog update capture if we have started it
• Bugfix: Harden GCv2 when errors occur and gracefully abort GC operation on error
• Bugfix: Always pre-cache oplog when creating project store GCv2 referencer
• Bugfix: Fix potential deadlock in project store GCv2 referencer
• Bugfix: Added dedicated timer for EnqueueStateExitFlagTimer
• Bugfix: Made log formatter fullformatter output consistent time stamps across threads
• Bugfix: Made Linux/Mac event implementation TSAN clean
• Bugfix: Properly set content type of chunks fetch from CidStore
• Feature: zen up command improvements
  • --port allows you to specify a base port when starting an instance
  • --base-dir allows you to specify a base directory for the zenserver executable if it is not located next to the zen.exe executable
• Feature: zen down
  • --port allows you to specify a base port when shutting down an instance
  • --base-dir allows you to specify a base directory for the zenserver executable if it is not located next to the zen.exe executable
  • --force if regular shutdown fails it tries to find a running zenserver.exe process and terminate it
  • --data-dir to specify a data directory to deduce which zen instance to bring down
  • If it fails to attach to the running server it now waits for it to exit when setting the RequestExit shared memory flag
• Feature: zen status
  • --port filter running zen instances based on port
  • --data-dir filter running zen instances based on information in the data directory
• Feature: zen attach
  • --data-dir to specify a data directory to deduce which zen instance to attach to
• Improvement: zenserver now checks the RequestExit flag in the shared memory and exist gracefully if it is set
• Improvement: When adding a sponsor process to a running zenserver instance, we wait for it to be picked up from the shared memory section to determine success/fail
• Improvement: Reduced details in remote store stats for oplog export/import to user
• Improvement: Transfer speed for oplog export/import is now an overall number rather than average of speed per single request
• Improvement: add validation of input buffer size when trying to parse package message
• Improvement: avoid doing memcopy when parsing package message
• Improvement: Detect zombie processes on Mac/Linux when checking for running processes
• Imrpovement: Make sure zenserver detaches itself as a child process at startup to avoid zombie process if parent process does not wait for zenserver child process
• Improvement: Trying to load a compact binary object from an empty file no longer causes access violation
• Improvement: When importing oplogs we now import all attachments first and (optionally clean) write the oplog on success to avoid invalid import results
• Improvement: Capture launched zenserver output to display to user if launch fails
• Improvement: Add disk buffering in http client (improves download speed for oplog import)
• Improvement: Add block hash verification for blocks received at oplog import
• Improvement: Offload block decoding and chunk writing form network worker pool threads (improves download speed for oplog import)
• Improvement: Add batching when writing multiple small chunks to block store - decreases I/O load significantly on oplog import
• Improvement: Reworked GetChunkInfos in oplog store to reduce disk thrashing and improve performance
• Improvement: Bumped xmake to 2.9.1 and vcpkg version to 2024.03.25
• Improvement: Refactor IoHash::HashBuffer and BLAKE3::HashBuffer to not use memory mapped files. Performs better and saves ~10% of oplog export time on CI
• Improvement: Add IterateChunks(std::span) for better performance in get oplog

• Bugfix: Get raw size for compressed chunks correctly for /prj/{project}/oplog/{log}/chunkinfos
• Bugfix: Fix log of Success/Failure for oplog import
• Bugfix: Use proper API when checking oplog export blob existence in Jupiter
• Improvement: It is now possible to control which fields to include in /prj/{project}/oplog/{log}/chunkinfos request by adding a comma delimited list of filed names for fieldnames parameter
  • Default fields are: id, rawhash and rawsize (translates to ?fieldnames=id,rawhash,rawsize)
  • Use ?fieldnames=* to get all the fields
• Improvement: It is now possible to control which fields to include in /prj/{project}/oplog/{log}/files request by adding a comma delimited list of filed names for fieldnames parameter
  • Default fields are: id, clientpath and serverpath (translates to ?fieldnames=id,clientpath,serverpath), filter=client only applies if fieldnames is not given as a parameter
  • Use ?fieldnames=* to get all the fields
• Improvement: Use multithreading to fetch size/rawsize of entries in /prj/{project}/oplog/{log}/chunkinfos and /prj/{project}/oplog/{log}/files
• Improvement: Use HttpClient when doing oplog export/import with a zenserver as a remote target. Includes retry logic
• Improvement: Increase the retry count to 4 (5 attempts in total) when talking to Jupiter for oplog export/import
• Improvement: Optimize CompressedBuffer::GetRange() with new CompressedBuffer::ReadHeader() that does one less read from source data resulting in a 30% perf increase.
• Improvement: Validate lengths of chunks fetched from CAS/Cache store, if full chunk can not be retrieved, treat it as missing
• Improvement: Add file and line to ASSERT exceptions
• Improvement: Catch call stack when throwing assert exceptions and log/output call stack at important places to provide more context to caller
• Improvement: Hardening of ParsePackageMessage and added extended details of all malformed attachments detected
• Improvement: Allow import-oplog command to include GC marker path as an argument for when it creates the destination oplog
• Improvement: Oplog export and import now writes out statistics about requests and transfer speeds at end of successful execution

• Bugfix: Fix sentry using wrong folder path when data path contains non-ascii characters UE-210530
• Improvement: Faster reading of compressed buffer headers by not materializing entire source buffer
• Bugfix: Get raw size for compressed chunks correctly for /prj/{project}/oplog/{log}/chunkinfos
• Improvement: It is now possible to control which fields to include in /prj/{project}/oplog/{log}/chunkinfos request by adding a comma delimited list of filed names for fieldnames parameter
  • Default fields are: id, rawhash and rawsize (translates to ?fieldnames=id,rawhash,rawsize)
  • Use ?fieldnames=* to get all the fields
• Improvement: It is now possible to control which fields to include in /prj/{project}/oplog/{log}/files request by adding a comma delimited list of filed names for fieldnames parameter
  • Default fields are: id, clientpath and serverpath (translates to ?fieldnames=id,clientpath,serverpath), filter=client only applies if fieldnames is not given as a parameter
  • Use ?fieldnames=* to get all the fields
• Improvement: Use multithreading to fetch size/rawsize of entries in /prj/{project}/oplog/{log}/chunkinfos and /prj/{project}/oplog/{log}/files
• Improvement: Add GetMediumWorkerPool() in addition to LargeWorkerPool() and SmallWorkerPool()

• Bugfix: Shared memory for zenserver state may hang around after all zenserver processes exit - make sure we find a valid entry in zen up before bailing
• Bugfix: Httpasio only call listen() once
• Bugfix: Make sure exception do not leak out of async (worker thread pool) work and make sure we always wait for completion of all work
• Bugfix: Limit number of headers parsed to 127 as that is the maximum supported by Zen
• Bugfix: Don't capture for loop variables by reference when executing async code
• Bugfix: Make sure WriteFile() does not leave incomplete files
• Bugfix: Use TemporaryFile and MoveTemporaryIntoPlace to avoid leaving partial files on error
• Bugfix: Install Ctrl+C handler earlier when doing zen oplog-export and zen oplog-export to properly cancel jobs
• Bugfix: Fix startup issues where data path contains non-ascii characters UE-210530
• Feature: Added option --access-token-path to zen oplog-export and zen-oplog-import enabling it to read a cloud access token from a json file
• Feature: Added support for generating yaml format responses via Accept: yaml or .yaml suffix
• Improvement: Add ability to block a set of CAS entries from GC in project store
• Improvement: Httpasio explicitly close acceptor sockets
• Improvement: Httpasio add retry for desired port
• Improvement: Move structuredcachestore tests to zenstore-test
• Improvement: CompositeBuffer::Mid no longer materializes segment buffers
• Improvement: Don't materialize entire files when hashing if it is a large file
• Improvement: Added context to some http.sys warnings caused by HTTP API error returns
• Improvement: Improved logging for block store GCV2 operations
• Improvement: Added more tests for GCV2 (added GCV2 versions of existing GCV2 tests)
• Improvement: Add disk cache to reading and writing blocks when moving data in GCV2
• Improvement: Cleaned up some asio server state machine details (minor)
• Improvement: Added support for request tracing when using asio path (use --log-trace=http_requests to enable)
• Improvement: Large attachments and loose files are now split into smaller chunks and stored in blocks during oplog export
• Improvement: Make sure zenserver reacts and exist on SIGTERM signal
• Improvement: Retry to create the .lock file at startup to avoid failing launch due to race condition with UE
• Improvement: Add CompressedBuffer::GetRange that references source data rather than make a memory copy
• Improvement: Delay exiting due to no sponsor processes by one second to handle race conditions
• Improvement: Safer IsProcessRunning check
• Improvement: Make sure we can RequestApplicationExit safely from any thread
• Improvement: Check if a block exists in remote store before considering it for reuse in oplog export
• Improvement: Add limit to the number of times we attempt to finalize and exported oplog
• Improvement: Switch to large thread pool when executing oplog export/import
• Improvement: Clean up reporting of missing attachments in oplog export/import
• Improvement: Remove double-reporting of abort reason for oplog export/import
• Improvement: Add support to filter projectstore entries request using the fieldfilter where the wanted fields are comma (,) delimited
• Improvement: Add support for responding with compressed payloads for projectstore entries requests by adding AcceptType compressed-binary to the request header
• Improvement: Add support for responding with compressed payloads for projectstore files requests by adding AcceptType compressed-binary to the request header
• Improvement: Add support for responding with compressed payloads for projectstore chunkinfo requests by adding AcceptType compressed-binary to the request header
• Removed: --cache-reference-cache-enabled option has been removed along with the implementation for reference caching in disk cache

• Feature: Added --copy-log, --copy-cache-log and copy-http-log option to zen logs command to copy logs from a local running zenserver instance
• Improvement: More details in oplog import/export logs
• Improvement: Switch from Download to Get when fetching Refs from Jupiter as they can't be resumed anyway and streaming to disk is redundant
• Improvement: Return system error code on exception throw in zen command
• Improvement: Clean up HttpClient::Response::ErrorMessage to remove redundant ": "
• Improvement: respond with BadRequest result instead of throwing exception and causing a Sentry report on bad request input
• Improvement: Speed up oplog export by fetching/compressing big attachments on demand
• Improvement: Speed up oplog export by batch-fetcing small attachments
• Improvement: Speed up oplog import by batching writes of oplog ops
• Improvement: Tweak oplog export default block size and embed size limit
• Improvement: Add more messaging and progress during oplog import/export
• Improvement: Large loose file attachments are now saved to temp files after compressing during oplog export to reduce memory pressure
• Improvement: Keep track of added ops during GCV2 instead of rescanning full oplog when added ops are detected
• Bugfix: Make sure we clear read callback when doing Put in HttpClient to avoid timeout due to not sending data when reusing sessions
• Bugfix: Respect --ignore-missing-attachments in oplog-export command when loose file is missing on disk
• Bugfix: Only try to traverse an objectstore bucket if it really exists
• Bugfix: Actually throw exception if we can't parse the JobId when starting async job
• Bugfix: Implement two listening sockets in ASIO (ipv4+ipv6) when either we start with --http-forceloopback or we resort to that mode because of a failure to bind to the "any" address

• Improvement: Add details when reading from BasicFile fails (number of bytes read is not expected size)
• Bugfix: No longer outputs illegal characters when Jupiter responds with an error payload in compact binary format

• Feature: add --ignore-missing-attachments to oplog-import command
• Feature: add --ignore-missing-attachments to oplog-export command
• Improvement: Removed use of in stats, for better performance (runtime as well as build)
• Improvement: Separated cache RPC handling code from general structured cache HTTP code
• Improvement: Get more detailed information on Jupiter upstream errors
• Improvement: Improved performance when saving oplog via oplog import command
• Improvement: Add more feedback and progress information when executing oplog import/export
• Improvement: Refactored Jupiter upstream to use HttpClient
• Improvement: Added retry and resume logic to HttpClient
• Improvement: Added authentication support to HttpClient
• Improvement: Clearer logging in GCV2 compact of FileCas/BlockStore
• Improvement: Size details in oplog import logging
• Improvement: Reduce oplog block size to 64MB to reduce amount of redundant chunks to download
• Bugfix: RPC recording would not release memory as early as intended which resulted in memory buildup during long recording sessions. Previously certain memory was only released when recording stopped, now it gets released immediately when a segment is complete and written to disk.
• Bugfix: File log format now contains dates again (PR #631)
• Bugfix: Jobqueue - Allow multiple threads to report progress/messages (oplog import/export)
• Bugfix: Jobqueue - Add AbortReason and properly propagate error when running async command (oplog import/export)
• Bugfix: Make sure to write the correct data in BasicFileWriter when writing items that are not a multiple of the buffer size
• Bugfix: Restructured RPC recorder to use a MPSC queue instead of shared datastructures to fix a data race condition

• Bugfix: Cache RPC recording would drop data when it reached 4GB of inline chunk data in a segment
• Bugfix: Fixed thread safety issues in RPC recorder v2
• Bugfix: IoBuffer::Materialize would leak memory for small buffers
• Bugfix: Fix crash bug when trying to inspect non-open block file in GC
• Bugfix: Fixed up code so we can build everything even when trace support is disabled
• Bugfix: Make sure we initialize the pattern of FileSink before it is added as a usable logger
• Bugfix: Fixed capture of loop-local variables in lambdas for GCv2
• Bugfix: Various minor TSAN/ASAN fixes (see PR #622)
• Improvement: Cache RPC replay can now process partial recordings by recovering metadata from available files
• Improvement: Cache RPC recording now limits duration of individual segments to 1h
• Improvement: Made RPC replay command line parsing more robust by ensuring at least one processing thread is in use
• Improvement: Windows executables are now signed with official cert when creating a release
• Improvement: Each block in block store that is rewritten will now be logged for better feedback

• Bugfix: ShutdownLogging code would throw an exception if it was called before everything had been initialised properly
• Bugfix: Reorder shutdown to avoid crash due to late async log messages (spdlog workaround)
• Bugfix: Correctly calculate peak disk write size in GC status message
• Bugfix: Skip invalid chunks in block store GC when moving existing chunks
• Bugfix: Don't use copy of Payloads array when fetching memcached payload in GC
• Bugfix: Make sure IoBuffer is a valid null-buffer after move operation
• Improvement: Adjusted and added some trace scopes

• Feature: Added xmake task updatefrontend which updates the zip file containing the frontend html (/src/zenserver/frontend/html.zip)
• Feature: Added --powercycle option to zenserver which causes it do shut down immediately after initialization is completed. This is useful for profiling startup/shutdown primarily but could also be useful for some kinds of validation/state upgrade scenarios
• Feature: New endpoint /admin/gc-stop to cancel a running garbage collect operation
• Feature: Added zen gc-stop command to cancel a running garbage collect operation
• Feature: Added ability to configure logger verbosity on the command line. You can now use --log-debug=http_requests to configure the http_requests logger to DEBUG level. The provided options are --log-trace, --log-debug, --log-info, --log-warn, --log-error, --log-critical, --log-off and each accepts a comma-separated list of logger names to apply the threshold to.
• Bugfix: Fix sentry host name where last character of name was being truncated
• Bugfix: GCv2 - make sure to discover all projects and oplogs before checking for expired data
• Bugfix: Fix sync of log position and state log when writing cas index snapshot
• Bugfix: Make sure we can override flags to "false" when running zen gc commmand
  • smallobjects, skipcid, skipdelete, verbose
• Bugfix: fixed file log timestamp format so the milliseconds are appended after the time not the date
• Bugfix: Shut down thread pools earlier so worker threads have a chance to terminate before main thread calls atexit()
• Bugfix: Use correct lookup index when checking for memcached buffer when finding references in diskcache GC
• Bugfix: CasContainerStrategy::ReadIndexFile issue could cause CAS items to not be found after a shutdown/restart cycle
• Bugfix: Make sure we don't hold the namespace bucket lock when we create buckets to avoid deadlock
• Bugfix: Make sure that PathFromHandle don't hide true error when throwing exceptions
• Bugfix: Allow attachments that contains a raw size of zero
• Improvement: The frontend html content is no longer appended at the end of the executable which prevented signing, instead it is compiled in from the /src/zenserver/frontend/html.zip archive
• Improvement: MacOS now does ad-hoc code signing by default when issuing xmake bundle, signing with proper cert is done on CI builds
• Improvement: Updated branding to be consistent with current working name ("Unreal Zen Storage Server" etc)
• Improvement: GcScheduler will now cancel any running GC when it shuts down.
  • Current GC is rather limited in when it reacts to cancel of GC. GCv2 is more responsive.
• Improvement: Cache metadata snapshot (*.uidx/.zen_manifest and now *.meta) files are read and written in a streaming fashion instead of all-at-once to/from memory like before. This eliminates some spiky memory usage patterns during garbage collection and also improves overall performance considerably.
• Improvement: zen copy-state command now utilizes block cloning where possible (i.e on ReFS volumes) for near-instant snapshots
• Improvement: Added more trace scopes for GCv2
• Improvement: Use two global worker thread pools instead of ad-hoc creation of worker pools
• Improvement: GCv2: Use separate PreCache step to improve concurrency when checking references
• Improvement: GCv2: Improved verbose logging
• Improvement: GCv2: Sort chunks to read by block/offset when finding references
• Improvement: GCv2: Exit as soon as no more unreferenced items are left
• Improvement: Reduce memory usage in GC and diskbucket flush
• Improvement: Added command line to trace initialization (Windows only for now)
• Improvement: Added a {project}/oplog/{log}/chunkinfos endpoint that can be used for getting all chunk info within an oplog in batch
• Improvement: Reserve vector sizes in GCv2 to reduce reallocations
• Improvement: Set min/max load factor for cachedisk/compactcas/filecas indexes to reduce memory footprint
• Improvement: Added context (upstream host name) to Zen upstream resolve error message
• Improvement: Make a more accurate estimation of memory usage for in-memory cache values
• Improvement: Added detailed debug logging for pluggable transports
• Improvement: Improved formatting of multi-line logging. Each line is now indented to line up with the initial line to make reading the output easier
• Improvement: Refactor memory cache for faster trimming and correct trim reporting
• Improvement: Added trace scopes for memory cache trimming
• Improvement: Pass lock scope to helper functions to clarify locking rules
• Improvement: Block flush and gc operations for a bucket that is not yet initialized
• Improvement: Add ZenCacheDiskLayer::GetOrCreateBucket to avoid code duplication
• Improvement: Scrub operation now validates compressed buffer hashes in filecas storage (used for large chunks)
• Improvement: Added --dry, --no-gc and --no-cas options to zen scrub command
• Improvement: Implemented oplog scrubbing (previously was a no-op)
• Improvement: Implemented support for running scrubbint at startup with --scrub=

• Bugfix: Fix timeout calculation for semtimedop call
• Bugfix: Fix NameEvent test to avoid race condition
• Bugfix: Fix BlockingQueue asserts
• Bugfix: Catch exceptions in WorkerThreadPool when running single-threaded
• Bugfix: Improved block cloning copy argument validation, to properly catching the case where source or target trees overlap
• Feature: Adding a file named root_manifest.ignore_schema_mismatch in the root of zenserver data dir prevents wipe of data when schema mismatches
• Feature: Added zen run command which can be used to run a stress test or benchmark repeatedly while redirecting output and other state to separate subdirectories
  • Example usage: zen run -n 10 -- zenserver-test will run the zenserver-test command 10 times
  • Example usage: zen run -n 10 -- zenserver-test --ts=core.assert will` run zenserver-test 10 times (testing only the core.assert test suite)
  • Example usage: zen run --time 600 --basepath=d:\test_dir\test1 -- zenserver-test keeps spawning new instances for 10 minutes (600 seconds). Each run will execute in a separate subdirectory in d:\test_dir\test1\ where stdout will be captured alongside any data generated by the executed command
• Feature: Added new options to zenserver for GC V2
  • --gc-compactblock-threshold GCV2 - how much of a compact block should be used to skip compacting the block, default is 90%
  • --gc-verbose GCV2 - enable more verbose output when running a GC pass
• Feature: Added new options to zen gc command for GC V2
  • --compactblockthreshold GCV2 - how much of a compact block should be used to skip compacting the block, default is 90%
  • --verbose GCV2 - enable more verbose output when running a GC pass
• Feature: Added new parameters for endpoint admin/gc (PUT)
  • compactblockthreshold GCV2 - how much of a compact block should be used to skip compacting the block, default is 90%
  • verbose GCV2 - enable more verbose output when running a GC pass
• Improvement: Removed zen runtests command since it's no longer useful
• Improvement: Simplified zenserver-test code by implementing dynamic port assignment, and also implemented transparent handling of port relocation for increased test robustness against environmental differences and socket lifetime 'noise'
• Improvement: Refactor GCV2 so GcReferencer::RemoveExpiredData returns a store compactor, moving out the actual disk work from deleting items in the index.
• Improvement: Refactor GCV2 GcResult to reuse GcCompactStoreStats and GcStats
• Improvement: Make GCV2 Compacting of stores non-parallel to not eat all the disk I/O when running GC
• Improvement: Added ZEN_ASSERT_FORMAT implementation in zencore/assertfmt.h for better logging of errors. Introduced it into compact binary building code which had some existing use cases.

• Bumped zenserver data schema (to '5') to wipe corrupted state caused by version 0.2.31/0.2.32
• Bugfix: Fix hang on ZenServerInstance shutdown on Mac/Linux
• Improvement: Event implementation now uses std::atomic_bool instead of volatile bool for correctness
• Improvement: Removed dependency on cxxopts exception types, to enable use of library versions >3.0.0
• Improvement: Posix event implementation is now more robust and works around apparent condition variable issue
• Improvement: ProcessHandle::Wait no longer returns without calling kill
• Improvement: We now set Sentry to <username>@<hostname> if personal information is allowed to be sent
• Improvement: Make object store endpoint S3 compatible.

• Bugfix: Fix index out of bounds in CacheBucket::CompactState
• Bugfix: Implement != operator for DiskLocation to avoid comparing uninitialized data
• Improvement: Shrink data structures to fit after CacheBucket::CompactReferences

• Feature: Writes a gc.log with settings and detailed result after each GC execution (version 2 only)
• Bugfix: Fix memory stop in disk cache bucket ReadManifest
• Improvement: Package dependency clean-ups

• Feature: New parameter for endpoint admin/gc (GET) details=true which gives details stats on GC operation when using GC V2
• Feature: New options for zen command gc-status
  • --details that enables the detailed output from the last GC operation when using GC V2
• Feature: Add new copy-state zen command which copies a zenserver data directory state excluding bulk data for analysis purposes
• Feature: New garbage collection implementation, still in evaluation mode. Enabled by --gc-v2 command line option
• Feature: zen print command can now print the gc.log file
• Feature: New option for zenserver - --http-forceloopback which forces opening of the server http server using loopback (local) connection only (UE-199776)
• Bugfix: Build script now sets up arch properly when running tests on MacOS
• Bugfix: Corrected initialization of block store MaxBlockCount
• Bugfix: Corrected total disk size usage in block store
• Bugfix: Server log files were using the wrong log line prefix due to a mistake when consolidating logging setup code
• Bugfix: Sponsor processes are now registered synchronously at startup, to close potential race condition in very short-lived subprocesses such as the automated tests
• Bugfix: Fix error in GC when reclaiming disk reserve is not enough to accommodate the new block
• Bugfix: If a directory is deleted while we try to traverse it, skip it and continue
• Improvement: Multithread init and flush of cache bucket for faster startup and exit
• Improvement: Renamed BlockStoreCompactState::AddBlock to BlockStoreCompactState::IncludeBlock for clarity
• Improvement: Added tests for BlockStore::CompactBlocks
• Improvement: Reduce memory consumption in cache disk layer
• Improvement: Refactored logging so that spdlog details are hidden from the majority of client code
• Improvement: Use GC reserve when writing index/manifest for a disk cache bucket when disk is low when available
• Improvement: Demote errors to warning for issues that are not critical and we handle gracefully
• Improvement: Treat more out of memory errors from windows as Out Of Memory errors
• Improvement: Factored out some compiler / platform definitions into standalone zenbase header-only library, along with header-only helpers which can be used standalone. This is intended to support out-of-tree code like pluggable transports etc but also provides more fine grained dependencies
• Improvement: Replaced use of openssl on Windows with bcrypt, which reduces executable size by some 40%
• Improvement: We no longer put cache entries into the memory cache on Put, only on Get
• Improvement: Dedicated servers now have a different heuristic for deciding when to use standalone files to store cache records/values in the structured cache disk layer. This helps performance on heavy-traffic servers
• Improvement: Reduced memory footprint of cache index by 10% or so by limiting the maximum number of entries in a bucket to 2^32 (was 2^64)

• Bugfix: Block sending error reports from sentry_sink to Sentry unless the log is actually an error log
• Bugfix: Make sure we have an exclusive lock in CacheBucket::CollectGarbage when removing standalone entries from the index
• Bugfix: Fixed problem with missing session/request context in cache record PUT operations

• Feature: Add skipdelete parameter to admin/gc endpoint to do a dry run of GC
• Feature: Add --skipdelete option to zen gc command to do a dry run of GC
• Feature: New endpoint /admin/flush to flush all storage - CAS, Cache and ProjectStore
• Feature: New command zen flush to flush all storage - CAS, Cache and ProjectStore
• Feature: Added --cache-memlayer-sizethreshold option to zenserver to control at which size cache entries get cached in memory
• Bugfix: Filter cache keys against set of expired cache keys, not against set of CAS keys to keep
• Bugfix: Fix implementation when claiming GC reserve during GC
• Bugfix: Catch exceptions when processing requests with asio http and log errors
• Improvement: Command zen gc-status now gives details about storage, when last GC occured, how long until next GC etc
• Improvement: New rotating file logger that keeps on running regardless of errors to avoid spamming error reports on OOM or OOD
• Improvement: Removed HttpCidStore (was already deprecated)
• Improvement: Optimized cases in CompactBinary reader where we would measure a variable integer twice
• Changed: Cache access and write log are disabled by default
• Changed: We no longer purge out block location for missing blocks to allow testing/analisys of snapshots of server states without copying full set of data
• Changed: Merged cache memory layer with cache disk layer to reduce memory and cpu overhead

• Feature: Implemented initial and experimental support for pluggable transports via (--http=plugin)
• Feature: Add caching of referenced CId content for structured cache records, this avoid disk thrashing when gathering references for GC
  • disabled by default, enable with --cache-reference-cache-enabled
• Feature: Added --clean command line option which can be used to wipe all server state at startup (useful for testing/benchmarking)
• Feature: Added --dry command line option to zen rpc-record-replay to allow analysis of recordings without making changes to any server
• Changed: The default port for zenserver has been changed from 1337 to 8558
• Bugfix: GC logging now correctly reports used/free disk space in log message
• Bugfix: Fixed calculation of cache memory layer total size
• Improvement: Implemented a new RPC recording strategy (aka v2) to make long-running recordings use less resources and perform better over time
  • Recordings are split into segments to limit the amount of data and number of files in each segment directory. Each segment is independent to allow future disk space management improvements and partial analysis/replay of recordings
  • The new recording strategy also records the client session id for each entry to enable better traffic analysis
• Improvement: Rewrite the state_marker file at startup to make sure we have write access to the data directory
• Improvement: Faster reading of project store oplogs
• Improvement: Faster collection of referenced CId content in project store
• Improvement: Also reject bad bucket GET operations to prevent the buckets from being created on disk
• Improvement: GC will now skip a lightweight GC if a full GC is due to run within the next lightweight GC interval
• Improvement: when dedicated mode is enabled via --dedicated or server.dedicated then we tune http.sys server settings to be more suitable for a shared server. Initially we tune two things:
  • the thread pool used to service I/O requests allows a larger number of threads to be created when needed. The minimum thread count is unchanged but in dedicated server mode we double the maximum number of threads allowed
  • the http.sys request queue length (HttpServerQueueLengthProperty) is increased to 50,000 in dedicated mode. The regular default is 1,000. A larger queue means the server will deal with small intermittent stalls (for example due to GC locking) even at higher loads like 100k req/s without rejecting requests via HTTP 503 results
• Removed: Removed legacy compute interface (will be replaced with new implementation in the future)

• Bugfix: Remove double counting of memory usage in memcachelayer
• Bugfix: Make sure we don't busy loop if Garbage Collection fails
• Bugfix: If we can't check if a project/oplog is expired via the marker file, assume it is not expired
• Improvement: added rejection of known bad buckets in structured cache (caused by inappropriate client use of buckets)
  • buckets consisting of 32 characters in hexadecimal are rejected in put operations
  • buckets are also rejected at bucket discovery time, and any bad directories are deleted
• Improvement: restructured zenserver project to improve maintainability (PR#442)

• Feature: Limit the size ZenCacheMemoryLayer may use
  • --cache-memlayer-targetfootprint option to set which size (in bytes) it should be limited to, zero to have it unbounded
  • --cache-memlayer-maxage option to set how long (in seconds) cache items should be kept in the memory cache
• Feature: Add lightweight GC that only removes items from cache/project store without cleaning up data referenced in Cid store
  • Add skipcid parameter to http endpoint admin/gc, defaults to "false"
  • Add --skipcid option to zen gc command, defaults to false
  • Add --gc-lightweight-interval-seconds option to zenserver
• Bugfix: Correctly calculate the total number of RPC ops in the stats page for structured cache
• Bugfix: Change "chunks" title to "count" for RPC chunk requests in stats page for structured cache
• Bugfix: Sentry username string no longer includes the trailing NUL
• Bugfix: Fix scrub messing up payload and access time in disk cache bucket when compacting index
• Bugfix: Cache bucket index of chunks in block store could get corrupted after a GC
• Bugfix: Probe disk for existsing block file before writing a new block
• Bugfix: IoBufferBuilder::ReadFromFileMaybe did not propagate content type to a new IoBuffer
• Bugfix: IoBufferBuilder::ReadFromFileMaybe Linux/MacOS pread success/error condition check was incorrect
• Bugfix: Memory cache layer could end up holding references to file handles between GC runs, delaying file deletion
• Improvement: Split up disk cache bucket index into hash lookup and payload array to improve performance
• Improvement: Reserve space up front for compact binary output when saving cache bucket manifest to improve performance
• Improvement: Reduce time a cache bucket is locked for write when flushing/garbage collecting
  • Change format for faster read/write and reduced size on disk
  • Don't lock index while writing manifest to disk
  • Skip garbage collect if we are currently in a Flush operation
  • BlockStore::Flush no longer terminates currently writing block
  • Garbage collect references to currently writing block but keep the block as new data may be added
  • Fix BlockStore::Prune used disk space calculation
  • Don't materialize data in filecas when we just need the size
  • Don't lock entire disk cache layer when doing GatherReferences/CollectGarbage
• Improvement: Catch Out Of Memory and Out Of Disk exceptions and report back to requester without reporting an error to Sentry
• Improvement: If creating bucket fails when storing an item in the structured cache, log a warning and propagate error to requester without reporting an error to Sentry
• Improvement: Make an explicit flush of the active block written to in blockstore flush
• Improvement: Make sure cache and cas MakeIndexSnapshot does not throw exception on failure which would cause and abnormal termniation at exit
• Improvement: http.sys I/O completion handler no longer holds transaction lock while enqueueing new requests. This eliminates some lock contention and improves latency/throughput for certain types of requests
• Improvement: date logging in GC no longer emits extraneous newlines
• Improvement: named main thread and thread handle cache thread
• Improvement: removed websockets support as it is not used and likely won't be used in the future
• Improvement: added --quiet command line option which can be used to suppress all stdout logging

• Feature: Add detailed stats on requests and data sizes on a per-bucket level, use parameter cachestorestats=true on the /stats/z$ endpoint to enable
• Feature: Add detailed stats on requests and data sizes on cidstore, use parameter cidstorestats=true on the /stats/z$ endpoint to enable
• Feature: Dashboard now accepts parameters in the URL which is passed on to the /stats/z$ endpoint
• Improvement: GarbageCollect for ZenCacheMemoryLayer now respects --gc-cache-duration-seconds
• Improvement: HttpSys: When a response fails, we now include more information including metadata about the contents of the reponse
• Improvement: Flush current data block to disk when switching to a new block
• Improvement: Handle cache RPCs synchronously instead of dispatching to async worker threads when there is no upstream server
• Improvement: Endpoint for cache upstream stats improved - added active and worker_threads reflects actual number of threads
• Improvement: Cache upstream only starts worker threads once at least one Endpoint is registered
• Improvement: http.sys only starts async work threads when there is async work to do (which is often never if there is no upstream)

• Feature: New endpoint /admin/logs to query status of logging and log file locations and cache logging
  • cacheenablewritelog=true/false parameter to control cache write logging
  • cacheenableaccesslog=true/false parameter to control cache access logging
  • loglevel = trace/debug/info/warning/error
• Feature: New zen command logs to query/control zen logging
  • No arguments gives status of logging and paths to log files
  • --cache-write-log enable/disable to control cache write logging
  • --cache-access-log enable/disable to control cache access logging
  • --loglevel trace/debug/info/warning/error to set debug level
• Feature: Add endpoint for controlling Insights tracing
  • GET /admin/trace to query if tracing is currently running or not
  • POST /admin/trace/start to start tracing
    • host=<tracehostip> start tracing to a trace server at ip <tracehost>
    • file=<filepath> start tracing to file at path <filepath>
  • POST /admin/trace/stop stop the currently running trace
• Feature: Add zen trace command to control Insights tracing
  • zen trace to show the status of tracing ("enabled" or "enabled")
  • zen trace --host=<tracehostip> start tracing to a trace server at ip <tracehost>
  • zen trace --file=<filepath> start tracing to file at path <filepath>
  • zen trace --stop stop the currently running trace
• Feature: Implemented virtual file system (VFS) support for debugging and introspection purposes
  • zen vfs mount <path> will initialize a virtual file system at the specified mount point. The mount point should ideally not exist already as the server can delete the entirety of it at exit or in other situations. Within the mounted tree you will find directories which allow you to enumerate contents of DDC and the project store
  • zen vfs unmount will stop the VFS
  • zen vfs info can be used to check the status of the VFS
• Bugfix: Do controlled shut down order of zenserver and catch exceptions thrown during shutdown
• Bugfix: Make sure we don't throw exceptions when reporting errors to Sentry
• Improvement: Add names to background jobs for easier debugging
• Improvement: Background jobs now temporarily sets thread name to background job name while executing
• Improvement: Background jobs tracks worker thread id used while executing
• Improvement: xmake sln can now be used on Mac as well to generate project files
• Improvement: http.sys request queues are named to make it easier to find performance counters in Performance Monitor and such
• Improvement: http.sys - if request rate is too high then rejected requests will get a 503 response instead of a dropped connection

• Bugfix: Respect result from FinalizeRef in Jupiter oplog upload where it requests missing attachments
• Improvement: Increase timeout when doing import/export of oplogs to jupiter to 30 min per request
• Improvement: Better logging/progress report on oplog export
• Improvement: Ignore OOM errors in spdlog, just drop the error since we can't do anything useful if we run out of memory here
• Improvement: Try to catch any exceptions in spdlog error handling to avoid abort termination of process
• Improvement: Block cache access/write log from writing to log if disk is low on free space

• Bugfix: Under heavy load, an http.sys async response handler could end up deleting the HTTP transaction object before the issuing call had completed. This is now fixed
• Improvement: More tracing scopes in zenserver

• Feature: New http endpoint for background jobs /admin/jobs which will return a response listing the currently active background jobs and their status
• Feature: New http endpoint for background jobs information /admin/jobs/{jobid} which will return a response detailing status, pending messages and progress status
  • GET will return a response detailing status, pending messages and progress status
  • DELETE will mark the job for cancelling and return without waiting for completion
  • If status returned is "Complete" or "Aborted" the jobid will be removed from the server and can not be queried again
• Feature: New zen command jobs to list, get info about and cancel background jobs
  • If no options are given it will display a list of active background jobs
  • --jobid accepts an id (returned from for example oplog-export with --async) and will return a response detailing status, pending messages and progress status for that job
  • --cancel can be added when --jobid is given which will request zenserver to cancel the background job
• Feature: oplog import and export http rpc requests are now async operations that will run in the background
• Feature: oplog-export and oplog-import now reports progress to the console as work progress by default
• Feature: oplog-export and oplog-import can now be cancelled using Ctrl+C
• Feature: oplog-export and oplog-import has a new option --async which will only trigger the work and report a background job id back
• Feature: Incremental oplog export for block-base target (Cloud/File). If a base is given it will download an existing oplog (excluding attachments) and try to reuse existing block references in that oplog.
  • --basename option for file based oplog-export
  • --basekey option for cloud based (Jupiter) oplog-export
• Feature: Added --cache-write-log and --cache-access-log command line option to enable/disable cache write/access logs
• Feature: Added --http-threads, --httpsys-async-work-threads, --httpsys-enable-request-logging and --httpsys-enable-async-response command line options to zenserver
• Feature: More statistics for Cache, Project Store and Cid Store
  • Cache: requestcount, badrequestcount, writes
  • Project Store: requestcount
  • Cid Store: cidhits, cidmisses, cidwrites
• Bugfix: Make sure cache logging thread does not crash on errors
• Bugfix: Make sure error logging or destructors don't throw exception when trying to get file name from handle
• Bugfix: Issue warning instead of assert on bad data in cid store
• Bugfix: Don't index out of string_view range when parsing URI in httpsys
• Improvement: Sorting attachments in oplog blocks based on Op key to group op attachments together
• Improvement: Don't split attachments associated with the same op across oplog blocks
• Improvement: 25% faster oplog op reading, only read and parse op data of latest op for particular key speeding up reading of oplog with old oplog data

• Feature: zen up command has two new command line options
  • --config <file_path> tells zenserver to start with a specific config file
  • --owner-pid <pid> tells zenserver to start with a owning process id
• Feature: zen attach command to add additional owning processes to a running zenserver instance
  • --owner-pid <pid> adds pid to running zenserver instance list of owning processes
• Feature: --write-config command line option for zenserver
  • --write-config <file_path> path to a file which will contain a lua config file for zenserver combining all command line options and optional lua config files
• Bugfix: Only write disk usage log if disk writes are allowed (disk space is not cirtically low)
• Improvement: zen up command will check if zenserver is currently running before starting up a new instance
• Improvement: Add retry logic when creating oplog temp files and cas block files
• Improvement: Large attachments fetched from Jupiter while doing oplog-import now streams to disk and are moved in place

• Bugfix: Fix deadlock in project store garbage collection
• Bugfix: Fix zen command executable not being able to access shared status memory (zen status, zen down etc fails) on MacOs
• Bugfix: All options given on command line now overrides lua config file settings
• Improvement: All options available from command line can now be configured in the lua config file (with a few exceptions such as owner-pid, install and uninstall)

• Feature: Add --embedloosefiles option to oplog-export which adds loose files to the export, removing need to call oplog-snapshot
• Bugfix: Fix construction order in OpenProcessCache to avoid crash in OpenProcessCache::GcWorker
• Bugfix: Retain ServerPath in oplog when performing oplog-snapshot. This is a short-term fix for current incompatibility with the UE cooker.
• Bugfix: Fix OpenProcessCache state error causing assert/error
• Bugfix: Make sure to reset cache logging worker thread event to avoid busy-looping looking for more work
• Improvement: Make sure we have disk space available to do GC and use reserve up front if need be
• Improvement: We now build the Linux target using the UE toolchain to be compliant with the VFX platform that UE uses for Linux.

• Feature: Add oplog-mirror command to Zen command line tool. It can be used to export the contents of an oplog as files. Currently it will export all files, filtering options will be added at a later time
• Feature: Add --force-update option to Zen command line tool project-create to update or create a project store project. It will update meta information about the project without affecting existing oplogs if project exists.
• Feature: Add --force-update option to Zen command line tool oplog-update to update or create a project store oplog. It will update meta information about the project without affecting existing oplog data if oplog exists.
• Feature: Zen command line tool project-delete to delete a project store project and all its oplogs.
  • --project Project name (id)
• Feature: Zen command line tool oplog-delete to delete a project store oplog and the oplog data.
  • --project Project name (id)
  • --oplog Oplog name (id)
• Bugfix: Make sure to check oplog op attachments when gathering references for GC
• Bugfix: Reduce log level of RLIMIT message on Mac/Linux to avoid it interfering with parsing of stdout from zen version command
• Bugfix: Make sure we close our trace session properly at exit when trace is enabled
• Improvement: Use buffered file reading when replaying oplog
• Improvement: Add endpoint in project store to update the information in a project without deleting the stored data/oplog
• Improvement: Add endpoint in project store to update the information in a project without deleting the stored data/oplog
• Improvement: Add oplog op content to error result if attachment is missing when doing oplog-export
• Improvement: Windows: Cache process handles for FormatPackageMessage reducing function execution time from 100+us to ~1 us
• Improvement: Skip upstream logic early if we have no upstream endpoints
• Improvement: Cachestore logging of CbObjects are now async
• Improvement: Use better hashing algorithm for instance pointers when using shared lock in IoBufferExtendedCore::Materialize
• Improvement: Use tsl/robin-map/robin-set in compactcas and projectstore for 30% faster GC

• Feature: Add more stats for stats/prj
  • Project: read/write/delete count
  • Oplog: read/write/delete count
  • Chunk: hit/miss/write count
  • Op: hit/miss/write count
  • BadRequest count
• Bugfix: Allow oplog file mapping where ServerPath is missing if a attachment hash is specified
• Bugfix: Make sure we always write "data" attachment hash for snapshotted oplog entries
• Bugfix: Fixed expiry limit for GC of project/oplogs
• Improvement: Add response.text to output in log when jupiter request fails
• Improvement: Only hash jupiter oplog ref once when uploading
• Improvement: Increase request timeout when uploading to Jupiter to 3 min (to handle very large attachments)
• Improvement: Fix issues with latest fmt vcpk dependency (10.0.0) and sentry-native for linux

• Feature: Add --assume-http2 option to cloud style import/export command to use a HTTP/2 endpoint without without HTTP/1.1 upgrade
• Bugfix: Make sure out of memory condition does not terminate http-asio service thread. UE-191531
• Bugfix: oplog-import with --file source now sends the oplog folder correctly to zenserver
• Bugfix: If oplog-export fails while creating blocks, wait for background jobs to finish before aborting to avoid crash
• Bugfix: If GetChunkInfo in project store finds a chunk in the wrong format, return a readable error instead of ASSERT
• Bugfix: If checking for state_marker throws exception, exit gracefully rather than throw exception
• Improvement: More details in zenserver logfile if jupiter operation fails

• Feature: Added zen serve command for establishing a link to a directory tree for use with staged UE builds and the -Mount= option
• Bugfix: Make sure to validate return pointer when calling Memory::Alloc in all locations
• Bugfix: Log error instead of hard crash if GC scheduler thread throws exception
• Improvement: In oplog import/export, try to resolve access token via env variable on zen command side as first option, with resolve on zenserver side as second option. Resolves UE-189978
• Improvement: Keep reason and status code when parsing responses from jupiter remote requests
• Improvement: Add additional context for errors when importing/exporting oplogs
• Improvement: Added ZenServerInstance::SpawnServerAndWait and improved logic around process termination when using ZenServerInstance::AttachToRunningServer
• Improvement: When uploading compressed blob to jupiter, use streaming reading of source file if it is a "whole file" - a large attachment.

• Feature: Project store now has a snapshot RPC on oplogs which may be used to inline any files referenced by name into Zen store. This makes the oplog transportable
• Feature: Zen command line tool oplog-snapshot which may be used to inline any files referenced by name into Zen store. This makes the oplog transportable
  • --project Project name (id)
  • --oplog Oplog name (id)
• Feature: Session Id and Request Id are now logged in log for cache put/get operations
• Bugfix: Prevent destructors in ProjectStore::Project, ScopedActivityBase and FileMapping from throwing exceptions to avoid abort termination
• Bugfix: Zen CLI command help now include descriptions for positional arguments
• Bugfix: Correctly prefix auth token when using a bare token in project oplog import/export
• Bugfix: Make sure GetEnvVariable can handle values that are longer than 1023 characters
• Improvement: Throw exception with information on failed memory allocation instead of calling ZEN_ASSERT
• Improvement: Added support for streaming decompression
• Improvement: Added zenserver.exe and zen.exe/zen.pdb to Sentry debug information upload to populate unwind information
• Improvement: Front-end can now be served from a development directory in release mode as well as debug if there's no zipfs attached
• Improvement: Increased retry logic in diskcachelayer when we are denied moving a temporary file into place
• Improvement: Named some additional background threads for better debug / sentry reporting
• Update: Bump CI VCPKG version to 2023.04.15 and xmake to 2.7.9 (was 2022.08.15 and 2.6.5)

• Feature: zenserver/zen: Added zen command line command scrub which can be used to trigger a data scrubbing pass which traverses all stored data and verifies its integrity. If any checksum mismatches or structural errors are found the content is dropped. For now this does not provide much feedback in the console, but the zenserver logs will contain information about the operation
• Feature: zen: added zen bench command which has an option to empty Windows standby lists. This effectively empties the system (disk) cache, which can be useful when performing benchmarks since this puts the system in a more consistent state
• Feature: zen: added zen copy command which can be used to perform copy-on-write copies of files and directories on supported file systems (i.e ReFS on Windows). This is useful when working with test datasets where you want to avoid tests modifying the original test data
• Feature: zenserver: Add command line option --gc-projectstore-duration-seconds to control GC life time of project store data
• Bugfix: Improve error handling when processing requests in http asio
• Bugfix: Error out if test is passed to zenserver in release builds (tests are only compiled in for debug)
• Bugfix: Gracefully exit with error code if problems arise during startup (used to cause abort termination)
• Bugfix: Project oplog delete fixed so it works even right after server startup, before the oplog has been instantiated in memory
• Bugfix: Corrected argument name in oplog-export file target RPC message
• Improvement: Change state_marker detection (deletion of DDC folder) log to WARN, it is not an error but useful information in the log output
• Improvement: Added logging when bad chunks are detected in BlockStore
• Improvement: zen::SetCurrentThreadName now also sets trace (Insights) thread name
• Improvement: All thread pool threads now have names
• Improvement: zenserver now emits session information to trace (Insights) for a better session browser experience
• Improvement: Add more trace instrumentation
• Improvement: Eliminated ATL header dependency
• Improvement: If no -hosturl=... parameter is passed to zen CLI commands we use the current session state from shared memory to pick an instance to communicate with
• Improvement: Better option validation in zen command line parsing

• Feature: Gracefully exit if Ctrl-C is pressed
• Feature: Structured cache now writes an activity log to logs/z$ which may be used to understand client interactions better. Enabled by default for the time being
• Bugfix: Return error code on exit as set by application
• Bugfix: Fix crash at startup if dead process handles are detected in ZenServerState
• Bugfix: Fixed assert/error when running block store GC and a block to GC does not exist
• Bugfix: GC could mix up locations of cache bucket items causing it to return the wrong item for a specific key. All cache buckets from previous versions will be wiped to remove inconsistent state
• Improvement: Log details about file and read operation when it fails inside IoBuffer::Materialize()

• Feature: zenserver now writes a state_marker file in the root of the data directory. Deleting this file will cause zenserver to exit. This is used to detect if user is deleting the data folder while zenserver is running
• Feature: Disk writes are now blocked early and return an insufficient storage error if free disk space falls below the --low-diskspace-threshold value
• Feature: zenserver: Add command line option --sentry-allow-personal-info to allow personally identifiable information in sentry reports, disabled by default
• Feature: Age-based GC of oplogs in project store
• Improvement: Failing to write index snapshots or access times are now considered a warning rather than error
• Bugfix: Validate that block store entries points inside valid blocks when initializing
• Bugfix: Close down http server gracefully when exiting even while requests are still being processed
• Bugfix: Flush snapshot for filecase on flush/exit
• Bugfix: Fix log of size found when scanning for files in filecas

• Bugfix: Treat reading outside of block store file as a not found error. We may encounter truncated blocks due to earlier abnormal termination of zenserver or disk failures

• Feature: ASSERTs triggered at runtime are sent directly to Sentry with callstack if sentry is enabled
• Bugfix: Verify that there are blocks to GC for block store garbage collect (void division by zero)
• Bugfix: Write log error and flush log before reporting error to Sentry/error logger
• Bugfix: Log ERROR in scope guard if function throws exception, throwing exception causes application abort

• Bugfix: Safely handle missing blocks when doing garbage collection in block store data
• Bugfix: Only strip uri accept type suffix if it can be parsed to a known type
• Bugfix: Keep system error code on Windows when file mapping fails and propagate to log/exception
• Bugfix: Catch any errors throw in HttpAsioServer() destructor and log error.

• Strip FILE macro names in logging to only include the file name as to not expose file paths of the machine building the executable
• Bugfix: Reporting the correct callstack to sentry on ERROR/CRITICAL failure.
• Extend sentry message with triggering file/line/function

• Feature: Zen command line tool rpc-record-start to record all RPC requests to the structured cache
  • --path Recording file path where the rpc requests will be stored
• Feature: Zen command line tool rpc-record-stop stop the currently active RPC request recording started with rpc-record-start
• Feature: Zen command line tool rpc-record-replay replacy a RPC request recording created with rpc-record-start
  • --path Recording file path where the rpc requests are stored
  • --numthreads Number of worker threads to use while replaying the RPC requests
  • --numproc Number of worker processes to run, if more than one new processes will be spawn with <numthreads> workers each
  • --offset Offset into request playback to start at
  • --stride The stride to use when selecting requests to playback
  • --onhost Replay the recording inside the zenserver bypassing http overhead
  • --showmethodstats Show statistics of which RPC methods are used
  • --forceallowlocalrefs Force the requests to allow local references (file path/file handle)
  • --disablelocalrefs Force disable local references in request (file path/file handle)
  • --forceallowlocalhandlerefs Force the requests to allow local references via duplicated file handles for requests that allow local refs
  • --disablelocalhandlerefs Force disable local references via duplicated file handles in requests
  • --forceallowpartiallocalref Force the requests to allow local references for files that are not saved as whole files for requests that allow local refs
  • --disablepartiallocalrefs Force disable local references for files that are not saved as whole files for requests that allow local refs
• Feature: Zen command line tool cache-stats to give stats result about the zen cache
• Feature: Zen command line tool project-stats to give stats result about the zen project store
• Feature: Zen command line tool cache-details to give detail result about the zen cache, defaults to overview information about the cache
  • --namespace Get information about cache values in a namespace
  • --bucket Get information about cache values limited to a specific bucket in a namespace
  • --valuekey Get information about a cache value in a specific bucket in a namespace, valuekey is specified as IoHash hex string
  • --details Get detailed information about each cache record
  • --attachmentdetails Get detailed information about each attachments for each cache record
  • --csv Format the output as a comma delimited CSV file. If not specified it defaults to JSon style response.
• Feature: Zen command line tool project-details to give detail result about the zen project store, defaults to overview information about the project store
  • --project The project id to get information about
  • --oplog The oplog id to get information about
  • --opid The op Oid to get information about
  • --details Get detailed information about the op
  • --opdetails Extract the entire op information (not available in CSV output)
  • --attachmentdetails Get detailed information about each attachments for each op
  • --csv Format the output as a comma delimited CSV file. If not specified it defaults to JSon style response.
• Feature: New project store stats endpoint /stats/prj to get stats info for zen project store
• Feature: New project store details endpoints /prj/details$, /prj/details$/{project}, /prj/details$/{project}/{oplog}, /prj/details$/{project}/{oplog}/{op} to give detail result about the zen project store, defaults to overview information about the project store items
  • details=true Get detailed information about the op
  • opdetails=true Extract the entire op information
  • attachmentdetails=true Get detailed information about each attachments for each op
  • csv=true Format the output as a comma delimited CSV file. If not specified it defaults to JSon style response.
• Feature: New cache detail endpoints /z$/details$, /z$/details$/{namespace}, /z$/details$/{namespace}/{bucket}, /z$/details$/{namespace}/{bucket}/{key} has been added
  • details=true Get detailed information about each cache record
  • attachmentdetails=true Get detailed information about each attachments for each cache record
  • csv=true Format the response as a comma delimited CSV file. If not specified it defaults to CbObject but can auto-format to json
• Feature: --junit switch to xmake test to generate junit style reports of tests.
• Feature: CI build on GitHub now uploads junit test reports as artifact to the check for PR validation and mainline validation
• Feature: Payloads from zenserver can now be sent using duplicated file handles if caller requests provides client ProcessId (Windows only).
• Feature: Add --port option to zen down command to shut down servers on different base ports
• Bugfix: Make sure async responses are sent async correctly in httpsys
• Bugfix: Don't delete manifest file in cas root when initializing a new filecas folder
• Bugfix: Sentry does not like UNC paths, so strip the prefix before passing them to sentry
• Bugfix: Make sure zen down command uses the correct port for shutdown event
• Improvement: FileCas now keeps an up to date index of all the entries improving performance when getting cache misses on large payloads
• Improvement: Structured cache now keeps RawHash and RawSize in memory avoiding materialization of cache values before sending response
• Changed: Exit with failure code on port conflict rather than reporting crash to Sentry
• Changed: removed catch2 support for now since it does not handle multithreaded tests
• Bugfix: fixed bug in dashboard content serving (see PR #255)

• Bugfix: Don't send empty http responses with content type set to Text. Fixes UE-177895

• Feature: Add support for "packagedata" mapping in oplog entries
• Feature: Zen command line tool project-create to create a project store project
  • --project Project name (id)
  • --rootdir Absolute path to root directory (optional)
  • --enginedir Absolute path to engine root directory (optional)
  • --projectdir Absolute path to project directory (optional)
  • --projectfile Absolute path to .uproject file (optional)
• Feature: Zen command line tool oplog-create to create a project store oplog
  • --project Project name (id)
  • --oplog Oplog name (id)
  • --gcpath Absolute path to oplog lifetime marker file (optional)
• Feature: Build scripts and tooling to build zen compliant with VFX reference platform CY2022/2021 matching UE linux builds
• Feature: added xmake sln task which replaces generate_projects.bat
• Feature: Zen server endpoint prj/{project}/oplog/{log}/chunks to post multiple attachments in one request.
• Feature: Zen server endpoint prj/{project}/oplog/{log}/save to save an oplog container. Accepts CbObject containing a compressed oplog and attachment references organized in blocks.
• Feature: Zen server endpoint prj/{project}/oplog/{log}/load to request an oplog container. Responds with an CbObject containing a compressed oplog and attachment references organized in blocks.
• Feature: Zen server endpoint {project}/oplog/{log}/rpc to initiate an import to or export from an external location and other operations. Use either JSon or CbPackage as payload.
  • CbObject/JSon RPC format for import and export methods:

      {
        "method" : "<method>",
        "params" : {
          "maxblocksize": "<maxblocksize>",
          "maxchunkembedsize": "<maxchunkembedsize>",
          "file" : {
            "path" : "<file-system-folder-path>",
            "name" : "<oplog-file-name>"
          },
          "cloud" : {
            "url" : "<serviceurl>",
            "namespace" : "<namespace>",
            "bucket" : "<bucket>",
            "key" : "<iohash>",
            "openid-provider" : "<provider-id>",
            "access-token" : "<access-token>",
      	  "access-token-env": <envvariablename>",
            "disableblocks" : "<disableblocks>",
            "disabletempblocks" : "<disabletempblocks>"
          },
          "zen" : {
            "url" : "<url>",
            "project" : "<projectid>",
            "oplog" : "<oplogid>"
          }
        }
      }

    • "method"supported methods are "export" and "import" to import/export an oplog
    • "params" container for parameters
      • "maxblocksize" - Optional. The maximum size of a block of attachments, default 134217728 (128 Mb) (export only)
      • "maxchunkembedsize" - Optional. The maximum size of an attachment to be put in a block, larger attachments will be stored as usual attachments, default 1048576 (1Mb) (export only)
      • "force" - Optional. Boolean flag to indicate weather attachments should be uploaded/downloaded disregarding prior existance
      • External location types are "file" (File system), "cloud" (UE Cloud Storage service) or "zen" (Zen server instance), provide one of those as remote location.
        • "file" - Optional. Indicates remote location is the local file system
          • "path" - File system path folder to export to / import from
          • "name" - File name of oplog output, written into
        • "cloud" - Optional. Indicates remote location is UE Cloud Storage service
          • "url" - Jupiter service endpoint url
          • "namespace" - Name of namespace to store data to
          • "bucket" - Name of bucket to store data to
          • "key" - IoHash key to the stored oplog container
          • "openid-provider" - Optional. Name of openid provider used to authenticate with, requires that the zen server instance has been provided with a oids refresh token for
          • "access-token" - Optional. JWT access token to authenticate with
          • "access-token-env" - Optional. Name of environment variable that holds an JWT access token to authenticate with
          • "disableblocks" - Optional. Disable creation of attachments blocks - "true"/"false" (export only)
          • "disabletempblocks" - Optional. Disable creation of attachments temp blocks forcing upload before oplog container - "true"/"false" (export only)
        • "zen" - Optional. Indicates remote location is a Zen server instance
          • "url" - Zen server instance url
          • "project" - The remote project name (id)
          • `"oplog" - The remote oplog name (id)
  • CbObject RPC format for getchunks method, returns CbPackage with the found chunks, if all chunks are found the number of attachments matches number of chunks requested.

      {
        "method" : "getchunks",
        "chunks" : [
          "<rawhash>",
        ]
      }

  • CbPackage RPC format for putchunks method, attachments are stored in CidStore

      {
        "method" : "putchunks",
      }

• Feature: Zen server {project}/oplog/{log}/{hash} now accepts HttpVerb::kPost as well as HttpVerb::kGet.
• Feature: Zen command line tool oplog-export to export an oplog to an external target using the zenserver oplog export endpoint.
  • --project Project name (id)
  • --oplog Project name (id)
  • --maxblocksize The maximum size of a block of attachments (optional)
  • --maxchunkembedsize The maximum size of an attachment to be put in a block, larger attachments will be stored as usual attachments (optional)
  • --force Force upload/download of attachments even if they already exist.
  • --file File system path folder to export to / import from
    • --name File name of oplog output, written into --file path
    • --disableblocks Disable block creation and save all attachments individually
    • --forcetempblocks Force creation of temp attachment blocks
  • --cloud Jupiter service endpoint to export to / import from
    • namespace Name of namespace to store data to
    • bucket Name of bucket to store data to
    • key Key to the stored oplog container (If omitted a default key will be generated based on project/oplog/namespace/bucket)
    • openid-provider Optional name of openid provider used to authenticate with, requires that the zen server instance has been provided with a oids refresh token for the provider name
    • access-token Optional JWT access token to authenticate with
    • access-token-env - Optional name of environment variable that holds an JWT access token to authenticate with
    • disableblocks Disable block creation and save all attachments individually
    • disabletempblocks Disable temp block creation and upload blocks without waiting for oplog container to be uploaded
  • --zen Zen server instance url to export to / import from
    • --target-project The remote project name (id) (optional, defaults to same as project)
    • --taret-oplog The remote oplog name (id) (optional, defaults to same as olplog)
    • --clean Delete and create a new oplog before starting export
• Feature: Zen command line tool oplog-import to import an oplog from an external source using the zenserver oplog import endpoint.
  • --project Project name (id)
  • --oplog Project name (id)
  • --force Force upload/download of attachments even if they already exist.
  • --file File system path folder to export to / import from
    • --name File name of oplog output, written into --file path
  • --cloud Jupiter service endpoint to export to / import from
    • namespace Name of namespace to store data to
    • bucket Name of bucket to store data to
    • key Key to the stored oplog container (If omitted a default key will be generated based on project/oplog/namespace/bucket)
    • openid-provider Optional name of openid provider used to authenticate with, requires that the zen server instance has been provided with a oids refresh token for the provider name
    • access-token Optional JWT access token to authenticate with
    • access-token-env - Optional name of environment variable that holds an JWT access token to authenticate with
  • --zen Zen server instance url to export to / import from
    • --source-project The remote project name (id) (optional, defaults to same as project)
    • --source-oplog The remote oplog name (id) (optional, defaults to same as olplog)
    • --clean Delete and create a new oplog before starting import
• Improvement: Faster oplog replay - reduces time to open an existing oplog
• Improvement: Clearer error messages and logging when requests to project store fails
• Changed: Removed remnants of old mesh experiment
• Changed: Remove obsolete export-project command
• Changed: Removed remnants import-project command
• Changed: Removed unused remote build scripts
• Changed: Removed very old and invalid TODO.md
• Changed: Removed some deprecated scripts

• Feature: Added info (GET) endpoints for structured cache
• /z$ - get a list of namespaces and global info
• /z$/{namespace} - get list of buckets in a namespace and namespace related info
• /z$/{namespace}/{bucket} - get bucket info
• Feature: Added project store oplog info: markerpath, totalsize, opcount, expired on GET requests for oplog
• Feature: Added project store project info: expired on GET requests for project
• Feature :Added project store root route /prj which is identical to /prj/list
• Feature: Zen command line tool cache-info to show cache, namespace or bucket info
• Feature: Zen command line tool project-info to show store, project or oplog info
• Feature: Zen command line tool project-drop to drop project or oplog
• Feature: Zen command line tool gc to trigger a GC run
• Feature: Zen command line tool gc-info to check status of GC
• Feature: Added version information to zenserver and zen command line tool executables
• Bugfix: Don't log "time to next GC" if time to next GC is not set
• Improvement: Don't wait for GC monitor interval before doing first GC check
• Improvement: Zen command line tool now fails on any unrecognized arguments
• Improvement: Zen command line tool now displays extra help for all sub-commands
• Improvement: Host address can now be configured for zen command line tool drop command
• Improvement: Added precommit xmake task xmake precommit to run precommit checks
• Changed: Default GC interval set to 1 hour
• Changed: Default GC cache duration set to 2 weeks
• Changed: Removed HttpLaunchService and related code
• Changed: Removed dead/experimental file system related code
• Changed: Removed faux vfs config option

• Feature: Oplog level GC in project store. If gc marker file path is given by UE, oplogs will be GC:d when marker file is deleted (and GC is triggered)
• Bugfix: Index handling for cache large object store was broken resulting in log always being played back
• Bugfix: Make sure to flush cache store on call to flush on service and exit
• Improvement: Don't write index snapshots if no new entries has been added to log

• Feature: Recording and playback of cache request with full data - both get and put operations can be replayed. Invoke via web request
  • <host>/z$/exec$/start-recording?<disk-storage-path>
  • <host>/z$/exec$/stop-recording
  • <host>/z$/exec$/replay-recording?<thread-count>&<disk-storage-path>
• Feature: Disk size triggered GC, a soft disk usage limit for cache data
• Feature: New option --gc-disk-size-soft-limit (command line), gc.cache.disksizesoftlimit (lua config) controlling limit for soft disk usage limit. Defaults to zero which disables soft disk usage limit
• Improvement: Disk write pressure in GC log and cleaned up clutter in GC logging
• Improvement: Much improved performance, between 2x to 9x improvement under heavy load (excluding http service overhead). See https://github.com/EpicGames/zen/pull/200 for details
• Bugfix: Always store records or oplog entries before storing attachments to avoid GC finding unreferenced chunks i CidStore
• Bugfix: Updated Zen drop command to support namespaces
• Bugfix: Use ZEN_CONSOLE for output to console in Zen commands
• Bugfix: Zen status command now shows info about found Zen instances
• Bugfix: Zen top command now shows session id string
• Bugfix: On Windows platforms explicitly set the special SO_EXCLUSIVEADDRUSE flag as SO_REUSEADDR still allows shared use of sockets
• Bugfix: Fix logging of number of entries read from caslog at startup
• Bugfix: Fix asio http handling of very large/malformed headers and handle mismatching content size
• Changed: Reduced GC INFO spam by converting to DEBUG log messages
• Changed: Use Iso8601 format for logging start and end message

• Feature: Adds two command to Zen command tool to export/import project store oplogs with attachments
  • export-project <target-folder> <project> [oplogs...]
  • import-project <target-folder> <project> [oplogs...]
• Feature: Adds command to query Zen version, specify host-name url to query running service version, otherwise you get zen command version. detailed option gives you long form version.
  • version [host-name] [detailed]
• Feature: New service endpoint to query Zen server version, add ?detailed=true to get long form version
  • <host>/health/version
• Feature: Configure OpenID providers from cmd line and Lua cfg
• Feature: Added zen command line executable to release distribution
• Bugfix: Fix double reporting of disk usage for namespaces
• Bugfix: Fix double garbage collection analisys and garbage collection execution of namespaces
• Improvement: Improve tracking of used disk space for filecas and compactcas
• Improvement: Add tracking of used disk space for project store
• Improvement: Bumped limit for storing cache values as separate files to reduce number of loose files
• Improvement: Optimizations when handling compressed buffer (less materialization and reading of headers)
• Improvement: Send attachments as file references if the IoBuffer we find represents a complete file and AcceptFlags in RPC request allows it.
• Improvement: Don't reserve full block size for block store files at creation

• Change: Responding with new wire format for RPC requests requires the requestor to add a Accept field in the request. This is to allow compatability with older clients for shared instances.
• Improvement: Fixed concurrency issues in project store - project and oplog lifetime issues.
• Improvement: Don't open oplogs until we require use of them.
• Cleanup: Remove rocksdb experimental code.
• Feature: Add GC to projects store. Checks path to project file in UE side to determine when a project may be GCd on the Zen side.

• Change: All RPC responses are now formatted using dedicated wire format, Zen server has fallback to enable compatability with legacy upstreams
• Feature: Adding a .json extension to the --abslog option will make zenserver log in json format to file
• Feature: Create release in Sentry and use sentry_options_set_release to associate the executable
• Bugfix: CompactBinary: Fixed LoadCompactBinary to gracefully handle read failures and sizes larger than the archive. From http1s://p4-swarm.epicgames.net/changes/21983905
• Bugfix: Use bucket/key to get inline value in upstream for chunks without a chunkid
• Bugfix: Handle edge case when trying to materialize a IoBuffer of zero size via memory mapping
• Improvement: Logging: don't do formatting of messages the will not be logged
• Improvement: Logging: Timing and upstream source information in upstream logging when debug level logging is enabled
• Improvement: Reduce buffer creation and copying in ParsePackageMessage
• Improvement: Don't read attachments for oplogs we already have when parsing oplog message

• Bugfix: Bugfix: Use bucket/key to get inline value in upstream for chunks without a chunkid (UE-164966)

• Bugfix: Don't fail entire request if GetCacheValue from Horde fails for a single value

• Change: Bumped ZEN_SCHEMA_VERSION - this will invalidate entire local cache when deployed
• Change: Make CAS storage an hidden implementation detail of CidStore, we no longer hash and do mapping to compressed hash when storing cache values
• Feature: Extended zen print command to also handle CbPackage and CompressedBuffer format payloads
• Feature: Added /prj/{project}/oplog/{log}/{op} endpoint to allow retrieval of an op entry by LSN. Supports returning CbObject or CbPackage format payloads
• Improvement: asio: added some context to error reporting
• Improvement: namespace/bucket validation now uses AsciiSet for more efficient validation
• Improvement: Frontend: simplified content-type logic
• Improvement: Improved message indicating no GC is scheduled
• Improvement: Implement proper GetCacheValues upstream path
• Improvement: Demote a number of ZEN_ERROR log calls for problems that are recoverable and handled
• Bugfix: Use bucket/key to get inline value in upstream for chunks without a chunkid
• Bugfix: Fixed issue in CbPackage marshaling of local reference
• Bugfix: Fix crash when switching Zen upstream configured via DNS when one endpoint becomes unresposive
• Bugfix: Fixed issue where projects would not be discovered via DiscoverProjects due to use of stem() vs filename()
• Bugfix: Use "\\?\" prefixed paths on Windows and fix hardcoded path delimiters (UE-141222)
• Bugfix: Safer detection of html folder when running non-bundled executable
• Bugfix: Use "application/x-jupiter-inline" to fetch GetCacheValues from Horde (UE-162151)
• Sentry: Added logging of sentry_init error code
• Sentry: Attach log file to Sentry error reports
• Sentry: Capture capture error/critical log statements as errors in Sentry
• Update: Bump VCPKG version to 2022.08.15
• CI: MacOS build enable again in GitHub CI
• CI: Upload debug info and source files to Sentry when release is created

-- Enable adding namespace to URI based upstream requests -- Add logging of namespace name and bucket name if we get invalid names in requests -- Updated README.md with Linux dev prerequisites -- asio: added some logging to indicate concurrency -- Fixed expired cache keys overwriting between namespaces when bucket names were the same in multiple namespaces

• Tweak bundle compression settings to streamline build
• ZenCacheDiskLayer::CacheBucket::GatherReferences: Don't hold index lock while reading standalone values
• hardening of ZenCacheDiskLayer::CacheBucket::PutStandaloneCacheValue
• GitHub Actions: Move release job to in-house linux agent

• BlockStore (small object store) Always block GC of current write block
• Make it possible to configure GC monitoring interval using --gc-monitor-interval-seconds
• Keep "reason" from upstream response so we can present it even if the request fails without outright error
• New GitHub Actions release flow - Add release flow in GitHub actions on pushed tag v0.1.2 gives full release, v0.1.2-pre0 gives pre-release

• Fixes issue with broken Zen instances for legacy requests

• Enable FILE_SHARE_DELETE on standalone files in disk buckets - fixes Jira UE-154234
• Make sure we can properly create the block file before assigning it for use - fixes Jira UE-154438
• Horde execute compressed input blobs
• Drop namespace support
• Safer delete of cache buckets

• Namespaces: This introduces namespaces to the zenserver but only the default ue4.ddc is supported. Clients that don't send a namespace in the request will keep old behviour, new clients that sends namespace is required to use ue4.ddc (which they currently do)
• Aligned bucket naming rules with UE code base
• Fix retry counter and add an extra iteration to give more time for success during contention for standalone files in cache
• Make sure CacheBucket::PutStandaloneCacheValue cleans up the temp file
• Restore logic where we accept failed overwrite if resulting size is the same for standlone file in cache
• Correctly calculate the m_TotalSize difference when overwriting file for standalone files in cache
• Fix namespace folder scanning