We'd like to thank Aaron J. Scantlin (@scantlina), Aashish Sharma (@initconf),
AbdolRashid Forghani (@soorooghadim), Anthony Alayo (@anthonyalayo), Bauti
Peirone (@bautipeirone), @Breppe, @cccs-graeme, @cccs-will, Connor
(@agent-connor), Craig Leres (@leres), @dagecko, Doğukan Çağatay
(@dogukancagatay), Fatema Bannat Wala (@fatemabw), François De Keersmaeker
(@fdekeers), Jan Grashöfer (@J-Gras), Klemens Nanni (@klemensn), Maor Hamami
(@mamaorha), Martin Camara (@martincmr), Michael Peters (@MP-Corelight), Mohan
Dhawan (@Mohan-Dhawan), Octave Charrin (@OctaveCharrin), Peter Cullen
(@pbcullen), @RageAgainstTheOrganic, @siavashta, Steve Smoot (@stevesmoot),
Stefan Götz (@stefangotz), @timo-mue and Zach Robinette (@zrobinette12) for
their contributions to this release.

Breaking Changes

• The minimum version requirement for ZeroMQ is now 4.3.0. This version has
  been released at the end of 2018 - a good 7 years ago.

• Enums are now nominally typed. Zeek will produce warnings of the following form
  if you have scripts where enum types are mixed.

  Remove in v9.1. Mixing incompatible enum types color and city will become an error.

  With version 9.1, these warnings will become errors. Previously, all enum types were
  compatible with each other, resulting in quirks and making certain optimizations more
  difficult than they should be.

• The BifEnum namespace was moved to be part of the zeek namespace for consistency
  with the other namespaces generated by bifcl (BifType, BifConst, etc).

• The MetricsAddress option in zeekctl.cfg default was changed from 0.0.0.0 to
  127.0.0.1. If you've been using a remote Prometheus server for metrics scraping
  or run Zeek in a container environment, you'll now need to explicitly set MetricsAddress
  to 0.0.0.0 (or the appropriate listen address) in zeekctl.cfg.

• The packet captures in the btest Traces library have standardized on .pcap and .pcapng
  file suffixes. This may cause breakage in plugins that were using the old .trace suffix.

• The public vlan and inner_vlan members within C++ Packet has changed from
  uint32_t to an optional VlanTag struct. The corresponding dei and pcp
  members have been removed and merged into the VlanTag struct. This does not change
  scripts. In order to access the ID as before, check the optional first:

  if ( packet.vlan )
  my_vlan = packet.vlan->id;

  This allows Zeek to properly handle VLAN IDs of 0, which before were handled as if
  there was no VLAN.

• Reverted the change that went into Zeek 8.1 where ASCII control characters < 32
  were encoded as \u00XX in JSON strings to the prior \xXX style. The escaping style
  will become configurable to allow for reliably representing the contained bytes
  in strings with Zeek 9.0.

  See the discussion at #5240 for more background
  and feel free to chime in with feedback.

• FreeBSD is no longer an officially-supported platform. We will make our best efforts to
  provide fixes to reported issues, like any other platform, but will not longer do
  regular builds on our CI provider.

New Functionality

• Added a new attribute &publish_on_change for global tables and sets as an
  alternative to the Broker-specific and deprecated &backend and &broker_store
  synchronization attributes. The &publish_on_change attribute allows efficient
  publishing of selected table changes as remote events to a configurable cluster topic.
  Other cluster nodes by default apply received changes to their local table instances.

  Publish any new entry as Cluster::table_change_infos() event to /zeek/table/endpoints

  global endpoints: set[addr, addr] &write_expire=5min &publish_on_change=[
  changes=set(TABLE_ELEMENT_NEW),
  ];

  See the online documentation for a detailed description and further examples:

  https://docs.zeek.org/en/master/reference/zeekscript/attributes.html#attr-&publish_on_change

• Well-known ports of built-in analyzers can now be configured via redef. For example:

  redef HTTP::ports += { 8181/tcp };

  For protocol analyzers, the variable name is generally Analyzer_Module::ports,
  but in certain cases (SSL, QUIC, RDP, ...) the variable name includes the protocol
  name to distinguish different internal analyzers. SSL::ssl_ports and SSL::dtls_ports
  as a concrete example. Check the analyzer module's export section or online documentation.
  For packet analyzers, the analyzer name is included in the variable name for historical
  reasons. Example:

  redef PacketAnalyzer::VXLAN::vxlan_ports += { 4711/udp };

• The dns.log now also contains information about DNS notifications (RFC 1996).
  The opcode is set to 4 and opcode_code name will be "notify".

• ZeroMQ cluster communication can now be encrypted using the CURVE mechanism built
  into ZeroMQ. This requires a pair of keys, one for the server role and one for
  the client role. The central XPUB/XSUB sockets and LOG pull sockets are configured
  as CURVE servers, all other sockets (the connecting Zeek nodes) as CURVE clients.

  You can generate Z85-encoded keypair as follows with Zeek:

  $ zeek -e 'print to_json(Cluster::Backend::ZeroMQ::generate_keypair())' | jq
  {
  "public": "}}6zuTySXq(C(6dF#D3Kvt@A5D}LcFlOgVav5{%)",
  "secret": "h.<[iyv]O#cWdO%2yuLiEJQ}6tC)@HxN6Hp?![td"
  }

  When using ZeekControl, keys are automatically generated and configured when a multi-node
  cluster based on node IP addresses is detected. To force encryption to be always on, set
  the following ZeroMQ specific setting in zeekctl.cfg (default is "auto"):

  cluster_backend_zeromq.use_curve_encryption = 1

  More details and background is available in the
  policy/frameworks/cluster/backend/zeromq/main.zeek script documentation.

• The number of messages and bytes processed by the central ZeroMQ XPUB/XSUB
  cluster component is now exposed as metrics of the form:

  zeek_cluster_zeromq_proxy_{frontend,backend}{bytes,messages}{sent,received}

  by the process running this component in a cluster. Note that Zeek uses multipart
  messages for events: Publishing a single event currently counts as four messages
  as we're simply exposing ZeroMQ's low-level zmq::proxy_steerable() statistics.

• The @load directive now supports loading shared object files. This can
  be leveraged to experiment with alternative package and plugin directory layouts
  where a __load__.zeek file in a directory loads the required shared object
  explicitly rather than the current "magic discovery" via ZEEK_PLUGIN_PATH
  and __zeek_plugin__ files.

  Files are expected to end with .so regardless of the platform and need to
  be explicitly loaded with the suffix provided:

  @load ./plugin.so

  Such shared objects should instantiate a single zeek::plugin::Plugin instance at
  load time. The zeek::plugin::Plugin() will register it accordingly.

• bifcl now accepts missing return types in functions signatures, creating
  void functions rather than complaining. The C++ signature will continue
  to have a ValPtr return type and the implementation should return a Val::nil,
  but the Zeek script function signature will have a void return type.

• The &on_change attribute for tables or sets now accepts variable argument functions.

• Added the modulo BiF, which behaves differently from Zeek's % operator
  for negative numbers.

• Added the connection_timing_out hook which, if break is used within, will
  prevent Zeek from timing out a particular connection. The motivation is to align
  a shunted connection's timeout with Zeek. Since Zeek does not see packets from
  a shunted connection, it needs special handling to keep the connection alive while
  shunted.

• The Zeek documentation has received a major refresh. A new tutorial walks newcomers
  through setup, basic Zeek invocations, the use of Zeek packages, ZeekControl for cluster
  maintenance, the basics of Zeek's logs, as well as scripting in Zeek's own language as
  well as JavaScript. A new reference section houses the introduction to common logs, the
  Zeek scripting language, Zeek's frameworks, and the scripting index.

• A new analyzer for IGMP was imported from an external project, originally written by
  François De Keersmaeker (@fdekeers). It adds new events for each of the types of IGMP
  messages, but does not add a new log.

• Windows support has been greatly improved, thanks to a raft of contributions from an
  engineering team at Microsoft. The btest suite now runs on CI builds. The great majority
  of btests pass, with a small number skipped for various documented reasons.

Changed Functionality

• Protocol parsers implemented in Spicy now suppress analyzer violations when
  a gap occurs and the parser doesn't implement error recovery. The most user
  visible aspect is that the analyzer's identifier is not removed from the
  service column in the conn.log.

• The to_subnet() built-in function's parsing of IPv6 subnets was changed
  to align with how IPv6 subnet literals are parsed in Zeek scripts. Previously,
  any subnet containing the ::ffff:0:0/96 IPv4-mapped prefix would produce
  an IPv4 subnet, even if provided prefix was shorter than the required 96 bits.

  Internally, the behavior of the IPPrefix::ConvertString() API was changed
  to accommodate for this.

• The policy/frameworks/analyzer/detect-protocols script now respects analyzer
  violations to suppress Server_Found or Protocol_Found for connections which
  first had an analyzer confirmation, but later a analyzer violation happened. Previously,
  Zeek would report a HTTP server when a client issued a HTTP request to an SSH servers,
  or SSL when a client sent a TLS Client Hello to a HTTP server.

• The events ssl_extension_pre_shared_key_server_hello and ssl_extension_pre_shared_key_client_hello
  can now be used independently. Previously, both had to have a handler implemented for
  either of them to actually be invoked due to some logic confusion in the analyzer.

• Zeek new properly handles VLAN IDs of 0. Before, these were simply discarded.

• X.509 parsing now supports additional key types that were not handled previously. The
  certificate key length field will now be filled for additional ciphers, if supported by
  your OpenSSL version (like ML-DSA). Similarly, the key_type field will be filled out for
  additional ciphers, if supported by OpenSSL.

• The SMTP analyzer now forwards the first gap encountered during RFC 822 message analysis
  to the file object. This allows to determine if message analysis stopped due to a gap by
  inspecting the missing_bytes field during file_state_remove(), or in files.log.

  The SMTP analyzer tears down the active file upon encountering a gap, so the reported
  missing bytes will the gap length only.

Deprecated Functionality

• The Broker::create_master(), Broker::create_clone() and Cluster::create_store()
  builtin functions have been deprecated. Consider using the new &publish_on_change
  attribute on tables for distributing table entries, explicit Cluster::publish() calls
  with remote events, or using the storage framework for persistence.

  When ZeroMQ is selected as cluster backend, calling any of the above functions
  results in a fatal error as these are all Broker specific.