This document summarizes the most important changes in the current Zeek

release. For an exhaustive list of changes, see the ``CHANGES`` file

(note that submodules, such as Broker, come with their own ``CHANGES``.)

Zeek 8.2.0

==========

We'd like to thank ... for their contributions to this release.

Breaking Changes

----------------

- The minimum version requirement for ZeroMQ is now 4.3.0. This version has

been released at the end of 2018 - a good 7 years ago.

- Enums are now nominally typed. Zeek will produce warnings of the following form

if you have scripts where enum types are mixed.

Remove in v9.1. Mixing incompatible enum types color and city will become an error.

With version 9.1, these warnings will become errors. Previously, all enum types were

compatible with each other, resulting in quirks and making certain optimizations more

difficult than they should be.

New Functionality

-----------------

- Well-known ports of built-in analyzers can now be configured via redef. For example:

redef HTTP::ports += { 8181/tcp };

For protocol analyzers, the variable name is generally ``Analyzer_Module::ports``,

but in certain cases (SSL, QUIC, RDP, ...) the variable name includes the protocol

name to distinguish different internal analyzers. ``SSL::ssl_ports`` and ``SSL::dtls_ports``

as a concrete example. Check the analyzer module's ``export`` section or online documentation.

For packet analyzers, the analyzer name is included in the variable name for historical

reasons. Example:

redef PacketAnalyzer::VXLAN::vxlan_ports += { 4711/udp };

- The number of messages and bytes processed by the central ZeroMQ XPUB/XSUB

cluster component is now exposed as metrics of the form:

zeek_cluster_zeromq_proxy_{frontend,backend}_{bytes,messages}_{sent,received}

by the process running this component in a cluster. Note that Zeek uses multipart

messages for events: Publishing a single event currently counts as four messages

as we're simply exposing ZeroMQ's low-level ``zmq::proxy_steerable()`` statistics.

- ``bifcl`` now accepts missing return types in functions signatures, creating

void functions rather than complaining. The C++ signature will continue

to have a ``ValPtr`` return type and the implementation should return a ``Val::nil``,

but the Zeek script function signature will have a void return type.

- The ``&on_change`` attribute for tables or sets now accepts variable argument functions.

- Added the ``modulo`` BiF, which behaves differently from Zeek's ``%`` operator

for negative numbers.

Changed Functionality

---------------------

- The ``to_subnet()`` built-in function's parsing of IPv6 subnets was changed

to align with how IPv6 subnet literals are parsed in Zeek scripts. Previously,

any subnet containing the ``::ffff:0:0/96`` IPv4-mapped prefix would produce

an IPv4 subnet, even if provided prefix was shorter than the required 96 bits.

Internally, the behavior of the ``IPPrefix::ConvertString()`` API was changed

to accommodate for this.

Deprecated Functionality

------------------------

Zeek 8.1.0

==========

We'd like to thank Andrew Raffman (@andyraf), @arm7mm, Artyom Kalabukhov (@predator89090),

Benjamin Grap (@blightzero), @biswajitutil, Jean-Samuel Marier (@cccs-jsjm), @chrisjlly,

Craig P (@multicast-bytes), @DigiAngel, Mike Dopheide (@dopheide-esnet), Edoardo Mich

(@edoardomich), @franky-m, Fupeng Zhao (@AmazingPP), Jan Grashöfer (@J-Gras), Klemens

Nanni (@klemens-ya), Klemens Nanni (@klemensn), @kshitiz56, Mark Overholser

(@markoverholser), Mohan Dhawan (@Mohan-Dhawan), Peter Cullen (@pbcullen), RyugaXhypeR

(@RyugaXhypeR), Seth Hall (@sethhall), Yacin Nadji (@ynadji), Zhang Liang (@JosenLiang),

Zach Robinette (@zrobinette12) and @zzzdong for their contributions to this release.

Breaking Changes

----------------

- A standalone ``zeek`` invocation on the command line will now default to running a No-op

Cluster backend (``Cluster::CLUSTER_BACKEND_NONE``). Previously this defaulted to Broker

without having any real effect. For a multi-process / cluster deployment of Zeek, a

cluster backend policy script needs to be loaded explicitly.

Zeekctl has been updated to use the ZeroMQ cluster backend by default and should require

no other action. If you encounter issues with ZeroMQ, it is possible to re-enable the

Broker cluster backend by adding the following entry into ``zeekctl.cfg`` and running

`zeekctl deploy``:

ClusterBackend = Broker

Note that ZeroMQ cluster communication is unencrypted and in the clear by default.

If you're running Zeek on a single host, communication happens via localhost only.

When running a multi host Zeek setup, Zeekctl will output the following warning,

informing you about this behavior:

Warning: ZeroMQ cluster backend enabled and multi-node cluster detected (10.0.0.1, 10.0.0.2).

Communication between Zeek nodes using ZeroMQ is currently unencrypted. Use Broker with TLS if this

is concerning to you. ZeroMQ encryption is tracked at https://github.com/zeek/zeek/issues/4432

You may disable this warning by setting the following option in zeekctl.cfg:

cluster_backend_zeromq.disable_unencrypted_warning = 1

Support for opt-in ZeroMQ encryption is targeted for Zeek 8.2. If this is a

requirement for your environment, please report and for the time being continue

using the Broker backend.

If you manage a non-Zeekctl Zeek setup, load

``policy/frameworks/cluster/backend/zeromq`` or

``policy/frameworks/cluster/backend/broker``. If you forget to do so, Zeek will

recognize that ``Cluster::node`` is set (e.g. the ``CLUSTER_NODE`` environment variable

or ``Cluster::node=...`` is set) but no cluster backend was chosen and will terminate

with a fatal error:

fatal error in <...>/main.zeek, line 425: Cluster::node set to 'worker-1', but

Cluster::backend is Cluster::CLUSTER_BACKEND_NONE - please select a cluster backend to

use.

If you used Zeekctl on a single physical system or virtual machine previously, this

should all be transparent to you.

- The Broker-specific ``&backend`` and ``&broker_store`` attributes will result in startup

errors when a non-Broker cluster backend is selected. See the entry about their

deprecation below for more details.

- Python 3.10 is now required for Zeek and all of its associated subprojects.

- The ``&optional`` script attribute will now error when applied to anything that's not a

record field. Previously, this would have surprising behavior.

- The BinPAC, Bifcl, and Gen-ZAM tools have all moved directly into the Zeek repo, which

should ease maintenance on them a bit. They were moved from the ``auxil`` directory to

the ``tools`` directory. Along with this, the ``--gen-zam`` argument for ``configure``

was removed and the internal version will always be used.

- The zeek-af_packet-plugin git submodule was moved directly into the Zeek repo. This used

to live in the ``auxil`` directory, after having moved there from an external plugin.

It is now automatically built as part of the main Zeek build whenever building on Linux.

- The global ``g_frame_stack`` C++ variable was removed. This variable was used to track

the current stack of script-function frames, mostly for the built-in script debugger. We

realized that we were also tracking this information in the global ``call_stack``

variable, used by the script profiler. These two have now been combined together into

``call_stack``. Plugins using ``g_frame_stack`` will need to be adapted to use the other

variable. We opted to not use a deprecation cycle for this due to the complication with

managing the state across both variables.

- The `unique_id` and `unique_id_from` BIFs now return 96-bit UIDs, which makes them

consistent with other UIDs created by Zeek such as connection IDs. They were previously

returning shorter 64-bit UIDs.

- The metrics ``zeek_msgthread_pending_messages_in_buckets`` and

``zeek_msgthread_pending_messages_out_buckets`` had labels that did not conform with

Prometheus' checks via ``promtool``. The labels have been renamed from ``le`` to

``leq``, given that they technically are less-than-or-equal values.

- The ``iosource/Packet.h`` header no longer depends on libpcap being installed and no

longer directly includes the ``pcap.h`` header. This may cause some knock-on effects in

building plugins that were depending on this header and others that it included

indirectly.

- External plugins do not implicitly require OpenSSL and Broker anymore. Starting with

Zeek 8.1, plugins that depend on these libraries are now required to explicitly call

`find_package(OpenSSL REQUIRED)` and pass `OPENSSL_LIBRARIES` as dependencies to the

`zeek_add_plugin()` call:

find_package(OpenSSL REQUIRED)

zeek_add_plugin(

...

INCLUDE_DIRS

${OPENSSL_INCLUDE_DIR}

DEPENDENCIES

${OPENSSL_LIBRARIES}

)

- We are revisiting removing support for MD5 hashing across the Zeek code base. In support

of this, the following changes were made:

- The ``host_key`` field in ssh.log was removed and replaced with a new

``host_key_fingerprint`` field. The old field held an MD5 hash of the host key. The

new field holds a base64-encoded version of the hash of the host key, in the same

format that the ``ssh-keygen -l`` command would return. It currently uses SHA256 for

the hashing, so the output is of the format ``SHA256:<fingerprint>``. A new

``policy/protocols/ssh/md5-host-key-logging.zeek`` script was added to restore the old

field if desired.

- The ``auth_ticket`` and ``new_ticket`` fields in krb.log were removed and replaced

with new ``auth_ticket_sha256`` and ``new_ticket_sha256`` fields. The old fields held

MD5 hashes of the tickets and the new fields hold SHA256 hashes of the same. A new

``policy/protocols/krb/md5-ticket-logging.zeek`` script was added to restore the old

fields if desired.

- A new ``mime_content_hash_sha256`` event was added to send a SHA256 version of the

content in MIME messages. This is similar to the existing ``mime_content_hash`` event

that sends MD5 versions.

- The MD5 IP anonymizers (``RANDOM_MD5`` and ``PREFIX_PRESERVING_MD5``) have been

deprecated in favor of new SHA256 anonymizers.

- The ``md5_hmac`` BIF has been deprecated in favor of a new ``sha256_hmac`` BIF.

New Functionality

-----------------

- After recently-merged fixes to the ``node`` formula for Homebrew, JavaScript support via

ZeekJS can now be built on macOS using that installation of Node.

- A new ``TapAnalyzer`` class was added allowing to tap into all packets delivered to

child analyzers attached to session adapters.

- Two new hooks, ``Cluster::on_subscribe()`` and ``Cluster::on_unsubscribe()`` have been

added to allow observing ``Subscribe()`` and ``Unsubscribe()`` calls on backends by Zeek

scripts.

- The ability to control the length of strings and containers in log output was added. The

maximum length of individual log fields can be set, as well as the total length of all

string or container fields in a single log record. This feature is controlled via four

new script-level variables:

Log::default_max_field_string_bytes

Log::default_max_total_string_bytes

Log::default_max_field_container_elements

Log::default_max_total_container_elements

When one of the ``field`` limits is reached, the individual field is truncated. When one

of the ``total`` limits is reached, all further strings will returned as empty and all

further container elements will not be output. See the documentation for those variables

for more detail.

The above variables control the truncation globally, but they can also be set for log

streams individually. This is controlled by variables with the same names that can be

set when the log stream is created.

Two new weirds were added to report the truncation: ``log_string_field_truncated`` and

``log_container_field_truncated``. New metrics were added to track how many truncations

have occurred: ``zeek_log_writer_truncated_string_fields_total`` and

``zeek_log_writer_truncated_containers_total``. The metrics are reported for each log

stream.

- There exists a new configuration file ``<PREFIX>/etc/zeek/zeek.conf`` and a small

executable called ``zeek-systemd-generator`` that acts as a systemd unit file

generator. These two pieces allow for quickly running an opinionated single node cluster

deployment using systemd. See the ``tools/systemd-generator/README.md`` and

``tools/systemd-generator/etc/zeek/zeek.conf`` files for more information.

If you've previously built Zeek process management by hand or with tools like

supervisord, runit, s6, etc on Linux, this might be a viable option to explore.

This is all Linux specific right now. However, the ``zeek.conf`` file is OS-agnostic to

keep options open for re-use it elsewhere.

- The DNS analyzer now returns the set of parameters for SVCB data. It previously handled

SVCB packets, but omitted the parameters while parsing.

- The QUIC analyzer now raises a ``QUIC::discarded_packet`` event when a packet with

``fixed_bit`` set to 0 is encountered. Such an occurrence is included in the QUIC

history as ``X``. This functionality can be controlled with

``QUIC::max_discarded_packet_events``. Setting this variable to -1 disables the

``QUIC::discarded_packet`` event.

- Added SHA224, SHA384, and SHA512 calculation BiFs: ``sha224_hash``, ``sha224_hash_init`,

``sha224_hash_update``, ``sha224_hash_finish``. ``sha384_hash``, ``sha384_hash_init`,

``sha384_hash_update``, ``sha384_hash_finish``. ``sha512_hash``, ``sha512_hash_init`,

``sha512_hash_update``, and ``sha512_hash_finish``.

- Added SHA224, SHA384 and SHA512 file hash analyzers. The analyzers are not enabled by

default.

- Added the ``TCP::raw_options`` BiF, which returns the list of TCP options in the current

packet, including their raw data.

- For SNMPv3, Zeek now parses the Security Parameters for the User-Based Security

model. If present, the data is added to the new ``user_security_parameters`` field of

the ``SNMP::HeaderV3`` record.

- The QUIC analyzer now raises ``QUIC::short_header_packet_threshold_crossed()`` for every

binary logarithmic threshold for QUIC packets with short headers. Each threshold is

added to the history as ``O`` or ``o``. E.g., a history containing ``oOoO`` indicates

that originator and responder each send at least 3 short packets, while ``oOOoooooo``

would indicate the originator sent at least 3 short packets early in the connection,

while the responder sent at least 127 packets.

- Added SHA256 calculation BiFs: ``sha512_hash``, ``sha512_hash_init`,

``sha512_hash_update``, and ``sha512_hash_finish``.

- Added VLAN PCP and DEI bits when parsing packets. This is exposed alongside ``vlan`` and

``inner_vlan`` in a packet's ``l2_hdr`` when scripting, if a VLAN header was present.

- The DNS analyzer now supports parsing Dynamic Update messages (RFC 2136) and reporting

information about them into ``dns.log``. The ``answer`` field in the log will contain

the prerequisites and updates for the dynamic update requests, whereas the ``query``

field logs the zone name for the update request.

This also adds new ``opcode`` and ``opcode_name`` fields to the DNS log. Previously Zeek

was only handling opcode 0 (normal requests) so that information wasn't pertinent, but

with the addition of dynamic updates we're now handling a second opcode. The new policy

script ``policy/protocols/dns/disable-opcode-log-fields.zeek`` can be loaded to remove

these fields from the log if they are not desired.

As part of these additions, an extra field ``is_netbios`` was added to the the

``dns_msg`` script-layer record. This boolean flag is ``T`` if the DNS message is from

NetBIOS, and ``F`` if the message is standard DNS.

- The `policy/misc/dump-events.zeek` script now features a ``DumpEvents::use_json``

boolean toggle, false by default, to report Zeek's events in JSON format.

- The list of HTTP methods accepted by dynamic protocol detection was updated to add some

missing methods: BIND, LINK, MKCALENDAR, MKREDIRECTREF, REBIND, UNBIND, UNLINK.

- The Docker images generated by Zeek's CI process now contain OCI-approved annotation

labels containing various metadata about the image. See

https://github.com/opencontainers/image-spec/blob/main/annotations.md for more details

about the labels.

- A new packet source was added that supports reading pcapng files. This supports packet

captures with multiple interfaces, packet metadata, and capture file metadata. It will

be automatically used when the magic number in the file header is detected, or if

``pcapng::`` is passed as a prefix to the filename with Zeek's ``-r`` argument. It does

not currently support BPF filtering via Zeek's ``-f`` argument.

Changed Functionality

---------------------

- A Zeekctl-managed Zeek cluster now uses the ZeroMQ cluster backend by default.

- The manager node of a Zeekctl-managed Zeek cluster now listens on 127.0.0.1:27759 for

WebSocket connections by default. This can be disabled with the ``UseWebSocket`` setting

in ``zeekctl.cfg``, or adapted using the ``WebSocketHost`` and ``WebSocketPort``

settings.

- The var-extraction-uri.zeek policy does not include the path in the ``uri_vars`` field

anymore.

- The ``get_current_packet_header()`` BIF now populates the returned record also for

fragmented IP datagrams.

- The QUIC parser discards packets with the ``fixed_bit`` field set to 0, rather than

continuing to parse and potentially running into analyzer violations.

- The Notice framework's suppression implementation was changed to batch suppression

information for up up to 10msec before distributing it to other nodes in a Zeek cluster.

This reduces cluster communication overhead when there's a storm of notices

generated. Two new options, ``Notice::suppression_batch_period`` and

``Notice::suppression_batch_max_size`` can be used to tune this behavior. See their

documentation for details. This most notably reduces CPU usage and cluster event traffic

during cluster startup.

- Non-printable ASCII control characters now always result in ``\uXXXX`` characters when

logging them as JSON. This follows the JSON standard more closely. Previously, they

would have been logged in the ``\xXX`` format that Zeek typically uses. For example, an

HTTP request with a non-printable character in the URI would have a ``uri`` field that

changes from:

"uri":"/non_printable_\\x07"

to

"uri":"/non_printable_\u0007"

- For SNMPv3, the ``community`` field of ``snmp.log`` now contains the SNMPv3 username (if

present). Previously, the field was empty for all SNMPv3 connections.

- The ``x509.log`` deduplication logic has been reimplemented with explicit calls to

``Cluster::publish()``, rather than relying ``&backend`` tables using Broker.

- Sanitizer flags passed to a Zeek build with ``configure``'s ``--sanitizers`` argument

are now automatically passed to plugins built against that Zeek source tree.

- The ``hash-all-files`` policy script now enables SHA256 logging, in addition to MD5 and

SHA1.

- The LDAP analyzer was fixed to correctly escape non-ASCII characters when logging filter

strings.

- The AF_Packet packet source now reports a warning if you attempt to use it on a loopback

interface. This is because AF_PACKET will receive duplicates of every packet when run on

a loopback interface and this can be confusing.

- The GSSAPI analyzer gained some fixes to reliably process Kerberos data when that data

isn't preceded by an OID. This is valid a ASN.1 structure that wasn't being handled

correctly.

- SSL certificates are now correctly logged even when protocol violations occur, assuming

the certificate was encountered prior to the violation.

Deprecated Functionality

------------------------

- The ``&backend`` and ``&broker_store`` attributes for script-layer tables have been

deprecated. These are only functional with the Broker cluster backend and their

semantics and guarantees considered too loose. There is ongoing discussion and

experiments how to provide state propagation and synchronization across cluster

processes. Explicit topic-based state propagation using ``Cluster::publish()``, or

switching to the storage framework for persistence are available today.

If you've used ``&backend`` and ``&broker_store`` in your scripts and find it hard or

inconvenient to replace them, please reach out with your use-cases, on GitHub or

Slack. We're happy to support.

Some references to GitHub discussion and prototypical ideas:

https://github.com/zeek/zeek/discussions/5020

https://github.com/zeek/zeek/pull/5040

- In Zeek 9.1, returning a value from a hook will become an error. Return values from a

hooks were previously silently discarded. This behavior was found surprising in relation

with the ``break`` statement available in hooks.

- The ``zeek::util::json_escape_utf8`` methods are being replaced by a new ``escape_utf8``

method in order to fix a bug with how the ``escape_printable_controls`` argument was

implemented. Previously, it was implemented in the reverse of what was intuitive given

the name. It would escape the control characters when false, and not escape them when

true. The new method has the correct semantics, as well as a new argument for

controlling whether non-printable control characters are escaped.

- The ``zeromq/connect`` script has been deprecated. Load the

``frameworks/cluster/backend/zeromq`` package directly instead.

- The ``rfb_auth_result`` event from the RFB analyzer was deprecated and replaced with a

new ``rfb_authentication_result`` event. This new event sets the correct value in the

``rfb$auth`` field in the connection.

Zeek 8.0.0

==========

We would like to thank @aidans111, Anthony Verez (@netantho), Baa (@Baa14453),

Bhaskar Bhar (@bhaskarbhar), @dwhitemv25, EdKo (@ephikos), @edoardomich, Fupeng

Zhao (@AmazingPP), hendrik.schwartke@os-s.de (@hendrikschwartke), @i2z1, Jan

Grashöfer (@J-Gras) Jean-Samuel Marier, Justin Azoff (@JustinAzoff), Mario D

(@mari0d), Markus Elfring (@elfring), Peter Cullen (@pbcullen), Sean Donaghy,

Simeon Miteff (@simeonmiteff), Steve Smoot (@stevesmoot), @timo-mue,

@wojciech-graj, and Xiaochuan Ye (@XueSongTap) for their contributions to this

release.

Breaking Changes

----------------

- Zeek by default now depends on the availability of the ZeroMQ library for building

and running. This is in preparation of switching to the ZeroMQ-based cluster backend

by default in future Zeek versions. On an Ubuntu based system, the required system

packages are ``libzmq5``, ``libzmq3-dev`` and ``cppzmq-dev``. See the Dockerfiles

in the ``ci/`` directory for other supported platforms.

- Zeek and all of its associated submodules now require C++20-capable compilers to

build. This will let us move forward in using more modern C++ features and replace some

workarounds that we have been carrying. Minimum recommended versions of compilers are

GCC 10, Clang 8, and Visual Studio 2022.

- The ``zeek::Span`` class has been deprecated and the APIs in the telemetry subsystem

switched to use ``std::span`` instead of ``zeek::Span``. If your plugin instantiates

counter or gauge instances using the telemetry subsystem and you've previously used

``zeek::Span`` explicitly, updates may be needed.

- The code base underwent a big cleanup of #include usage, across almost all of the

files. We tested builds of all of the existing third-party packages and only noticed one

or two failures, but there is a possibility for breakage related to this cleanup.

- The ``lookup_connection()`` and ``connection_exists()`` builtin functions

now require ``conn_id`` instances as argument, rather than internally supporting

duck type matching ``conn_id``-like records.

- Network timestamps are not added to events by default anymore. Use the following

redef line to enable them:

redef EventMetadata::add_network_timestamp = T;

The background is that event metadata has become more generic and may incur

a small overhead when enabled. There's not enough users of network timestamp

metadata to justify the complexity of treating it separate.

- The ASCII writer's ``JSON::TS_MILLIS`` timestamp format was changed to produce

signed integers. This matters for the representation for timestamps that are

before the UNIX epoch. These are now written as negative values, while previously

the negative value was interpreted as an unsigned integer, resulting in very large

timestamps, potentially causing issues for downstream consumers.

If you prefer to always have unsigned values, it's possible to revert to the previous

behavior by setting:

redef LogAscii::json_timestamps = JSON::TS_MILLIS_UNSIGNED;

- The "endpoint" label of metrics exposed via Prometheus or the ``telemetry.log``

was renamed to "node". This is done for consistency with cluster terminology:

The label values have always been the value of ``Cluster::node`, so it's more intuitive

to call it. The "endpoint" name originated from a time when the telemetry framework

was implemented in Broker.

To revert to the "endpoint" label, you can do the following, but we strongly

suggest to migrate to the new default "node" instead:

redef Telemetry::metrics_endpoint_label = "endpoint";

- The ``current_event_time()`` builtin function as well as ``Event::Time()``

and ``EventMgr::CurrentEventTime()`` now return ``-1.0`` if no timestamp

metadata is available for the current event, or if no event is being

dispatched. Previously this would've been 0.0, or the timestamp of the previously

dispatched event.

- Missing network timestamp metadata on remote events is not set to the local

network time anymore by default. This potentially hid useful debugging information

about another node not sending timestamp metadata. The old behavior can be

re-enabled as follows:

redef EventMetadata::add_missing_remote_network_timestamp = T;

- The ``IsPacketSource()`` method on ``IOSource`` was removed. It was unused

and incorrectly returned ``false`` on all packet sources.

- The ``--with-binpac`` and ``--with-bifcl`` arguments for ``configure`` are now

deprecated. Both arguments have for a long time just used the internal version of the

tooling even if something was passed, so they were mostly useless. This may cause

breakage of cross-compiling, where the ``binpac`` and ``bifcl`` tooling needs to be run

on the host machine. We haven't heard from anyone that this is the case with the

arguments in their currently-broken state.

- The parsing of data for the ``ssl_session_ticket_handshake`` event was fixed.

In the past, the data contained two extra bytes before the session ticket

data. The event now contains only the session ticket data. You might have to

adjust your scripts if you manually worked around this bug in the past.

New Functionality

-----------------

- Zeek now supports pluggable and customizable connection tracking. The default

behavior remains unchanged and uses a connection's five tuple based on the

IP/port pairs and proto field. Zeek 8 ships with one additional implementation,

to factor VLAN tags into the connection tracking. To switch to VLAN-aware

connection tracking:

@load frameworks/conn_key/vlan_fivetuple

By convention, additional fields used by alternative ConnKey implementations are

added into the new ``ctx`` field of ``conn_id``. The type of ``ctx`` is ``conn_id_ctx``.

The ``vlan_fivetuple`` script adds two additional fields to the ``conn_id_ctx``

record type, representing any VLAN tags involved. Accordingly, every log

using ``conn_id`` reflects the change as well as ``ctx`` and the VLAN fields have

the ``&log`` attribute. The columns used for logging will be named ``id.ctx.vlan``

and ``id.ctx.inner_vlan``.

This feature does not automatically provide a notion of endpoint that

corresponds with the effective connection tuple. For example, applications tracking

endpoints by IP address do not somehow become VLAN-aware when enabling

VLAN-aware tracking.

Users may experiment with their own notion of endpoint by combining the ``orig_h``

or ``resp_h`` field of ``conn_id`` with the new ``ctx`` field. For example, tracking

the number of connections from a given host in a VLAN-aware fashion can be done

as follows:

global connection_counts: table[conn_id_ctx, addr] of count &default=0;

event new_connection(c: connection) {

++connection_counts[c$id$ctx, c$id$orig_h];

}

Note that this script snippet isn't VLAN-specific, yet it is VLAN-aware if the

``vlan_fivetuple`` script is loaded. In future Zeek versions, this pattern is

likely to be used to adapt base and policy scripts for more "context awareness".

Users may add their own plugins (for example via a zkg package) to provide

alternative implementations. This involves implementing a factory for

connection "keys" that factor in additional flow information. See the VLAN

implementation in the ``src/packet_analysis/protocol/ip/conn_key/vlan_fivetuple``

directory for an example.

- Added support to ZeekControl for seamlessly switching to ZeroMQ as cluster

backend by adding the following settings to zeekctl.cfg:

ClusterBackend = ZeroMQ

UseWebSocket = 1

With the ZeroMQ cluster backend, Zeekctl requires to use Zeek's WebSocket API

to communicate with individual nodes for the ``print`` and ``netstats`` commands.

Setting the ``UseWebSocket`` option enables a WebSocket server on the manager

node, listening on 127.0.0.1:27759 by default (this is configurable with using

the newly introduced ``WebSocketHost`` and ``WebSocketPort`` options).

The ``UseWebSocket`` option can also be used when ``ClusterBackend`` is set

to ``Broker``, but isn't strictly required.

For ZeroMQ (or other future cluster backends), setting ``UseWebSocket`` is a

requirement as Zeekctl does not speak the native ZeroMQ protocol to communicate

with cluster nodes for executing commands. This functionality requires the

``websockets`` Python package with version 11.0 or higher.

- Cluster telemetry improvements. Zeek now exposes a configurable number of

metrics regarding outgoing and incoming cluster events. By default, the number

of events sent and received by a Zeek cluster node and any attached WebSocket

clients is tracked as four individual counters. It's possible to gather more

detailed information by adding ``Cluster::Telemetry::VERBOSE`` and

``Cluster::Telemetry::DEBUG`` to the variables ``Cluster::core_metrics`` and

``Cluster::webscoket_metrics``:

redef Cluster::core_metrics += { Cluster::Telemetry::VERBOSE };

redef Cluster::websocket_metrics += { Cluster::Telemetry::DEBUG };

Configuring verbose, adds metrics that are labeled with the event handler

and topic name. Configuring debug, uses histogram metrics to additionally track

the distribution of the serialized event size. Additionally, when debug is selected,

outgoing events are labeled with the script location from where they were published.

- Support for the X-Application-Name HTTP header was added to the WebSocket API at

``v1/messages/json``. A WebSocket application connecting to Zeek may set the

X-Application-Name header to a descriptive identifier. The value of this header

will be added to the cluster metrics as ``app`` label. This allows to gather

incoming and outgoing event metrics of a specific WebSocket application, simply

by setting the X-Application-Name header.

- The SMTP analyzer can now optionally forward the top-level RFC 822 message individual

SMTP transactions to the file analysis framework. This can be leveraged to extract

emails in form of ``.eml`` files from SMTP traffic to disk.

To enable this feature, set the ``SMTP::enable_rfc822_msg_file_analysis`` option

and implement an appropriate ``file_new()`` or ``file_over_new_connection()`` handler:

redef SMTP::enable_rfc822_msg_file_analysis = T;

event file_over_new_connection(f: fa_file, c: connection, is_orig: bool) {

if ( f$id == c$smtp$rfc822_msg_fuid )

Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename="email"]);

}

- Generic event metadata support. A new ``EventMetadata`` module was added allowing

to register generic event metadata types and accessing the current event's metadata

using the functions ``current()`` and ``current_all()`` of this module.

- A new plugin hook, ``HookPublishEvent()``, has been added for intercepting

publishing of Zeek events. This hook may be used for monitoring purposes,

modifying or rerouting remote events.

Plugins can implement and enable this hook by calling the following method

within their Configure() implementation.

EnableHook(HOOK_PUBLISH_EVENT)

The signature of ``HookPublishEvent()`` is as follows.

bool HookPublishEvent(zeek::cluster::Backend& backend,

const std::string& topic,

zeek::cluster::detail::Event& event);

- Zeek now includes the Redis protocol analyzer from the evantypanski/spicy-redis

project (https://github.com/evantypanski/spicy-redis). This analyzer is enabled

by default. This analyzer logs Redis commands and their associated replies in

``redis.log``.

To disable the analyzer in case of issues, use the following snippet:

redef Analyzer::disabled_analyzers += {

Analyzer::ANALYZER_REDIS,

};

- The FTP analyzer now supports explicit TLS via AUTH TLS.

- Two new script-level hooks in the Intel framework have been added.

hook indicator_inserted(indicator_value: string, indicator_type: Intel::Type)

hook indicator_removed(indicator_value: string, indicator_type: Intel::Type)

These are reliably invoked on worker and manager nodes the first time an

indicator value is inserted into the store and once it has been completely

removed from the store.

- The ``frameworks/intel/seen`` scripts have been annotated with event groups

and a new ``frameworks/intel/seen/manage-event-groups`` policy script added.

The motivation is to allow Zeek distributors to load the ``intel/seen`` scripts

by default without incurring their event overhead when no Intel indicators are

loaded. Corresponding event handlers are enabled once the first Intel indicator

of a given ``Intel::Type`` is added. Event handlers are disabled when the last

indicator is removed, again.

Note that the ``manage-event-groups`` script interacts with the ``Intel::seen_policy``

hook: If no indicators for a given ``Intel::Type`` are loaded, the ``Intel::seen_policy``

will not be invoked as the event handlers extracting indicators aren't executed.

If you rely on the ``Intel::seen_policy`` hook to be invoked regardless of the

contents of the Intel store, do not load the ``manage-event-groups`` or set:

redef Intel::manage_seen_event_groups = F;

- The DNS analyzer was extended to support NAPTR RRs (RFC 2915, RFC 3403).

A corresponding ``dns_NAPTR_reply`` event was added.

- A new ``get_tags_by_category`` BIF method was added that returns a list of tags for a

specified plugin category. This can be used in lieu of calling ``zeek -NN`` and

parsing the output. For example, this will return the list of all analyzer plugins

currently loaded:

get_tags_by_category("ANALYZER");

- A new ``conn_generic_packet_threshold_crossed`` event was introduced. The event triggers

for any IP-based session that reaches a given threshold. Multiple packet thresholds can

be defined in ``ConnThreshold::generic_packet_thresholds``. The generic thresholds refer

to the total number of packets on a connection without taking direction into account

(i.e. the event also triggers on one-sided connections).

The event is intended as an alternative to the ``new_connection`` event that allows for

ignoring short-lived connections like DNS or scans. For example, it can be used to set

up traditional connection monitoring without introducing overhead for connections that

would never reach a larger threshold anyway.

- Zeek now supports extracting the PPPoE session ID. The ``PacketAnalyzer::PPPoE::session_id``

BiF can be used to get the session ID of the current packet.

The ``conn/pppoe-session-id-logging.zeek`` policy script adds pppoe session IDs to the

connection log.

The ``get_conn_stats()`` function's return value now includes the number of packets

that have not been processed by any analyzer. Using data from ``get_conn_stats()`` and

``get_net_stats()``, it's possible to determine the number of packets that have

been received and accepted by Zeek, but eventually discarded without processing.

Changed Functionality

---------------------

- The `Conn::set_conn` function is now always run in `new_connection`, instead of only

being run in `connection_state_remove`.

- Logging of failed analyzers has been overhauled. `dpd.log` was replaced

by a new `analyzer.log` that presents a more unified and consistent view

of failed analyzers. The previous `analyzer.log` was renamed to `analyzer-debug.log`;

see below for more details.

For protocol analyzers, `analyzer.log` now reports initially confirmed analyzers that

Zeek subsequently removed from the connection due to a protocol violation.

For file and packet analyzers, all errors will be logged to `analyzer.log`.

As part of this work, a new `analyzer_failed` event has been introduced. This event

is raised when an analyzer is removed because of raising a violation.

- `analyzer.log` was renamed to `analyzer_debug.log`, and is no longer created

by default. The log file will be created if the `frameworks/analyzer/debug-logging.zeek`

policy script is loaded.

Note that the namespace for options in the script changed to

`Analyzer::DebugLogging`. Furthermore the default options changed to enable

more detailed output by default.

- Record fields with a ``&default`` attribute are now consistently re-initialized

after deleting such fields. Previously, this would only work for constant

expressions, but has been extended to apply to arbitrary expressions.

- Publishing remote events with vector arguments that contain holes is now

rejected. The receiver side never had a chance to figure out where these

holes would have been. There's a chance this breaks scripts that accidentally

published vectors with holes. A reporter error is produced at runtime when

serialization of vectors with holes is attempted.

- Kerberos support on macOS has been enabled. Due to incompatibilities, the system

provided libkrb5 is ignored, however. Only versions from homebrew are supported and

found/picked-up by default. Use --with-krb5 for pointing at a custom librkb5

installation.

- The ``$listen_host`` configuration for ``Cluster::listen_websocket()``'s

``WebSocketServerOptions`` was deprecated. Use the new ``$listen_addr`` field

instead.

- The `service_violation` field of the connection record was marked as deprecated.

Consider using the new `failed_analyzers` field of the connection record instead.

- `detect-protocol.zeek was the last non-deprecated policy script left in

`frameworks/dpd`. It was moved to `frameworks/analyzer/detect-protocol.zeek`.

- Running Zeek with Zeekygen for documentation extraction (-X|--zeekygen

<cfgfile>) now implies -a, i.e., parse-only mode.

- The `not_valid_before` and `not_valid_after` times of X509 certificates are

now logged as GMT timestamps. Before, they were logged as local times; thus

the output was dependent on the timezone that your system is set to.

Similarly, the related events and the Zeek data structures all interpreted

times in X509 certificates as local times.

- The PPPoE parser now respects the size value given in the PPPoE header. Data

beyond the size given in the header will be truncated.

- Record fields with ``&default`` attributes initializing empty ``vector``, ``table``

or ``set`` instances are now deferred until they are accessed, potentially

improving memory usage when such fields are never accessed.

- The handling of truncated 802.3 Ethernet frames changed. Before, truncated 802.3 frames

were always discarded. Now, truncated 802.3 Ethernet frame data is forwarded to downstream

analyzers. This aligns the handling of 802.3 Ethernet frames with the (much more common)

Ethernet 2 frames which are forwarded when truncated.

Due to this change, the ``truncated_vlan_frame`` was removed. 802.3 frames with a truncated

header now raise the ``truncated_VLAN_header`` weird, which is similarly raised by Ethernet 2

frames with a truncated VLAN header.

- Data beyond the length indicated in the header of an 802.3 Ethernet frame is no longer

forwarded to downstream analyzers. This aligns our handling of 802.3 Ethernet frames with

other protocols that have length headers.

Removed Functionality

---------------------

- The ``--with-bind`` argument for ``configure`` was removed. We removed the need for the

BIND library from our CMake setup in the v7.2 release, but this non-functional argument

was left behind.

- The ``--disable-archiver`` argument for ``configure`` was removed. This was deprecated

and scheduled to be removed in v7.1, but we apparently missed it during the cleanup for

that release.

Deprecated Functionality

------------------------

- The `dpd.log` is now deprecated and replaced by `analyzer.log` (see above).

`dpd.log` is no longer created by default, but can be loaded using the

`frameworks/analyzer/deprecated-dpd-log.zeek` policy script.

Relatedly, the `service_violation` field of the connection record is

deprecated and will only be present if the

`frameworks/analyzer/deprecated-dpd-log.zeek` policy script is loaded.

- The ``protocols/http/detect-sqli.zeek`` script has been deprecated in favor of a

new ``protocols/http/detect-sql-injection.zeek`` script to switch from the victim

host being placed into the ``src`` field of a notice to instead use ``dst``.

The attacker host is now placed into ``src``. Further, notices hold the first

sampled connection uid.

Note that the ``Notice::Type`` enumeration names remain the same. You can determine

which script was used by the presence of populated ``uid`` and ``dst`` fields in the

``notice.log`` entries.

The replacement script doesn't populate the ``email_body_sections`` anymore either.

- Using ``&default`` and ``&optional`` together on a record field has been deprecated

as it would only result in ``&default`` behavior. This will become an error starting

with Zeek 8.1.

- The ``zeek::Event()`` constructor was deprecated. Use ``event_mgr::Enqueue()``

or ``event_mgr::Dispatch()`` instead.

- Passing ``ts`` as the last argument to ``EventMgr::Enqueue()`` has been deprecated

and will lead to compile time warnings. Use ``EventMgr::Enqueue(detail::MetadataVectorPtr meta, ...)``

for populating ``meta`` accordingly.

- For plugin authors: in the core, the constructor for Connection instances has

been deprecated in favor of a new one to support pluggable connection

tuples. The ConnTuple struct, used by this deprecated Connection constructor,

is now deprecated as well.

- The ``zeek::filesystem`` namespace alias is deprecated in favor of using

``std::filesystem`` directly. Similarly, the ``ghc::filesystem`` submodule stored in

``auxil/filessytem`` has been removed and the files included from it in the Zeek

installation will no longer be installed. Builds won't warn about the deprecation of

``zeek::filesystem`` due to limitations of how we can mark deprecations in C++.

- The ``zeek::util::starts_with`` and ``zeek::util::ends_with`` functions are deprecated.

``std::string`` and ``std::string_view`` added ``begins_with`` and ``ends_with`` methods

in C++ 20, and those should be used instead.

- The ``record_type_to_vector`` BIF is deprecated in favor of using the newly ordered

``record_fields`` BIF.

Zeek 7.2.0

==========

We would like to thank Aashish Sharma (@initconf), Anthony Verez (@netantho), Anthony

Kasza (@anthonykasza), @biswajitutil, Brendan Kapp (@BrendanKapp), Carlos Lopez, Chris

Hinshaw (@MMChrisHinshaw), Faan Rossouw (@faanross), @FishyFluffer, Fupeng Zhao

(@AmazingPP), Herbert (@Herbert-Karl), @jbaggs, Jan Grashöfer (@J-Gras), Julian Krieger

(@juliankrieger), Justin Azoff (@JustinAzoff), Kshitiz Bartariya (@kshitiz56), @Laotree,

Mark Overholser (@markoverholser), Mike Dopheide (@dopheide-esnet), @mnhsrj, Mohan Dhawan

(@Mohan-Dhawan), @philipp-tg, Seth Hall (@sethhall), and @timo-mue for their contributions

to this release.

Breaking Changes

----------------

- The ``is_remote_event()``, ``current_analyzer()`` and ``current_event_time()`` builtin

functions do not return the previous event's values anymore when event draining has

completed. The same applies to the corresponding C++ accessors on the ``EventMgr``

class. The functions now return false, 0 or the zero time instead.

- The ``to_int()`` built-in function was changed to match the return behavior of

``to_count()``. Previously, ``to_int()`` would silently ignore invalid inputs and return a

``0``. It now returns an error instead.

New Functionality

-----------------

- The following dependencies have had updates:

- The bundled version of c-ares has been updated to v1.34.5.

- The bundled version of ZeekJS has been updated to v0.17.0.

- Some DNS events are not raised when ``dns_skip_all_addl`` is set to true. Zeek now

raises a warning when a script declares these events while this option is set to true.

- Types can now be used as constants in Zeek script. This allows types to be directly

passed into BIFs without aliasing.

- A new ``enc_part`` field was added to the Kerberos ``KRB_Response`` record passed as

part of the ``krb_as_response`` event. This field contains the encrypted session

information from a Kerberos response, including the cipher and encrypted data.

- Geneve tunnel options of the current packet can be extracted from scripts using the new

``PacketAnalyzer::Geneve::get_options()`` builtin function.

- The new ``is_valid_subnet()`` function mirrors ``is_valid_ip()``, for subnets.

- A new Storage framework was merged into the Zeek tree. The intention with this framework

is to eventually replace the storage functionality that Broker provides, including

direct storage via calls such as ``Cluster::create_store`` and ``Broker::put_unique`` as

well as storage-backed tables via the ``&backend`` attribute. This is an initial version

for testing, and will be expanded upon in the future. The current state of the framework

is as follows:

- A new API was added for storage backend plugins.

- Script-level functions for opening and closing backends, and insertion, retrieval, and

erasure of elements are available.

- Backends can support both asynchronous mode (using ``when`` statements) and

synchronous mode (blocking until the operation completes). BIF methods were added

under new ``Storage::Async`` and ``Storage::Sync`` modules for these two modes. The

modes can be used interchangeably with the same backend handle.

- SQLite and Redis backends exist in the Zeek tree by default. We are working on a

backend for NATS that will be available as an external plugin, but it is not quite

ready yet. Both of the existing backends support usage in a cluster environment.

- Improved alternative cluster backend support.

The ZeroMQ cluster backend added in Zeek 7.1 has received various correctness,

performance and robustness fixes, particularly concerning shutdown and high-load

scenarios.

Initial performance testing indicates less CPU time used on a large single node

instance with high logging and eventing rates.

We're evaluating switching the default cluster backend from Broker to ZeroMQ With

Zeek 8.1. Therefore, we welcome early adopters and testers to validate ZeroMQ as an

alternative to Broker. If you're not using Broker specific integrations (e.g. Broker's

Python or C++ bindings) and run a single-node Zeek cluster, switching to ZeroMQ

should be as simple as loading the following script on each of cluster node.

@load frameworks/cluster/backend/zeromq/connect

A proof-of-concept plugin for the open-source NATS messaging system is available at

https://github.com/zeek/zeek-cluster-backend-nats for testing and experimentation.

- Broker now exposes more information through ``broker.log``. Broker generated log

messages are now propagated as events to Zeek. This allows exposing more information for

debugging and operational behavior of Broker via Zeek logs. Two new script-level

options ``Broker::log_severity_level`` and ``Broker::log_stderr_severity_level`` have

been introduced to control the which events to expose by default.

- Broker's new per-peer send buffer backpressure handling, introduced in 7.1,

has received several updates. We've increased the default buffer sizes to 8192

messages for both peers and websockets, and switched the default overflow

handling policy to "drop_oldest", meaning that in a full buffer the oldest

message enqueued gets dropped to allow enqueuing a new one. Three additional

metrics are available to understand the health of each peering's buffer,

regardless of the overflow policy active. These are:

- zeek_broker_peer_buffer_messages: a gauge of the current buffer fill level,

- zeek_broker_peer_buffer_recent_max_messages: a gauge that tracks the maximum

buffer fill level seen over the last ``Broker::buffer_stats_reset_interval`.

- zeek_broker_peer_buffer_overflows_total: a counter that tracks the number

of times a given peering's send buffer has overflowed. For the "drop_oldest"

and "drop_newest" policies, this is the count of messages dropped.

Each of these is labeled with the current endpoint and the peer's, as provided

by the cluster topology.

- New WebSocket functionality was added to Zeek's cluster component.

Users of Broker's WebSocket interface should replace their ``Broker::listen_websocket()``

usage with ``Cluster::listen_websocket()``. The latter will support any cluster

backends, while ``Broker::listen_websocket()`` is specific to Broker.

A crucial difference between these two methods is that ``Cluster::listen_websocket()``

will have TLS disabled by default, while Broker usually defaults to TLS enabled.

If you require a TLS configuration for your WebSocket deployment, pass an appropriate

``WebSocketTLSOptions`` record or setup a TLS reverse proxy in front of Zeek.

For WebSocket clients, there should not be an observable difference in behavior

regarding the handshake and publishing of events. However, the implementation uses