Logs are sent:

• in JSON format for easier parsing by the client.

  • Each log entry separated by a newline, so that each line can be parsed as a JSON object, without needing a JSON streaming parser.

• only if the level is greater or equal to WARNING.

• to the disk, under $JENKINS_VAR/logs/evergreen.log.N rotating files.

  • By rotating, we mean evergreen.log.0 is the one file to "tail" to receive new log entries. It is left to the consumer to decide how to know where to start from in case of a fresh start, or restart. Though it is a subject for another design document, it is suggested to:

    • store the last timestamp that was read ;

    • be able to idempotently send everything to the backend (in case of a fresh startup, or a missing value for the last read and sent timestamp), and that already sent logs are deduplicated or ignored there.

• in UTF-8 (The JSON specification allows some other encodings. We restrict it to UTF-8 only)

  • 5 files maximum ;

  • Approximately 10 MB maximum per file.

  • This should give largely enough time for the client to parse and send everything before it gets rotated, even more because only logs ≥ WARNING will be sent. A total maximum of 50 MB of disk consumption for all this seems very reasonable with modern disk sizes.

We will host the code for all this in a new Evergreen Jenkins Plugin that will be required to be installed and active on any Jenkins Evergreen instance.

See Packaging: Jenkins plugin or Jenkins module for why a Jenkins plugin.

The logs is sent in JSON format. A version tag is put with initial value 1. This will make it easier to evolve the format if needed.

The formatting is done by a JSONFormatter extending java.util.logging.Formatter.

version

Initially 1. Will be used in case we need to evolve the format.

timestamp

Timestamp for the log in Epoch format

name

The logger name. It is generally the name of the emitting class, but can be any text or even the empty string.

level

The level of criticality of the log. In practice, should always be SEVERE or WARNING. But it is expected that this could be any value of the java.util.logging.Level class, if this proves necessary in the future.

message

The message for this log entry.

exception

(optional) See below Exceptions handling.

The logs are written to disk in rotating log files under $JENKINS_VAR/logs. This uses a custom FileHandler.

The SocketHandler class provided by the JDK is not usable for any real life usage. There is no reconnection logic at all: this means that any even temporary unavailability of the server socket will break the connection, and no logs will ever be sent again when it becomes available.

So we chose the easiest path to not roll our own SocketHandler at least for now, and use a FileHandler and rotating log files instead for more out-of-the-box robustness (the producer can send its data, and the consumer can read it when ready).

We chose the Jenkins plugin path because this is a simpler path forward. The Jenkins Project has already everything ready to handle the hosting and release process of plugins.

Using a jenkins-module would mean we need to set up a custom Jenkins WAR build for Evergreen. This would also mean we cannot use the quality assurance improvements we have started to put in place in the same way for Evergreen, and for the standard Jenkins core delivery.

Though having a dedicated WAR packaging for Jenkins Evergreen is very likely something we will do in the future, we deemed preferrable to defer it to later, and focus instead on the other yet unresolved design questions of Evergreen.

Following YAGNI, we use a simple exception field where the exception is basically serialized as text. We will likely use hudson.Functions.printThrowable() ^[2] for the text formatting.

If needed, this might be replaced in the future by a richer structure. In such case we will bump the version field to a higher number.

As explained above, we put a version field in the JSON logs. This will help accomodate a format change if this becomes a necessity.

We might decide at some point to change the way we pass the logs from Jenkins to the Evergreen Client. If this happens, we will probably go through a period of time where we’d log both to the disk, and to the Socket, so that we are not dependent on the version of the client.

The biggest part of this feature should be tested directly in the Evergreen codebase. It is indeed easy to check that the logs are indeed found under the expected location. * See jenkins-infra/evergreen#43 for such example

The Evergreen Jenkins Plugin is going to start introducing some core changes to the way Jenkins logs things.

We want to check we do not inadvertently change, or disable totally for instance, the usual logging Jenkins has.