To see the release details, please check the the release notes.

More details about Radicle integration and Discord notifications are provided below (Read More).

Radicle​

Radilce leverages Git’s architecture combined with cryptography and a gossip protocol to enable a fully sovereign developer network. Unlike centralized platforms like GitHub, there is no single entity controlling the network or user data. Repositories are replicated across peers in a decentralized manner.

Radicle is reaching its maturity - the 1.0 version is going to be released in a few weeks.

Kraken CI now includes support to communicate with Radicle, enabling it to receive push events generated by a Radicle Node. These events trigger a build flow within Kraken CI.

The full guide about setting up integration of Kraken CI with Radicle can be found in the blog post Integration with Radicle.

Discord​

Another significat feature that has been added to this Kraken's release is notifications to Discord channels. Now the result of an executed run can be posted to indicated Discord channel. More details can be found in the notification docs.

That's all. To get started, simply install or upgrade to the 1.7 version and experience all the benefits firsthand. If you have any questions or need assistance, submit an issue on GitHub or ask for help on Discord.

Radilce leverages Git’s architecture combined with cryptography and a gossip protocol to enable a fully sovereign developer network. Unlike centralized platforms like GitHub, there is no single entity controlling the network or user data. Repositories are replicated across peers in a decentralized manner. More details about Radicle can be found it its Guides.

To arrange a CI process for Radicle there must be appointed Radicle server that will be used by Kraken CI server so:

• Kraken CI server can listen for webhooks coming from that Radicle server
• Kraken CI can provide links that lead to repositiories, commit changes, etc from Kraken CI web UI.

The configuration on Kraken CI side is described in Webhooks guide.

The details about setting up Radicle side are presented below. This description is heavily based on the documentation Radicle CI Integrations.

Integrating a CI system using Radicle Webhooks​

In order to add the desired webhook functionality, expected by Kraken CI, there must be run the Radicle CI Broker alongside the Radicle Node. Radicle CI Broker is responsible for grabbing any events that the Radicle Node generates and for each event it invokes the Radicle Webhooks Adapter that will invoke any registered outgoing webooks.

The second Radicle Node is required so that the first one can receive any updates from the network and then generate events so the broker will spawn any registered adapter.

Making changes to a node wont generate any events to the same node.

Radicle Setup​

The setup requires a Radicle Node that will be running alongside the Radicle CI Broker and the Radicle CI Broker that will invoke the webhooks adapter.

Another Radicle Node is required with network access to the previous one (through private, public IP - or even in the same host) that will make changes to a repository. This way the first node will generate events about the changes that the broker will grab and invoke the webhook adapter.

Radicle Nodes​

Radicle must be set up following one of the official installation methodologies from: https://radicle.xyz/#get-started. After the Radicle installation the rad auth command must be executed. More details about rad auth can be found here.

After this you will be able to start your node by running rad node start. At this point we all set!

You can run radicle-httpd and rad web so that you can access your Radicle Node through the web interface. radicle-httpd provides an HTTP API for managing the Radicle Node. rad web authenticates the Radicle Node using the radicle-httpd towards https://app.radicle.xyz/ so that users can manage their node through the web client.

Web client and API offers a subset of functionality compared to the rad cli.

You are able to setup and run more Radicle Nodes in the same host by altering the Radicle's home path through the RAD_HOME environment variable. So you can set up a new node using the RAD_HOME=~/.new_radicle rad auth. For the rad web you can provide the radicle-httpd URL to connect to rad web https://app.radicle.xyz --connect 127.0.0.1:8080.

So in order to setup two nodes in our host we have to run:

# Primary Node
rad auth
rad node start
radicle-httpd
rad web  # optionally append with: https://app.radicle.xyz --connect 127.0.0.1:8080

For the second node use

# Secondary Node
RAD_HOME=~/.secondary_radicle rad auth
RAD_HOME=~/.secondary_radicle rad node start

Radicle Webhooks Adapter​

As mentioned earlier, the Radicle CI Broker is responsible for grabbing the Radicle Node events and then invoke any registered adapter for any event. An adapter is responsible for conducting any tasks.

In our case, the Radicle Webhooks Adapter is responsible for invoking any registered webhooks when an event occurs.

The Radicle Webhooks Adapter binary does not run as a stand alone application. Radicle CI broker is responsible for spawning it with the appropriate input data.

Radicle Webhooks Adapter read the webhook configuration from the repository that the radicle node event occurred.

The Radicle Webhooks Adapter binary must have appropriate execute permissions.

The configuration that the adapter requires is:

More details about the radicle wbhooks adapter can be found at its repository.

Radicle CI Broker​

The Radicle CI Broker is responsible for reading events from the Radicle Node and invoke any registered adapter. Radicle CI Broker must be initiated and run at the background at all times in order to grab and process any new event from the Radicle Node. In our setup we should connect Radicle CI Broker to our primary node ie. #1.

The configuration of the broker should be stored in a .yaml file as follows:

$ cat ci-broker-cfg.yaml
default_adapter: webhooks
adapters:
  webhooks:
    command: radicle-webhooks-adapter
    env:
filters:

radicle-webhooks-adapter should be replaced with the full path of the Radicle webhooks adapter binary path.

Then the broker can be initiated through the command:

RUST_LOG=debug ./ci-broker ci-broker-cfg.yaml

ci-broker-cfg.yaml is the path of the yaml file that contains the broker's configuration.

More details about the Radicle CI Broker can be found at its repository.

Repository Setup​

Now that Radicle Primary Node with CI Broker is in place we have to properly setup our repository. We can choose to set it up to any of the available Radicle Nodes but we should seed it through the node that runs the broker.

Now it's time to initialize a Git repository. This will be done in our secondary node ie. #2. We do it as a normal Git repository through git init and then we should initialize it as a rad repository through RAD_HOME=~/.secondary_radicle rad init.

For more details about how to initialize a repository you can check this example.

Repository should contain the configuration for the webhooks. This file should be stored under the .radicle/webhooks/ directory in our repo. Each configuration file should be a .yaml (or .yml) that will contain all the details for each webhook. The schema of the file's content is like this:

outgoing_webhook_kraken_ci:
    payload_url: https://kraken-server/webhooks/25/radicle
    content_type: application/json
    shared_secret: <some-secret>

• outgoing_webhook_kraken_ci is the friendly name of the webhook;
• payload_url is the URL that will be invoked when an event occurs, e.g. https://kraken-server/webhooks/25/radicle where kraken-server should be replaced with real Kraken CI Server address and 25 with appropriate project ID in Kraken CI;
• content_type is the content type of the payload (currently only application/json is supported);
• shared_secret should be a high entropy secret key used to generate the SHA-256 hash signature using HMAC hex digest from the provided webhook's secret token alongside each payload. This result will be within the headers of the webhook request so that the recipient will be able to validate the request.

Multiple webhook configuration files are supported within the repository using multiple files or separating the entries using the three dashes ---. As a user stores all the webhooks' details within the repository these details must be somehow hidden to the rest of the users. So, these configuration files should be encrypted in order to ensure that the configuration data are not exposed.

In order to encrypt these files we should use the age file encryption tool. This tool permits the encryption of a file using multiple public keys and the decryption of each file using just one of the private keys. After the age tool is installed you can encrypt each file using the following command:

age -encrypt -R radicle1.pub -R radicle2.pub -R radicle3.pub -o destination_for_encrupted_file.yaml file_to_encrypt

The -R radicleX.pub is added for every public key we want to encrypt the file with. It is completely fine to use one or more keys. The keys that should e used are the radicle public/private keys located at $RAD_HOME/keys/ directory. We should encrypt the webhooks settings using the keys of the users that we permit to be able to view and propose changes to these settings. We should definitely use the key that the webhooks adapter will use in order to decrypt the webhooks' settings files. This key is located under the $RAD_HOME and it's passphrase should be at $RAD_PASSPHRASE.

The destination_for_encrupted_file.yaml is the file that should be committed to the repo. It's critical here to use the public key of the radicle node where the Radicle CI Broker runs in order to allow the adapter to decrypt it later.

For more details and examples for this process you can refer to the radicle-webhooks-adapter documentation.

For this setup there is no reason to follow the steps under the Automated encryption & decryption section but feel free to do so!

Now that we have our repository ready we have to ensure that the radicle node that runs alongside the broker (the primary node) is seeding the repository. Seeding a repository means that the Radicle node will seed that repo in the Radicle p2p network (as we seed in torrents). This can be done either by cloning the repo through the rad clode RAD_ID command or the rad seed RAD_ID command at the primary node (where the broker runs). RAD_ID is the repo's radicle ID. This can be retrieved through the rad ls command or by running the rad . command within the repository's folder.

By seeding a repository we ensure that the a node receives any events for the specific repo.

Testing​

We can now check our setup if everything works fine. On the Host #1 we should start the Radicle Node and the CI Broker. The broker will report some logs:

 Running `./ci-broker webhooks`
 INFO  ci_broker > Radicle CI broker starts
 DEBUG ci_broker > loaded configuration: Config {
    default_adapter: "webhooks",
    adapters: {
        "webhooks": Adapter {
            command: "/tmp/bin/radicle-webhooks-adapter",
            env: {},
        },
    },
    filters: [],
}
 INFO  radicle_ci_broker::event > subscribing to local node events

You can use a webhook listener service (such as the https://webhook.site/) in order to register the provided URL and check the requested payload alongside any headers.

Push​

From the secondary node we should push some changes to the repo. Pushing will automatically sync with other nodes and will print something like in the broker's log output:

$ git push
✓ Patch b1d3420 updated to 54884569f96ac1db8ea1fb19004a536bb6e0ab83
✓ Synced with 2 node(s)
To rad://z32iyJDyFLqvPFzwHm8YadK4HQ2EY/z6MksMpnzPF48pk4XAnqVotKmfs2SE3bxA57UA8KL9DnWnY3
   a680e4d..6a76ee5  ha -> patches/b1d3420337dc4817cb9f2c2a0320e60b11530a25

If the sync fails we can retry it through the RAD_HOME=~/.secondary_radicle rad sync command. A successful sync will generate some events to the broker.

If the webhook configuration is properly set up we can see in the logs the webhook URL printed out. We can see that we will receive two events from the broker which will make our broker invoke the webhooks for these two events. One event will be about the update of the sigrefs and the other one is for the actual push event on the repo.

And then in Kraken CI side we should see a triggered flow in a project associated with the repositored hosted in Radicle. On the flow page, in Repo Changes, there are information retrieved from Radicle Push event. We can visit hosted Git repo in Radicle or visit the change commits.

The 1.6 release brings several more changes and fixes. To see the details, please check the the release notes. Some of the changes included are:

• added link to docs page in the top bar
• added checking agents group correctness provided in schema
• several fixes

More details about the support for Windows in AWS EC2 are provided below (Read More).

There are a few requirements for an AWS AMI Windows image to make it work with Kraken CI. It should have preinstalled Python and Git. The generated image should also have the ability to run a user data script on the first boot. This script is used to install the Kraken Agent and your own init script (which can be passed via Kraken).

Here, you can find an example script for Ansible that prepares an AWS AMI image for Windows with all the dependencies included: https://github.com/Kraken-CI/kraken/tree/master/base-images/aws-ec2-win-ansible

The full guide about setting up Kraken execution agents in AWS EC2 can be found in Autoscale in Cloud docs.

That's all. To get started, simply install or upgrade to the 1.6 version and experience all the benefits firsthand. If you have any questions or need assistance, submit an issue on GitHub or ask for help on Discord.

The 1.5 release brings several more changes and fixes. To see the details, please check the the release notes. Some of the changes included are:

• fixed detecting Kraken Agent environment when it is run inside Docker that is using CGroup v2 - this solves #303
• fixed triggering a flow by GitHub webhook when another flow is in progress - this solves #296
• fixed handling multiline secrets
• improved handling agent inactivity - now it is not needed to reenable agent when it was disabled due to inactivity

More details about the support for Windows are provided below.

Information about the Kraken Agent installation on Windows can be found in Agent Setup Guide.

With enabling Kraken Agent on Windows, support for this system was also added to some Kraken tools:

• shell tool
• artifacts tool

This means that now it is possible to execute windows shell commands.

The following example shows how to define a shell or, rather, cmd batch script:

  {
    "tool": "shell",
    "script": """
       rem This is a windows batch comment
       echo 'hello world'
       dir c:\\
    """
  }

or this example showing PowerShell as custom shell executable:

  {
    "tool": "shell",
    "shell_exe": "powershell.exe",
    "script": """
       Write-Host 'hello'
       Get-Service kkagent
    """
  }

The Artifacts tool is now also working on Windows. Check the following job steps:

  {
    // create some content
    "tool": "shell",
    "cmd": "mkdir abc && echo 'hello world' > abc/a.txt"
  }, {
    // upload the content to Kraken Storage
    "tool": "artifacts",
    "source": "abc"
  },
  ...
  {
    // download the uploaded content
    "tool": "artifacts",
    "action": "download",
    "source": "abc"
  }, {
    // check the content
    "tool": "shell",
    "script": """
       dir
       type abc\\a.txt
    """
  }

That's all. To get started, simply install or upgrade to the 1.5 version and experience all the benefits firsthand. If you have any questions or need assistance, submit an issue on GitHub or ask for help on Discord.

1.4 release brings several more features - check the release notes.

More details about environment variables can be found in the documentation, specifically in the Environment Variables section.

1.3 release brings several more features - check the release notes.

More details about the conditional steps are below.

The conditional steps rely on lazy step execution. Previously, all steps were sent to an agent, and the agent executed them all at once until either all steps were executed or one of the steps failed. Now, steps are sent from the server to an agent one by one. So, the server first evaluates the when condition and decides if the step should be sent for execution to an agent.

The example below shows various cases of when condition:

def stage(ctx):
  return {
    "parent": "root",
      "triggers": {
        "parent": True,
      },
      "parameters": [],
      "configs": [],
      "jobs": [{
        "name": "hello world",
        "steps": [{
          "tool": "shell",
          "cmd": "echo 'hello world'"
        }, {
          "when": "was_no_error",
          "tool": "shell",
          "cmd": "echo 'was_no_error'"
        }, {
          "when": "was_any_error",
          "tool": "shell",
          "cmd": "echo 'was_any_error'"
        }, {
          "when": "is_ci",
          "tool": "shell",
          "cmd": "echo 'is_ci'"
        }, {
          "when": "is_dev",
          "tool": "shell",
          "cmd": "echo 'is_dev'"
        }, {
          "tool": "shell",
          "cmd": "missing-command-xyz"
        }, {
          "when": "was_no_error",
          "tool": "shell",
          "cmd": "echo 'was_no_error 2'"
        }, {
          "when": "was_any_error",
          "tool": "shell",
          "cmd": "echo 'was_any_error 2'"
        }, {
          "when": "always",
          "tool": "shell",
          "cmd": "echo 'hello world always' && sleep 5"
        }, {
          "when": "job.steps[step.index - 1].result.duration > 3",
          "tool": "shell",
          "cmd": "echo 'hello world #{job.steps[step.index - 1].result}'"
        }],
        "environments": [{
          "system": "any",
          "agents_group": "all",
          "config": "default"
        }]
      }]
    }

This code can also be found at https://github.com/Kraken-CI/workflow-examples/tree/main/conditional-steps/one.py.

The screenshot below shows the status of these steps execution.

The usage of several built-in variables can be observed, such as always, is_ci, is_dev, was_no_error, and was_any_error. The last step presents a more complicated case. It utilizes job.steps and step.index to verify the outcome of the previous step, specifically checking if the duration of the previous step was greater than 3 seconds. If this condition is met, the current step will be executed.

The major changes are:

• added storing user data server-side
• data of internal objects and user data are now exposed in a schema code
• added support for several new operating systems

1.1 release brings several more features - check the release notes.

More details about the major features are below.

User Data​

This feature enables users to store data in the workflow schema code on the server-side. For instance, a user can prepare data in JSON format in one step of the schema code. Then, in the subsequent step, the user can upload the data from the prepared file to the Kraken Server. The data can be attached to a flow, branch, or project. If it is attached to a flow, its scope is restricted to that particular flow, and it can be used in the next stage of that flow. However, it will not be visible in a new flow. If the data is attached to a branch or project, it will be shared by subsequent flows.

Details and examples are described in the Data Tool section of the Workflow Schema documentation.

Data in Schema Code​

The data stored by the user, as well as the data of other internal objects such as jobs, runs, flows, branches, and projects, can be accessed in a stage schema code.

The data can be accessed via a context that is passed to the stage function and also through string interpolation of special variables.

All of this allows for significant dynamism in a stage schema code. For example, the list of jobs or step details may be based on the data stored by a user in previous stages.

Details are described in the Data in Schema chapter in the docs.

New OSes Supported​

The new supported OSes include Rocky Linux 9, AlmaLinux and Debian Bookworm. The full list is availabled in Docker Executor docs.

The major changes are:

• added global and per-branch logs retention policy
• added presenting logs of server-side activities related to a branch, a flow, a run and a job

1.0 release brings several more features - check the release notes.

More details about the major features are below.

Logs Retention Policy​

The logs retention policy generally indicates how long logs should be kept in the logs store ie. Clickhouse. It is possible to set these policies on two levels: 1) global and 2) branch level. The time is expressed in a number of months.

The picture below shows the General Settings tab where under the Global Log Retention Policy section, it is possible to set the number of months for how long all the logs (from Kraken Server and all Agents) will be kept in Clickhouse.

The following picture shows the Retention Policy tab on a branch management page. Here, a similar policy can be set for logs related to this branch, either for CI flows or for Dev flows.

Server Logs Presentation​

New logs panel has been introduced. It can show logs from both the server and agent sides. This panel can be found in several places that are indicated in the pictures below.

Here a branch management page is presented with the Logs tab where logs related to this branch are shown.

Here is the case for a flow.

Here is the case for a run.

And a job case.

The standing-out features connected with logs are:

• single-line and multi-line secrets masking
• coloring ANSI codes
• no issue with handling huge logs containing even more than 10.000 lines
• switching timestamps on/off
• auto scroll when at the bottom
• paging logs

And the new features introduced by this Kraken version:

• job logs are split by steps sections in UI
• steps status and metadata presented at the top of each step logs section
• download a step log

More details are below.

Single-line and Multi-line Secrets Masking​

It is possible to define secrets on a project. Then these secrets can be used in workflow schema directly in shell commands or as environment variables. Most often, they might be visible in logs. Thankfully, Kratos mask secrets - a secret in the log is replaced by six asterisks (* sign). Kraken can mask secrets that are single line, for example, a password, but also it can secret multi-line secrets, e.g. SSH private key that may span about 30 lines. In both cases, these secrets will be replaced with asterisks in the log.

The secrets are caught quite early and do not leave a machine that traces them to logs. The log database stores only masked passwords. Example of masking below:

A very long, multi-line secret has been replaced by 6 asterisk signs.

Coloring ANSI Codes​

Quite often in CI process, there are used tools that color their output. This is achieved by ANSI codes printed to a terminal. Kraken can find them and replace them with colors in Web UI.

An example of a basic color palette in the log:

Swithing Timestamps On/Off​

Kraken allows a user to switch on and off displaying timestamps. These are timestamps added during collecting logs and storing them in a database. They may not match with timestamps that are already present inside the log, so they can be switched off. When there are no timestamps in the original log, Kraken timestamps can be switched on and be pretty helpful.

Timestamps are switched off:

Timestamps are switched on:

Autoscroll​

When the scroll bar is at the bottom position then any new logs fetched automatically will trigger scrolling to the bottom so the new log content is visible.

Job Logs Split by Steps​

In Kraken Web UI, the job log is divided into steps sections. Each section has its title bar that, by clicking, allows a user to expand the section to see its details and logs. Each step log is separate, so it is easier to browse logs of steps than handling one big job log.

Steps Status and Metadata​

The step details bar presents information about status and duration but also step parameters. Below the bar, there are logs with control buttons for navigating logs.

Download Whole Step Log​

A log of each section can be downloaded separately.

Paging Logs​

A step log pane has control buttons allowing a user to navigate logs. A user may load the previous or next page or jump to the first or last page. A user may load the whole log, he/she may also scroll to the bottom of the log.

Kraken CI exposes a URL for a given branch that can be put to such a client. This URL is available on a branch management page, in the Status Badge & CCTray tab:

These feature landed in 0.1097 version and is documented in notifications docs docs.

Clickhouse has been upgraded from 20.11.4.13 to 22.10.2.11. The migration is seamless from the Kraken CI point of view. There is no need for explicit data migration.

MinIO has been upgraded from RELEASE.2020-12-18T03-27-42Z to RELEASE.2022-10-24T18-35-07Z. The major visible change is the names of environment variables that hold credentials. Previously they were MINIO_ACCESS_KEY and MINIO_SECRET_KEY, now they are MINIO_ROOT_USER and MINIO_ROOT_PASSWORD, respectively. Values of these variables may stay the same; they do not need to be changed.

These changes landed in 0.1091 version.

LDAP allows administrator to connect Kraken system to corporate LDAP or Active Directory. OIDC & OAuth beside bringing another source of users' identity it also provides Signe Sign-On scheme.

This features landed in 0.1047 version and is documented in identity providers docs.

Here is a screen that shows web UI for configuring Identity Providers.

Here is a screen that shows web UI for managing users.

To add such agents we will extend Docker Compose from QuickStart.

There is already defined one built-in agent. But first, several modifications are needed to solve IP addressing issues. The problem is that it may get a different IP address after a container restart than before. Kraken Server requires that the addresses do not change otherwise it is not possible to identify agents by the Kraken Server.

First, open your compose file (e.g. kraken-docker-compose-X.Y.yaml) and enable IP Address Management (IPAM) in the lab_net where all agents reside:

  ...
  lab_net:
    driver: bridge
    # add the following lines at the end of compose file
    ipam:
      config:
        - subnet: 172.20.0.0/16

Here is added a subnet with particular addresses class, 172.20.0.0/16. You can define any subnet definition you need.

And now add extra agents by copy-pasting agent service:

  agent1:
    restart: always
    image: us-docker.pkg.dev/kraken-261806/kk/kkagent:0.998
    environment:
      - KRAKEN_CLICKHOUSE_ADDR
      - KRAKEN_SERVER_ADDR
    networks:
      lab_net:
        ipv4_address: 172.20.0.11
    depends_on:
      - server
      - minio
      - clickhouse-proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Important things:

1. New agent service name should be different e.g. agent1, agent2, etc.
2. Remove - KRAKEN_AGENT_BUILTIN=1 environment variable that can be only in one agent.
3. Add explicit IP address in networks section: ipv4_address: 172.20.0.11. Every new agent needs to have a different IP address.

Before running such a modified compose file, shut down the currently running compose and delete lab_net. To delete this network, first, check if it exists:

$ docker network ls

It should have name like this <current-folder-name>_lab_net. Please delete it:

$ docker network rm <current-folder-name>_lab_net

Now you can start Docker Compose again. lab_net network will be recreated with proper new settings:

$ docker-compose --env-file kraken-X.Y.env -f kraken-docker-compose-X.Y.yaml up

Go to Discovered Agents page in Kraken Web UI and check if new agents are present there.

That's it. You have new agents running.

Custom tools allow for developing such tools by third parties, sharing them with others and installing and using them in Kraken Server.

Custom Tools Details​

Docs for custom tools first present the whole architecture of Kraken CI with Agents and tools. Then the interaction between Kraken Agent and a tool is shown with an example tool. The following sections show how to declare a custom tool and add it to Kraken Server so it can be used in workflows. It is also presented how to develop custom tools locally, under the desk.

Hello World Programs​

Every programmer knows what does Hello World program looks like.

Here is an example in C language:

#include<stdio.h>

int main(){
  printf("Hello World\n");
  return 0;
}

But 22? These 22 can be seen in Kraken's repo on GitHub: https://github.com/kraken-CI/hello-worlds

Still, there are more languages with Hello Worlds. The biggest collection can be found here: http://helloworldcollection.de/.

These are just examples. In our case, they can be compiled and run. After downloading the repo you may run ./run.sh script providing the language folder, e.g.:

$ ./run.sh C

This will display, compile and run a C example.

In Kraken's case, there is defined a workflow that does it for us for each language. But first, we need to have systems with a compiler or a runtime for these languages. Docker to the rescue.

Kraken Base Language Images​

For each language, there is prepared a Docker image. They are based on official images for these languages. For Kraken, they got some extras like Kraken Agent that is running using kraken user (not root).

The list of all lang images is available in the docs.

The Workflow​

The mentioned workflow can be found in kraken-workflow.py file.

In short, it looks as follows:

def stage(ctx):
    langs = [
        ('C', 'gcc:11.3'),
        ('C++', 'gcc:11.3'),
        ('C#', 'mono:6.12'),
        # and more...
    ]

    jobs = []

    for lang_name, lang_image in langs:
        jobs.append({
            "name": "hello world %s" % lang_name,
            "steps": [{
            "tool": "git",
                "checkout": "https://github.com/Kraken-CI/hello-worlds.git",
                "branch": "main"
            }, {
                "tool": "shell",
                "cmd": "./run.sh %s" % lang_name,
                "cwd": "hello-worlds"
            }],
            "environments": [{
                "executor": "docker",
                "system": "krakenci/%s" % lang_image,
                "agents_group": "all",
                "config": "default"
            }]
        })

    return {
        "parent": "root",
        "triggers": {
            "parent": True,
        },
        "parameters": [],
        "configs": [],
        "jobs": jobs
    }

As you may notice, the list of jobs is built dynamically. For each entry of the list of tuples (lang-name, docker-image), a job is defined with the same steps but various values for language and Docker image name. The lang name is passed to ./run.sh script in shell step. And Docker image name is used in the environments section, in the system field. Such a list of jobs is then returned in the whole workflow definition. Each job will be run in parallel in separate Docker containers.

The example of workflow execution is present in Kraken lab: https://lab.kraken.ci/runs/5009/jobs. Check the workflow definitions in branch management page.

This screenshot shows the execution of this workflow:

Summary​

This article shows how Kraken CI can build programs written in various languages. Kraken provides prepared Docker images for many languages. It is easy to define sophisticated workflows that dynamically define jobs that can be run in parallel. In the end, it is nice to observe the results of workflow execution in Kraken UI.

Besides that, it contains:

• support for multiline shell scripts and selecting shell executable (#183 and #184) charts
• new chart showing flows duration in a branch (#173)
• multiple new base docker images for various programming languages
• split results chart to pass ratio chart and passed/total chart (#187)

More information about these changes is below.

Job Designer​

Now it is possible to design job steps using a UI available on a branch management page.

The Prepare Step button shows the job designer dialog.

The dialog shows all available fields of a step with descriptions and input boxes pre-filled with default values. These fields are available for a tool selected in the dropdown at the top of the dialog. Below, there is the step generated in JSON format.

In the case of the picture above, there is a shell tool selected. There are multiple fields available. A few are used: cmd, cwd and shell_exec. The generated step in JSON contains these fields with user-provided values.

Now a user may copy the generated step and then paste it into the stage schema code.

Shell Tool Improvements​

The shell tool got two improvements. One allows for entering a multiline script instead of just one command. The other one allows for indicating an alternative shell executable - the default one is /bin/sh.

A multiline script can be provided in the following way:

{
   "steps": [{
       "tool": "shell",
       "script": """
           echo 'multiline script'
           ls -al
           echo 'completed'
       """
   }]
}

Changing default shell executable to e.g. zsh goes this way:

{
   "steps": [{
       "tool": "shell",
       "cmd": "echo $ZSH_VERSION",
       "shell_exec": "zsh"
   }]
}

More in the docs.

Flow Duration on Branch Chart​

On the branch management page, on Stats and Charts tab, there are basic stats about the branch, like the total number of flows or the average flow duration in the last week. Below these stats, a chart shows the duration of flows over time.

The whole stats and charts are presented in two instances: one for CI and the other one for DEV flows.

New Base Docker Images​

There are multiple new Docker images prepared to be used in Kraken CI. The full list is available in the docs page.

Split Results Chart​

Initially, the pass ratio was shown together with total and passed counts on the same chart. Now that is split into two charts: 1) pass ratio chart and 2) total and passed counts.

MLOps and MLflow​

MLOps is a set of practices that aims to build and maintain machine learning models in production reliably and efficiently. One of prominent tools in this area is MLflow.

MLflow is an open-source platform for managing the end-to-end machine learning lifecycle. It tackles four primary functions:

• Tracking experiments to record and compare parameters and results (MLflow Tracking).
• Packaging ML code in a reusable, reproducible form to share with other data scientists or transfer to production (MLflow Projects).
• Managing and deploying models from various ML libraries to a variety of model serving and inference platforms (MLflow Models).
• Providing a central model store to collaboratively manage the entire lifecycle of an MLflow Model, including model versioning, stage transitions, and annotations (MLflow Model Registry).

MLflow in Kraken CI​

In the following sections, I will describe how to prepare a workflow in Kraken CI to train an ML model. This is an LSTM model that will predict stock prices based on historical data.

The workflow will be:

1. pulling live stock data and preparing it for training (source 1, source 2)

2. performing the training (source 3)

3. storing model metrics in Kraken CI for charting

The MLflow project is described in MLproject.

Workflow Definition​

The whole Kraken CI workload is defined here.

There are 3 steps:

   "steps": [{
       "tool": "git",
       "checkout": "https://github.com/Kraken-CI/mlflow-example.git"
   }, {
       "tool": "shell",
       "cmd": "/opt/conda/bin/mlflow run .",
       "cwd": "mlflow-example",
       "timeout": 1200
   }, {
       "tool": "values_collect",
       "files": [{
           "name": "metrics.json",
           "namespace": "metrics"
       }, {
           "name": "params.json",
           "namespace": "params"
       }],
       "cwd": "mlflow-example"
   }],

1. Checkout mflow example project sources

2. Run the mlflow project ie. download data, prepare it, run a training and at the end store metrics about the trained model to metrics.json

3. Upload collected metrics together with hyperparameters from params.json to Kraken server

The last step allows for charting accuracy and RMS of the model over builds.

There is one more element defined in the workflow: the definition of execution environment:

   "environments": [{
       "system": "krakenci/mlflow",
       "executor": "docker",
       "agents_group": "all",
       "config": "default"
   }]

Here we can notice the use of a pre-prepared image with mlflow. It is available in Docker hub: krakenci/mlflow.

The whole example of workflow is present in Kraken lab: https://lab.kraken.ci/branches/32/ci. Check the steps definitions in branch management page.

Execution and Monitoring​

Besides the workflow definition, Kraken UI also shows collected data and the charts drawn from this data: https://lab.kraken.ci/test_case_results/595950, the charts tab.

The right chart shows value of loss collected over time:

Summary​

This article shows how Kraken CI can be used to build an MLOps pipeline. The pipeline downloads raw data, prepares the data for training and then executes the training. The trained model metrics are collected and charted in Kraken UI at the end.

Besides that, it contains:

• added a new tool values_collect that allows collecting values and then doing charts in UI based on them
• added cleaning cache in MinIO
• updated Go base image
• added new base image for MLflow

More information about these changes is below.

Dark Mode​

Now it is possible to look at Kraken CI web UI at night using dark mode.

If the dark mode is switched on in the browser that it will be automatically activated in Kraken CI web UI. Otherwise, it is possible to manually switch the dark mode on or off using the profile drop-down button in the top left corner of the screen (see below).

New Tool values_collect​

When testing is being carried out by a Kraken agent, custom measured metrics can be returned besides test results. Now it is possible to just generate such measurements, store them in a JSON file and then load them to the Kraken server. Then they will be visible on a job results page and it is possible to display a chart based on these values.

The reported file should look like this (e.g. metrics.json):

{
    "metric-1": 432,
    "metric-2": 2.12,
}

The step that consumes these values should look like this:

{
    "tool": "values_collect",
    "files": [{
        "name": "metrics.json",
        "namespace": "metrics"
    }, {
        "name": "params.json",
        "namespace": "params"
    }],
    "cwd": "example"
}

The end results looks like that:

The whole example of workflow with collected results is present in Kraken lab: https://lab.kraken.ci/branches/32/ci. Check the steps definitions in branch management page and then check the charts on tests results page, in the charts tab.

Cleaning Cache in MinIO​

The files that are always pulled to a build environmnt can be cached between builds, so they don't have to be downloaded from the internet each time. More about caching here.

The Current release introduces a feature that automatically removes old cached files. Now, after 10 days, these files are deleted from MinIO.

Changes in Base Images​

The Go lang base image has been updated. The latest image contains Go in 1.18. The location of the image is: krakenci/golang:1.18.

There is also a new image with MLflow. It is available under: krakenci/mlflow.

The changes include:

• added a chart with results history on a flow page
• added a multi-dimentional table with pass ratio stats of a flow
• added masking secrets in logs
• many improvements to the job's log panel like coloring and timestamps on/off
• added navigation to prev and next flow in breadcrumb on the flow page

More information about these changes is below.

Chart with Pass Ratio History​

The flow page has now two new tabs. One of them is a Results Charts tab. On this tab, a results history chart can be found. It presents a pass ratio line in % units and two more lines: passed and total that shows the number of passed tests and the total number of tests over time.

Pass Ratio Stats of Flow​

On the other tab, Results Stats, a table shows tests passing stats. The dimensions of the table are dynamic. They can be selected from dropdowns available over the table. There can be chosen: Stage, Agents Group and System.

Secrets Masking​

Another feature is related to security. Quite often, a password or some access key needs to be passed during build or test execution. Sometimes this secret is revealed in logs. Now Kraken knows secrets defined in a project and it masks them in the logs. The secrets are caught quite early and they do not leave a machine that traced them to logs. The log database stores only masked passwords.

There is one interesting property of this masking that cannot be found in other CI systems: it can mask multiline secrets like SSH private key.

Log Panel Improvements​

There were added several improvements to the log panel:

• coloring ANSI color codes in HTML log box
• turning timestamps on/off
• showing missing timestamps
• log loading indicator

The screens below show colored logs and timestamps that are either switched off or switched on.

Flow Navigation Improved​

The last feature presented here is flows navigation. In the breadcrumb bar there are shown two arrows that allow for navigating to the previous flow and to the next flow.