Katla

Katla is a data flow generator.

It is designed to generate data over time, transform it, and output it to different destinations (files, Kafka, Elasticsearch, MQ, etc.) according to a schedule.

It can be configured via creating rust structs or config files in JSON or RON format.

Getting started

Cargo.toml

[dependencies]
katla = "0.1.0"

Using RON config

src/main.rs

fn main() {
    use katla::config::*;
    use std::io::prelude::*;
    use std::fs::File;

    let ron_cfg = r#"#![enable(implicit_some)]
(
	name: "flow-basic",
	description: "Flow -- Basic",
	appenders: [
		(
			type: "file",
			name: "flow-basic--appender",
			fileName: "flow-basic.txt",
		),
	],
	entityProviders: [
		(
			type: "sequence",
			name: "flow-basic--entity-name",
		),
	],
	entries: [
		(
			name: "flow-basic--entry",
			entity: "flow-basic--entity-name",
		),
	],
	feeds: [
		(
			type: "file",
			name: "flow-basic--file-feed",
			content: (
				type: "katla",
				uri: "data:application/x.katla-rs.katla,{{ flow-basic--entry }}\n",
			),
			pagination: (
				type: "inFile",
				count: 1000,
			),
		),
	],
	flows: [
		(
			name: "flow-basic--flow",
			timeInterval: "R20000",
			entries: [
				"flow-basic--entry",
			],
			outputs: [
				(
					duration: "PT0S",
					feed: "flow-basic--file-feed",
					appender: "flow-basic--appender",
				),
			],
		),
	],
)"#;
    let include_paths = &".:inventory".split(":").collect();
    let mut katla = Katla::new("name".to_owned(), "description".to_owned());
    katla.load_config(ron_cfg.to_string(), include_paths).unwrap();
    katla.generate();
}

Main Building Blocks

Entity Providers

They are the lowest-level data producers.

Entries

An entry is a record, made up of one or more entities. Entries are used by the feed to produce output from templates and entries

Feeds

A feed produces output to be sinked by an appender according to a flows schedule.

Flows

A flow orchestrates feeds over time using a schedule.

Appenders

Appenders consume feed output and route it somewhere on

• Files (different formats possible)
• Kafka
• ElasticSearch
• ...

Appenders are decoupled from feeds in order to be able to output the same data to different outputs. data to be outputted

Crate Features

• ibm-mq--mqi: Enables IBM-MQ appender using native c library

Legal

Copyright

Copyright (C) 2019 - 2026
  - Twimpiex <twimpiex@gnx.be>

License

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License version 3 as
published by the Free Software Foundation.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.

Please contact the copyright holder(s) if you wish to use this program
under a different license.

Disclaimer

Be aware that at any time this program may (unintentionally)
introduce bugs, backward incompatibility, undesired behavior,
backdoors, ...

Trust your own testing and quality assurance, do not rely solely on
semantic versioning.

Contributing

Any contribution intentionally submitted for inclusion in this
program shall have it's copyright (automatically) transferred to the
current copyright holder(s) of this program, without any additional
terms or conditions.