Ketexon's dialogue system

• KDiag
  • TOC
  • Why?
  • Building
  • State
  • Database
  • Expression Nodes
  • JSON Parser
  • Custom JSON Parser
  • Default functions

The main purpose of this project is to make a dialogue system that is fast, small, embeddable, and easy to customize. This comes after frustration with Yarnspinner and Ink.

The philosophy is this: you want a dialogue system. You try this engine out. You might want some very special behavior with the way dialogue runs in your game, or the way dialogue progresses. If you don't like one of our systems, they are so simple that you write your own, but use our defaults for everything else.

I believe that there are too many features in Yarnspinner and Ink. The basic dialogue system for this project took about 1 day to implement. For a lot of projects, that is all you'd need.

• C++ for embeddability and portability. Eventual C API planned.
• The model of the data is a directed graph. Easy to store, easy to understand.
• Decoupling of dialogue control flow (nodes) from content (lines)
  • Easy to export and translate
  • Easy to implement custom line databases (eg. for voice lines)
• Decoupling of file format from internal state
  • No de-facto file format. You can populate the state from existing file formats, like Yarn or Ink, or preprocess these formats into the preexisting .json format.
  • State is very easy to process. One function call advanced dialog.
  • No de-facto scripting language. As long as the language can be represented through an AST, it will work.

You can build using CMake.

cmake -B build && cmake --build build

The project is organized into modules that you can turn on and off using CMake flags (using -D<FLAG_NAME>=ON/OFF). The current modules are:

option(KDIAG_ENABLE_EXPRESSION_NODES "Include expression nodes" ON)
option(KDIAG_ENABLE_JSON_PARSER "Include JSON parser" ON)
option(KDIAG_ENABLE_CUSTOM_JSON_PARSER "Include custom JSON parser" ON)
option(KDIAG_ENABLE_DEFAULT_FUNCTIONS "Include default functions" ON)

The State represents the state of the loaded dialogue. This does not contain state about any running dialogues, which are each represented in the Dialogue class.

The important components of State are:

1. The Database, which stores nodes, lines, and "user functions". Anything loaded from a file should be in the Database.
2. functions, which are functions written in code for Dialogue to call. These can either be used within dialogue (eg. to_uppercase) or used to interact between the two (eg. emit_signal). This is intended to be the main way to add features and interact with the rest of the program.

Databases are a collection of nodes, lines, and functions. Databases are intended to be loaded from a file using a DatabaseLoader, such as JsonDatabaseLoader. Each line and node is represented with an ID (NodeID and LineID), which you can use to access a node/line using try_get_line and try_get_node

Expression nodes are an implementation of an AST used for scripting. An expression node is just a LazyValue, which is an object with a get(State&) function that returns a ValueType. Unless you are implementing a parser, you should only interact with the LazyValue interface.

All expression nodes must output a ValueType, which right now does not support polymorphism. Thus, you are stuck to these types:

using NullType = std::monostate;
using IntegerType = std::int64_t;
using FloatType = double;
using StringType = std::string;
using BoolType = bool;

There are two current expression nodes: ExpressionNodeLiteral, which holds a literal value, and ExpressionNodeFunctionCall, which holds a function name and expression node arguments.

ExpressionNodeFunctionCall uses the function obtained via State::try_get_function to get a function, and then calls it. See functions for more information.

The JSON parser parses any JSON file acceptable via nlohmann::json::parse following the schema below and turns it into a database. You are allowed to have trailing commas.

The schema is as follows, in TypeScript:

type NodeID = string
type LineID = string

type NodeEdge = {
  "node": NodeID,
  "line": LineID | undefined,
  "disabled": bool | undefined,
}

type Node = {
  "id": NodeID,
  "line": LineID,
  "metadata": { [k: string]: string } | undefined,
  "edges": Edge[],
}

// string, int, float, and boolean are converted to literals
// Expression[] is converted to concatenate(...)
type ShorthandExpression = string | int | float | boolean | Expression[]

type LiteralExpression = {
  "type": "literal",
  "value": string | int | float | boolean,
}

type FunctionCallExpression = {
  "type": "function_call",
  "name": string,
  "arguments": Expression[],
}

type LonghandExpression = LiteralExpression | FunctionCallExpression

type Expression = ShorthandExpression | LonghandExpression

type Line = {
  "id": LineID,
  "expression": Expression,
  "meta": string | undefined,
}

type Database = {
  "nodes": Node[] | undefined,
  "lines": Line[] | undefined,
}

JsonDatabaseParser is a thin wrapper over CustomJsonDatabaseParser, which uses nlohmann/json to parse.

The pipeline is:

• CustomJsonDatabaseParser() calls register_expression_node_parser to register function_call and literal expression node parsers
• parse_from_stream calls parse_json to convert std::istream& to nlohmann::json
• parse_from_stream calls parse_from_json to convert nlohmann::json to std::unique_ptr<Database>
  • parse_from_json parses the schema, using try_parse_expression_node to parse Line Expressions.
    • try_parse_expression_node first calls try_parse_shorthand_expression_node to parse a shorthand expression (non-object type)
    • If that fails, it uses try_parse_expression_node to parse a long form expression using the type field to index expression_node_parser_functions

class CustomJsonDatabaseParser : public DatabaseParser {
public:
    using ExpressionNodeParserFunction = Result<std::unique_ptr<ExpressionNode>, Error> (*)(CustomJsonDatabaseParser&, const nlohmann::json&);

    CustomJsonDatabaseParser();

    virtual Result<std::unique_ptr<Database>, Error> parse_from_stream(std::istream& stream) override;
    virtual Result<nlohmann::json, Error> parse_json(std::istream& stream);
    virtual Result<std::unique_ptr<Database>, Error> parse_from_json(const nlohmann::json& j);

    void register_expression_node_parser(const std::string& type, ExpressionNodeParserFunction parser_func);

    Result<ValueType, Error> try_parse_value_type(const nlohmann::json& j);

    virtual Result<std::optional<std::unique_ptr<ExpressionNode>>, Error>
    try_parse_shorthand_expression_node(const nlohmann::json& j);

    Result<std::unique_ptr<ExpressionNode>, Error>
    try_parse_expression_node(const nlohmann::json& j);
private:
    std::unordered_map<std::string, ExpressionNodeParserFunction> expression_node_parser_functions;
};

There are 4 ways to customize this parser:

1. Use register_expression_node_parser to add a new expression node parser, which will extend the schema above to add a new LonghandExpression for a certain type. You do not need to subclass CustomJsonDatabaseParser here, you can just register_expression_node_parser after construction.
2. Override try_parse_shorthand_expression_node to customize how ShorthandExpressions are parsed. For example, you can have string expressions not be converted into literals, but parsed again into ExpressionNodes.
3. Override parse_from_json to change the general schema of the JSON file (eg. changing how Nodes themselves are parsed).
4. Override parse_json to add a custom filetype (eg. BSON). nlohmann/json supports other binary formats like BSON if you are interested.