My experience with Rust has mainly been through minor projects, which have mostly been web projects for university courses and unlaunched side projects. I have also used the language heavily for different coding challenges as a way to get more familiar. I think I am a good representation of an intermediate Rust programmer, which is the book’s target audience, since I have run into all the beginner traps. I completed a Master’s in Information Technology relatively recently (4 years ago), so most of the computer science subjects are still relatively fresh as of this writing. With this, my overall aim for the book is to learn something new about Rust and use it to develop more advanced Rust projects in the future.

Review

The book covers more advanced subjects that are really useful to know before diving into them. The initial chapters mostly sum up subjects an intermediate Rust developer should be somewhat familiar with, where the later chapters does a good deep dive in to more advanced details. Especially the chapters about async/await and unsafe had a lot of interesting details that I did not know. It is well written and worth a read.

The book is very well written, but that is also a minor downside. It describes everything well, but primarily using words. Sometimes code examples are missing, and other times there are too many code examples. For the harder-to-understand sections, some illustrations would have been useful — for example, in understanding concurrency or unsafe.

Learnings

These learnings are mostly notes about the book, giving an overview of interesting subjects. They mostly serve to help me remember the book, but can also be useful to check if the concepts are new to you too.

Memory

A wide pointer (or fat pointer) is a pointer where the size is encoded as part of the pointer instead of being known at compile time.

The second chapter has an interesting point about alignment and the layout of the Rust types in memory. In Rust, the layout of the memory is not guaranteed like in C. The Rust compiler can move fields around such that they are stored more efficiently. You control this by adding a repr to the type. Additionally, it describes how alignment can cause types to take up additional space for padding such that the fields are byte-aligned and easier for the CPU to access.

This reminded me of an old example showing that C actually contains the possibility of polymorphism. You just have to make the structs have the same fields at the start, and those will have the same memory layout and can be easily reused. This does not work in Rust, since without explicitly choosing the layout, the compiler will optimize it, meaning two structs with the same fields might look different in their memory layout.

Dynamic dispatching and trait objects

Dynamic dispatching happens when you use the dyn keyword instead of impl. I had an intuitive understanding of impl as it works similarly to template in C++, but I have never quite understood dyn. This part explains how the dynamic dispatching happens: instead of creating a method, it passes a trait object consisting of a reference and a vtable such that the correct method can be chosen dynamically at runtime. This is somewhat similar to how most other languages do it, so it demystified the dyn keyword for me.

Generic Traits

Generic traits work mostly as I would expect. The interesting points are the ways you can “share” the details. For example, you can make a blanket implementation that works for all the generics with impl<T> MyTrait for T where T:, which is kind of the same as having a shared base object, but without the inheritance.

Interfaces

Chapter 3, Designing Interfaces, is interesting, but mostly something I already understood well. It argues that a good interface should follow four principles:

• Unsurprising: Names should be consistent with the ecosystem. Commonly used traits should be implemented: Debug, PartialEq, PartialOrd…
• Flexible: Only take ownership of the type when it is required. Use the least constrained type. For example, instead of String use &str or even better impl AsRef<str>.
• Obvious: Document every public interface. Use the type system to guide in only providing valid input and calling correct methods.
• Constrained: Constrain what is available to the user, as any change to an exposed type could be a breaking change. Re-export types in the base to allow for moving them internally.

Errors

Errors should implement std::error::Error and Display, which provide extra helpful structures. I did not know this and could have used it a couple of times.

Opaque Errors can be implemented with Box<dyn Error + Send + Sync + 'static>. They can be used in Result when the Error is not something the user can handle. In reality they should probably panic, but returning this allows them to clean up before they panic. I do not like this approach. I previously ran into it and was really annoyed that I was not able to have better error handling, but this is apparently something that is discussed in the community as a good practice.

Testing

The Test Harness being a Rust binary just compiled to run the tests is an interesting detail. It can also be overridden with a custom one.

Doctests mean any examples included in Rust docs are tested by compiling them, ensuring they work. This is an interesting detail that ensures that users can copy them directly.

The Clippy linter is important. I was already using it.

cargo-fuzz and proptest can be used for fuzzing and property-based testing. These seem like interesting crates to try in a project.

Miri can be run next to tests to check for failures to uphold safety requirements.

Loom can be used to verify different scenarios in a thread context.

Performance testing should watch out for performance variance, compiler optimizations, and I/O overhead. These are quite common worries, but especially with Rust as it performs a lot of optimizations.

Macros

Macros are something I have seen, but never had a use case for. This is probably because I have not yet ventured into more advanced programs. They are useful for code minimization.

Declarative macros are advanced regexes that allow matching and injecting code. They have hygiene because their variable names do not conflict with the other variables.

Procedural macros are the more advanced macros, which take a token stream and return a token stream. Common use cases are function-like macros, attribute macros (like #[test]) and derive macros (like #[derive(Debug)]). With them, the syn library can be used for easier definitions. They do not have hygiene, which the implementer has to take care of.

For most day-to-day use cases, the declarative macros are more than enough.

Async await

A generator is used by the compiler to store the state of an async function. This is kind of the same idea as most other async-await implementations. The main difference is that Rust needs to ensure the memory safety of the related information. This is done with the Pin trait, which specifies that memory is locked to a certain place and will not move. This is primarily used for the async functions and generators themselves, as they cannot be moved, to ensure they are still safe in relation to the rest of their context. There are macros like Box::pin(x) that pin to the heap. Alternatively, a new pin!() macro can pin to either. Most structs do not need to be pinned. For these, the Unpin trait is implemented automatically. This specifies that the pinned data can be moved around. The compiler does this by looking at all fields and checking whether they are Unpin. If they are, the overall struct is too.

All async needs a runtime, and Rust does not provide one by default, although most applications use the tokio crate.

An async method is a Future, which has to be poll()ed in order to continue. At some point a result is expected to be returned. An executor is responsible for doing this continuous checkup. If a future owns another future, the top level will be called, and then it has to call the levels further down until it reaches the leaf futures. The leaf futures are the places where the actual waiting happens. To avoid continuously calling these futures — for example, a future waiting for network traffic might wait a while — a Waker is used to wake up the awaited future when the traffic arrives. This is implemented in the Future’s context such that it is possible to use custom ones.

When working with async, all awaits will themselves be part of the same generator. In order to separate the work, the spawn!() macro has to be used to specify that work can continue independently of the current task. This is, for example, useful in a classic HTTP server setup that continuously accepts new clients.

Unsafe

Unsafe allows extra behaviour such as pointer dereferencing, and is used as a marker to tell:

• An unsafe block {} is used to tell that the author of that code has determined the code to be safe.
• unsafe fn is used to tell that a caller of the method needs to ensure it is safe.
• unsafe trait is used to tell that an implementation of the trait needs to ensure safety, but it can still be safe to call it.

These methods are not inherently dangerous, but the developer has to ensure certain conditions when they are called. It is important not just to ensure this condition holds at the time of calling the method, but also to ensure that it remains accurate when the program changes.

Naming of unsafe methods should follow the standards. A common pattern is to use _unchecked when a safety check is omitted.

The Send and Sync traits are examples of unsafe traits. These will usually be applied automatically by the compiler. If you want to apply them manually, it is because the compiler has not, because there is something it cannot guarantee. The implementation is therefore unsafe, as the developer needs to argue as to why it is safe.

When writing unsafe code, it is important to assert that certain behaviours do not happen. In cases where the code has to be performant, this can be done only in tests with cfg!(test) or debug_assert!. This provides extra safety in the implementation.

Concurrency

The difference between async and concurrency is that async could in theory all run on a single thread, kind of like Node.js does it. The concurrency chapter talks about how to work with Rust in multiple threads. The book describes basic concurrency at a good level, with the correctness and performance talk at the start, then a section about concurrency models — Shared Memory, Worker Pools and Actors — and a section about lower-level concurrency with atomic types and the like. Additionally, it gives recommendations about starting simple and only optimizing when something is proven slow, using stress tests to trigger failures, and using the Loom tool to test concurrency.

The section in relation to concurrency that I found the most interesting is about the memory ordering of atomic types. Specifically, about how the different orderings have different guarantees about how executions happen in threads, which are not always linear. The different memory orderings are interesting to know about, but I think in most cases I would stick to Ordering::SeqCst, which provides the most guarantees, but is also the slowest.

Foreign Function Interfaces

Foreign Function Interfaces allow Rust to work with other languages. They can use different setups, but most commonly the C interface is used, as most other languages also work with it. This is done by setting the symbols, like in a C program, in a way that other languages can find and call them. This is needed because the Rust compiler does not guarantee symbol names and allows for multiple functions with the same name.

bindgen and Build Scripts exist that make it possible to generate and inject other code into the build. This could, for example, be used for assembly optimizations if needed.

When designing an FFI, it is a good idea to try to hide away the memory unsafety from the other caller. This can be done in different ways. If it is not possible to use Rust’s type system to encapsulate the interface, it should be made unsafe.

No std + ecosystem

Chapter 12 is dedicated to describing how to work with Rust without a standard library. Interestingly, the language is built up around this by having a layer of a core lib with pure Rust, alloc for anything requiring dynamic allocation, and then std for the remaining parts and a re-export of the previous. This keeps the implementation details hidden from most std users, but allows embedded programming to use a subset. The book describes patterns for how to do custom dynamic memory allocation, panic handling, program initialization, out-of-memory handling, writing safe Rust for hardware and cross-compilation. This is not something I personally expect to use, but it’s an interesting chapter.

Chapter 13 is dedicated to the Rust ecosystem and further learning. The main interesting points are the patterns: index pointer and drop guard. An index pointer is simply a structure that stores both a Vec and indexes or usize numbers pointing to where certain elements are stored. This helps prevent pointers or hard-to-get-right lifetimes. The drop guard is a pattern for ensuring something happens in case a panic happens. It is simply implementing Drop for a custom type that cleans up some other data.

Tips and tricks

These are the tips and tricks I noted throughout the book:

• Use repr(transparent) when creating a wrapper type without any additional fields such that it shares the same in-memory representation as the underlying struct. For example, struct A and struct NewA(A) would share the same representation and be easily transformed.
• Use repr(C) when interacting with C programs to ensure the memory layout does not have to be transformed in the call.
• Check that a type is normal (has all the default implemented traits) with a test that fails if they are not implemented:

  fn is_normal<T: Sized + Send + Sync + Unpin>() {}
  #[test]
  fn normal_types() {
    is_normal::<MyType>();
  }
  

• “Sealed” traits can be done with:

  pub trait CanUseCannotImplement: sealed::Sealed { }
  mod sealed {
    pub trait Sealed {}
    impl<T> Sealed for T where T: TraitBounds {}
  }
  impl<T> CanUseCannotImplement for T where T: TraitBounds {}
  

• The Semver Trick: When a new release happens, an older release can be published that uses the types from the new release such that it interplays with an updated version of the dependency. This allows dependencies requiring newer or older versions to be used with the other version.
• Hide deprecated public items that should not be used with #[doc(hidden)]. Beware that they should still be documented; they are just not exposed in the docs, but still available in the source.
• When compile time grows too big, subcrates can be used to split the compilation into multiple crates, which can really speed up development.
• Features can be used to conditionally include dependencies. Some features are set by default, so you would have to override them if not wanted.
• A patch can be used in the crate to temporarily specify a different version, for example a git branch or folder, to use instead of the dependency. This allows for checking that a fix to a dependency actually fixes the problem, or allows continued development while the branch is being released in the dependency.
• opt-level can be used to optimize a program. 0 for “not at all” and 3 for “as much as you can”. Also “s” to optimize for binary size.
  • These can also be overridden for specific dependencies with [profile.dev.package.<package>].
• The default panic is an unwind that unwinds the stack, although this can be changed to an abort, which is for example useful in embedded programming.
• #[cfg(condition)] is used for conditional compilation.
  • cfg(windows) or cfg(unix) for the operating system.
  • Packages can also be enabled conditionally with [target.'cfg(windows)'.dependencies].
• The Minimum Supported Rust Version can be found by reverting versions until the library no longer compiles. This should be done for all libraries intended for public use, as it allows for the widest possible use.
• Enable the Clippy lints: missing_docs and missing_debug_implementations.
• Use #[no_mangle] to make the symbol global, so it can be used in FFIs

Looking it up did bring up a few articles of how to do it, but I wanted to understand it before doing so. I put it on hold then and picked it back up recently to dive more into the subject. This post describes the overall idea behind Coordinate Compression, how to do it, and then what I think was missing from other places; why it works.

Solving a big maze

The problem: Find the shortest path through the maze:

This can be easily solved by a Breadth-first search. In a 40×40 space it will not have any problems. However, BFS has to visit and remember every cell it reaches, so both its running time and its memory grow with the number of cells. Scale this same grid up 100 times or more and the search quickly becomes too slow and too memory-hungry to finish. This is hard to show in an image, which is why the illustrations will be of smaller graphs.

This maze in reality only consists of a few large zones separated by full-length walls, and each zone connects to its neighbour through a single narrow gap. This is very obvious to the human eye. Even if you scaled the whole maze up so each zone was ten times larger, there would still be only one gap between neighbouring zones (now just wider, with more possible ways through). The interior of each zone is what you can think of as empty space.

How to use Coordinate Compression

The idea is to use this empty space to form a new compressed graph, where the empty space is compressed into a smaller set of nodes. The compression is done by scanning the rows and then the columns and keeping a boundary wherever a row (or column) differs from the one before it. A run of identical rows collapses into a single compressed row, and the same is done for columns. (The “Why it works” section explains exactly which rows count as different, and why.) This can be seen in the following gif:

The gif shows the two graphs side by side. You can see that it produces a somewhat similar graph, but with a different ratio. This can also be illustrated as mapping the compressed coordinate boxes onto the original graph as seen in this image:

The highlighted areas are to show how regions of 3×3 original cells are grouped into a single compressed cell in the compressed graph. You might wonder why we split the zones so many times, and that will be discussed further in the “Why it works”-section.

In code this can be stored as two 1D arrays, one per axis, where each index is the compressed coordinate and the value is the corresponding original coordinate. The x and y axes are compressed independently, so a compressed position (cx, cy) maps back to the original position (xs[cx], ys[cy]). The distance between two neighbouring compressed cells is just the difference between their stored original coordinates, which is what gives each edge its weight later.

After this compression, Dijkstra’s algorithm can be run on the compressed graph to find the shortest path. The steps should still be counted using the lengths in the real graph, but the compressed graph can be used for finding the neighbours. The following gif helps to understand how the search through the compressed graph correlates to the original graph:

Why it works

Coordinate compression works because every position inside a compressed cell has the same neighbourhood structure. This is where the splitting described earlier becomes important. By introducing splits at every change in the maze’s layout (and around those changes), every compressed cell is guaranteed to contain only positions that behave identically. No matter where you are inside a compressed cell, you can only leave it through the same neighbouring compressed cells.

So why split around every change, and not just at the change itself? Consider a single column where a wall ends and an opening begins. The cells in that column are not all the same: the cell right next to the opening can step sideways through the gap, while the cells further along the wall cannot. If you grouped that whole column into one compressed cell, the group would contain positions with different escape routes, and the “you can only leave through the same neighbours” guarantee would break. To avoid this, you take every coordinate where something changes and also keep the coordinate on either side of it as its own row or column. In practice this means collecting the significant coordinates and, for each one c, splitting at c-1, c, and c+1 (where splits from neighbouring coordinates overlap, they simply collapse together).

This is also why you cannot simply make each zone its own box. A turn in the shortest path can only happen at a boundary, so the boundaries must survive compression as distinct cells. If the splits are too coarse, the compressed graph loses the ability to turn at the right place, and Dijkstra can no longer reconstruct the true shortest path.

The compression does not remove any possible routes through the maze. Instead, it groups large regions of equivalent cells into a single node while recording the real distances between neighbouring regions. A move in the compressed graph may therefore represent many steps in the original maze.

This is why Dijkstra’s algorithm is used instead of Breadth-first search. In the original maze every edge has a cost of 1, making BFS sufficient. After compression, edges can have different costs corresponding to the actual distance travelled in the original maze. The compressed graph is therefore a weighted graph, which is exactly the type of problem Dijkstra’s algorithm solves.

Coordinate compression removes coordinates, not geometry. Large empty regions are replaced by weighted rectangles that preserve both the available routes and the distance of those routes. Dijkstra does not care whether an edge represents one step or one thousand steps; it only requires that the edge weight reflects the true travel cost. Because the compressed graph preserves those costs, the shortest path found in the compressed graph is identical to the shortest path in the original maze.

Conclusion

In Quest 15 the compression was actually a lot simpler, since each move represented a coordinate break of c-1, c and c+1 around the the x and y of each wall end.

Writing up this article helped me better understand why this solves the problem. I think that by having an understanding of why it works, it becomes easier to apply to problems. Specifically, this only fits a graph with a huge empty space and would not make much sense in a denser maze. Additionally, you need more splits than I would initially have imagined. This does not work if you try to make each box its own, as then Dijkstra could not prove it was the shortest. I hope this helped you understand it too.

The example code is also available on Github.

The problem

An example is the registration of a User that contains fields for name, dateOfBirth, and email. This can be represented in a class like the following:

public class User {
    public String name;
    public LocalDate dateOfBirth;
    public String email;

    public User(String name, LocalDate dateOfBirth, String email) {
        this.name = validatedName(name);
        this.dateOfBirth = validatedDateOfBirth(dateOfBirth);
        this.email = validatedEmail(email);
    }

    private static String validatedName(String name) {
        if (name == null) {
            throw new IllegalArgumentException("Name cannot be null");
        }

        final var trimmedName = name.trim();
        if (trimmedName.isBlank()) {
            throw new IllegalArgumentException("Name cannot be blank");
        }

        return trimmedName;
    }

    private static LocalDate validatedDateOfBirth(LocalDate dateOfBirth) {
        if (dateOfBirth == null) {
            throw new IllegalArgumentException("Date of birth cannot be null");
        }

        if (dateOfBirth.isBefore(LocalDate.of(1900, 1, 1))) {
            throw new IllegalArgumentException("Date of birth cannot be before 1900-01-01");
        }

        if (dateOfBirth.isAfter(LocalDate.now().minusYears(18))) {
            throw new IllegalArgumentException("User must be at least 18 years old");
        }

        return dateOfBirth;
    }

    private static String validatedEmail(String email) {
        if (email == null) {
            throw new IllegalArgumentException("Email cannot be null");
        }

        final var trimmedEmail = email.trim();
        if (trimmedEmail.isBlank()) {
            throw new IllegalArgumentException("Email cannot be blank");
        }

        if (!trimmedEmail.contains("@")) {
            throw new IllegalArgumentException("Email must contain an '@' character");
        }

        return trimmedEmail;
    }
}

Each of the three fields is validated, and an exception will be thrown if the input is invalid. However, only a single exception can be reached at a time. This means the user will only ever see one error at a time. First they will get an exception for the name, which they have to fix before the date of birth validation runs, and only after that is valid will the email validation run.

This is terrible for UX, since the user will think they are close to completing the form but will continuously encounter new errors. A better approach is to ensure the user can see all errors discoverable from the current state. For an empty form, the system would find an error for each field and display them all at once. The next two sections will show how to achieve this with Exceptions and Result<T, E>s.

Throwing multiple exceptions

An aggregator for exceptions can be implemented quite easily in a non-type-safe way using a Varargs function:

public interface VarFunction<T> {
    T apply(Object... args);
}

public static class AggregatedException extends Exception {
    public AggregatedException(List<Exception> exceptions) {
        exceptions.forEach(this::addSuppressed);
    }
}

@SafeVarargs
public static <T> T exceptionAggregator(
        VarFunction<T> function,
        Supplier<Object>... argSuppliers
) throws AggregatedException {
    final var args = new Object[argSuppliers.length];
    final var exceptions = new ArrayList<Exception>();
    for (int i = 0; i < argSuppliers.length; i++) {
        final var argSupplier = argSuppliers[i];
        try {
            args[i] = argSupplier.get();
        } catch (final Exception e) {
            exceptions.add(e);
        }
    }
    if (!exceptions.isEmpty()) {
        throw new AggregatedException(exceptions);
    }
    return function.apply(args);
}

This uses a Varargs object as the arguments, which means it is not type-safe. To call it you will have to manually cast the objects to their concrete types. An overload can be placed on top of this to allow for type safety. This is done by implementing a Function interface that takes generic arguments and returns a specific type, and then a new function that combines those generic arguments.

Here are the code examples for 2 and 3 arguments (the repository contains examples for up to 8 arguments):

@FunctionalInterface
public interface Function2<T1, T2, T> {
    T apply(T1 t1, T2 t2);
}

@SuppressWarnings("unchecked")
public static <T1, T2, T> T exceptionAggregator(
        Function2<T1, T2, T> function,
        Supplier<T1> supplier1,
        Supplier<T2> supplier2
) throws AggregatedException {
    final VarFunction<T> varFunction = (args) -> function.apply((T1)args[0], (T2)args[1]);
    return exceptionAggregator(
        varFunction,
        (Supplier<Object>) supplier1,
        (Supplier<Object>) supplier2
    );
}

@FunctionalInterface
public interface Function3<T1, T2, T3, R> {
    R apply(T1 t1, T2 t2, T3 t3);
}

@SuppressWarnings("unchecked")
public static <T1, T2, T3, T> T exceptionAggregator(
        Function3<T1, T2, T3, T> function,
        Supplier<T1> supplier1,
        Supplier<T2> supplier2,
        Supplier<T3> supplier3
) throws AggregatedException {
    final VarFunction<T> varFunction = (args) -> function.apply((T1)args[0], (T2)args[1], (T3)args[2]);
    return exceptionAggregator(
        varFunction,
        (Supplier<Object>) supplier1,
        (Supplier<Object>) supplier2,
        (Supplier<Object>) supplier3
    );
}

This function can be used by a static factory method in the earlier example to aggregate the exceptions:

public static User of(String name, LocalDate dateOfBirth, String email) throws CombineThrow.AggregatedException {
    return CombineThrow.exceptionAggregator(
            User::new,
            () -> validatedName(name),
            () -> validatedDateOfBirth(dateOfBirth),
            () -> validatedEmail(email)
    );
}

Now the exception will contain all 3 suppressed exceptions, and they can be mapped and shown to the user as 3 errors at the same time. However, I think this approach can be further improved by using Result<T, E>.

Multiple errors with Result<T, E>

My library for Result<T, E> allows for expressing errors in the return type instead of throwing Exceptions. To use this, the exceptions have to be replaced with results. For each of the 3 validation methods, this is done by simply replacing the return value and throw with Result.ok and Result.err:

private static Result<String, String> validatedName(String name) {
    if (name == null) {
        return Result.err("Name cannot be null");
    }

    final var trimmedName = name.trim();
    if (trimmedName.isBlank()) {
        return Result.err("Name cannot be blank");
    }

    return Result.ok(trimmedName);
}

private static Result<LocalDate, String> validatedDateOfBirth(LocalDate dateOfBirth) {
    if (dateOfBirth == null) {
        return Result.err("Date of birth cannot be null");
    }

    if (dateOfBirth.isBefore(LocalDate.of(1900, 1, 1))) {
        return Result.err("Date of birth cannot be before 1900-01-01");
    }

    if (dateOfBirth.isAfter(LocalDate.now().minusYears(18))) {
        return Result.err("User must be at least 18 years old");
    }

    return Result.ok(dateOfBirth);
}

private static Result<String, String> validatedEmail(String email) {
    if (email == null) {
        return Result.err("Email cannot be null");
    }

    final var trimmedEmail = email.trim();
    if (trimmedEmail.isBlank()) {
        return Result.err("Email cannot be blank");
    }

    if (!trimmedEmail.contains("@")) {
        return Result.err("Email must contain an '@' character");
    }

    return Result.ok(trimmedEmail);
}

As with the exception example, a static factory method can be used to combine the Results:

public static Result<User, List<String>> of(String name, LocalDate dateOfBirth, String email) {
    return CombineResult.combine(
            User::new,
            validatedName(name),
            validatedDateOfBirth(dateOfBirth),
            validatedEmail(email)
    );
}

This is implemented in much the same way as the exception aggregator:

public interface VarFunction<T> {
    T apply(Object... args);
}

@SafeVarargs
public static <T, E> Result<T, List<E>> combine(
        VarFunction<T> function,
        Result<Object, E>... resultArgs
) {
    final var args = new Object[resultArgs.length];
    final var errors = new ArrayList<E>();
    for (int i = 0; i < resultArgs.length; i++) {
        final var arg = resultArgs[i];
        int finalI = i;
        arg.consume(object -> args[finalI] = object)
                .consumeError(errors::add);
    }
    if (!errors.isEmpty()) {
        return Result.err(errors);
    }
    return Result.ok(function.apply(args));
}

@SuppressWarnings("unchecked")
public static <T1, T2, T3, T, E> Result<T, List<E>> combine(
        Function3<T1, T2, T3, T> function,
        Result<T1, E> result1,
        Result<T2, E> result2,
        Result<T3, E> result3
) {
    final VarFunction<T> varFunction = (args) -> function.apply((T1)args[0], (T2)args[1], (T3)args[2]);
    return combine(varFunction, (Result<Object, E>) result1, (Result<Object, E>) result2, (Result<Object, E>) result3);
}

Now type safety is provided to the caller of the combine function, and it easily allows multiple errors to be returned.

This little example and experiment is something I have used for a while in real projects, though never with an explicit error type. It is something I expect to add to the Result<T, E> library shortly.

Conclusion

These examples show how to return multiple errors in Java. This can be used to display all relevant errors to a user at once, which helps improve the user experience of an application.

The example code is also available on Github.

Specifically, this post looks at switch-expressions that evaluate to a single value. For example, the following switch:

final var display = switch(enumValue) {
    case TYPE_1 -> "Type 1";
    case TYPE_2 -> "Type 2";
};

In this case, the code will only compile if the enumValue enum type only contains TYPE_1 and TYPE_2. If you later add a TYPE_3, it will give a compilation error. This is the result that we want, since any change will force the developer to update this too.

It is important to notice that this breaks the Open-closed principle, as a change to an enum owned by one module will force changes to another. This might still be worth it to ensure full coverage of all the options, but use it with consideration.

Object cast

If you have a base interface that other classes inherit from:

interface Base {}
record Derived1(String value) implements Base {}
record Derived2(int value) implements Base {}

You can use a switch-expression to extract the value:

final var display = switch (base) {
case Derived1(String value) -> value;
case Derived2(int value) -> "" + value;
};

However, this will fail because it is not exhaustive. Instead, you can make the interface sealed, and it will work:

sealed interface Base permits Derived1, Derived2 {}
record Derived1(String value) implements Base {}
record Derived2(int value) implements Base {}

The code will now compile.

The sealed interface can only permit classes that it can directly access. This is also a way of enforcing the Open-closed principle, as it will not allow the switch without complete ownership of the class.

Conclusion

Making the switch-expression exhaustive by using the type system will push errors further towards the development phase, which will help ensure fewer issues arise in any environment. If exhaustiveness is not possible, then a switch is probably not the correct pattern to use. In that case, a strategy pattern can be used instead.

In the JDK implementation, Optional is implemented as a final class that contains a single value field. For Optional.empty, this value is null. Additionally, a single empty instance is reused across all empty calls.

Instead, I implemented it as a sealed interface with two records for Present and Empty. I then reimplemented the API from Optional, which is quite straightforward as the methods are usually quite self-explanatory. Additionally, I added test cases based on the interface in order to test all possible code paths.

The implementation is available on: https://github.com/HHMagnus/Java-Serializable-Optional

Because of this implementation, you can call all the standard Optional methods, but it also adds new possibilities.

A switch can be used to extract the value:

final var message = switch (optional) {
    case Empty -> "No message";
    case Present(var message) -> message;
};

Additionally, for a normal Optional you would have to do something like this to guard against null:

if (optional.isEmpty()) {
    return;
}
final var message = optional.get();

System.out.println(message);

But with this implementation you could deconstruct the value with:

if (!(optional instanceof Present(var message))) {
    return;
}
System.out.println(message);

This is all in addition to it being Serializable, and I also encourage you to use it as a field or argument.

I performed a benchmark to test performance, and it does not seem like there is a noticeable difference between this Optional<T> implementation and the JDK’s.

An example of a Result class was introduced in a previous post. It is an Either pattern from functional programming for describing an operation that has either failed or succeeded.

Whether you actually want to roll back the transaction depends entirely on context. If you have a domain class that returns a Result, you might not want to roll back anything, as it is the consumer’s job to handle the error. However, if your application service returns a Result, you would want it to roll back the transaction, as that is part of its normal contract.

Transactional Rollback

One option is to roll back the transaction manually and programmatically:

TransactionAspectSupport.currentTransactionStatus().setRollbackOnly();

Beware of this approach, as it does not work well if any code afterwards calls a transactional method, because it will throw a UnexpectedRollbackException.

The approach that I have found to work is the most annoying one: throwing an exception. Instead of returning a Result at the application layer, it defaults to throwing an Exception. This is fairly straightforward to implement with the Message Bus pattern, as all application-level calls go through a single service.

Conclusion

It seems the only way to make the @Transactional annotation work with Result without triggering UnexpectedRollbackException is to throw an exception. If anyone finds a different solution, please reach out.

This post contains a simplified version of the full library available on GitHub: https://github.com/HHMagnus/Java-Result

The simplest form of the implementation is:

public sealed interface Result<T, E> {
    public record Ok<T, E>(T value) implements Result<T, E> {}
    public record Err<T, E>(E error) implements Result<T, E> {}
}

Using a sealed interface, it is effectively a discriminated union with Ok and Err being the only two options. Anyone using the Result can switch pattern match it. The following example handles the errors with an exception:

switch (result) {
    case Ok(var value) -> value;
    case Err(var error) -> throw new IllegalArgumentException(error);
}

This is quite a useful pattern because it helps prevent throwing errors too early. Consider, for example, that you have a domain method that does some string-handling logic to determine if a string is valid input:

enum Error {
    IS_BLANK,
    IS_PROFANE
}

Result<String, Error> handleInput(String input) {
    final var trimmed = input.trim();
    if (trimmed.isBlank()) {
        return Result.Err(Error.IS_BLANK);
    }

    if (isProfane(trimmed)) {
        return Result.Err(Error.IS_PROFANE);
    }

    return Result.Ok(trimmed);
}

Without the Result, this would have required two throws. Because it is instead described as a Result, it can very easily be handled differently:

void ignoreCase(String input) {
    final var result = handleInput(input);

    final var value = switch (result) {
        case Ok(var val) -> val;
        case Err(var val) -> "Default value";
    };

    // Proceed with other tasks
}

void rejectionCase(String input) {
    final var result = handleInput(input);

    if (!(result instanceof Result.Ok(val))) {
        System.out.println("Did nothing because of an error: " + result);
        return;
    }

    // Proceed with other tasks
}

Fluent like Optional

Result can be seen as an extension of Optional where the empty state is instead an error. Optional provides fluent functions like map and filter that allow for handling the case where a value is present. This can be done similarly with Result:

public sealed interface Result<T, E> {
    <N> Result<N, E> map(Function<T, N> mapper);

    public record Ok<T, E>(T value) implements Result<T, E> {
        @Override
        <N> Result<N, E> map(Function<T, N> mapper) {
            final var newValue = mapper.apply(value);
            return Result.Ok(newValue);
        }
    }
    public record Err<T, E>(E error) implements Result<T, E> {
        @Override
        <N> Result<N, E> map(Function<T, N> mapper) {
            // No mapping happens as it is an error
            return Result.Err(error);
        }
    }
}

By having polymorphism support whether it is failed or ok, the implementation is quite simple:

• In the ok case, it applies the mapper and returns the value wrapped in an Ok.
• In the error case, it does nothing and returns the same error wrapped in a new Err to support the new type.

Beyond map, a few additional fluent methods round out the API:

• flatMap(Function<T, Result<N, E>> mapper) — like map, but for when the mapping function itself returns a Result. This avoids ending up with a nested Result<Result<N, E>, E>. In the Ok case it applies the mapper and returns its result directly; in the Err case it passes the error through unchanged.
• consume(Consumer<T> consumer) — runs a side-effecting action (such as logging or saving) on the value if it is Ok, and does nothing if it is Err. Returns this so chaining can continue.
• consumeError(Consumer<E> consumer) — the mirror of consume: runs a side-effecting action on the error if it is Err, and does nothing if it is Ok. Also returns this.

Together, these allow a Result to be handled in a single fluent chain:

new Result.Ok(input)
    .map(String::trim)                  // transform the value if Ok
    .flatMap(this::handleInput)         // validate, returning Ok or Err
    .consume(System.out::println)       // print the value if still Ok
    .consumeError(System.err::println); // print the error if Err

This uses the previous handleInput function to either produce a cleaned string or an error, then routes the outcome to the appropriate output stream without a single if or try/catch.

OptionalResult and VoidResult

Two additional implementations are:

• OptionalResult — same as Result, but with a third state for being Empty
• VoidResult — same as Result, but always empty/void

By implementing these inside the same library as Result, one can transition between them very easily:

new Result.Ok(value)
    .toOptionalResult() // wraps in Optional
    .toVoidResult()     // removes entirely

OptionalResult provides a way to fluently transform a value if it is present without failing when something is empty. Take the following code:

new OptionalResult.Present(value)
    .mapValue(String::trim)  // removes whitespace
    .filter(String::isBlank) // defaults to Empty if blank

If the value is empty, it will default to an empty OptionalResult.

VoidResult is a way to describe an action that returns nothing but can fail:

VoidResult handleInput(String input) {
    final var trimmed = input.trim();
    if (trimmed.isBlank()) {
        return VoidResult.Err(Error.IS_BLANK);
    }

    if (isProfane(trimmed)) {
        return VoidResult.Err(Error.IS_PROFANE);
    }

    return VoidResult.Ok();
}

The library contains more examples of this.

Conclusion

The Result pattern is something I use heavily in my projects, and this new implementation is very simple while allowing complex flows to be described clearly. It is very well-suited for describing domain logic without using exceptions, allowing the consumer to specify what should happen when something goes wrong.

An implementation with many helper functions can be found on my GitHub: https://github.com/HHMagnus/Java-Result

Migrating data between systems is usually done directly at the data layer. This is the most convenient and performant approach, but it makes testing much harder. SQL databases tend to enforce only the most basic data requirements, and several common patterns fall through the cracks:

• Enum values are usually stored as numbers or strings, making it easy to insert a value the application has no corresponding model for.
• Dependent fields are hard to enforce. For example, if a boolean flag is set, another field may always need to be populated — but the database will only see a nullable column, not the business rule behind it.
• Calculated values stored for performance reasons may be impossible to validate or recompute in pure SQL.

These are just a few examples. In practice, any business rule that lives in application code rather than the schema is invisible to a migration working at the data layer.

Rich Domain Hydration

Rich domain models wrap primitives in typed value objects that enforce business rules at construction time. They typically support at least two ways to be constructed: a normal factory method (of) that validates external state, and a rehydration method (rehydrate) that reconstructs an object from persisted data. The key difference is intent — of enforces business rules, while rehydrate only checks that the data fits the expected format.

Here is an example:

public class Order {
    public final OrderNumber number;
    public final RecipientId recipient;
    public PaymentId payment;

    private Order(OrderNumber number, RecipientId recipient, PaymentId payment) {
        this.number = number;
        this.recipient = recipient;
        this.payment = payment;
    }

    public static Order of(
        RecipientId recipient,
        OrderNumberReservationSystem orderNumberReservationSystem
    ) {
        final var orderNumber = orderNumberReservationSystem.reserve();
        return new Order(orderNumber, recipient, null);
    }

    public void registerPayment(
        Payment payment
    ) {
        // verify throws if it does not match
        payment.verifyOrderNumber(number);
        this.payment = payment.getId();
    }

    public static Order rehydrate(
        final String orderNumber,
        final Long recipientId,
        final Long paymentId
    ) {
        // Any of the `of` methods could throw if they fail their checks
        final var orderNumberDomain = OrderNumber.of(orderNumber);
        final var recipientIdDomain = RecipientId.of(recipientId);
        final var paymentIdDomain = PaymentId.of(paymentId);
        return new Order(orderNumberDomain, recipientIdDomain, paymentIdDomain);
    }
}

An Order is initially created without a payment. The payment is attached later once received. The normal construction path does not support rehydration, so rehydrate allows the database values to be passed in directly to reconstruct the object. Importantly, rehydrate still delegates to the value object constructors (OrderNumber.of, RecipientId.of, etc.), so if a stored value is malformed or out of range, it will throw — surfacing the data problem rather than silently loading bad state.

Data Integrity Through Rehydration

Because rehydration still validates that data fits the expected format, it gives you a simple and powerful way to verify your database after a migration. Here is an example test using a JPA entity OrderJpa that maps directly to the orders table:

EntityManager em;

@Test
public void verifyData() {
    final var orders = em.createQuery("select o from OrderJpa o", OrderJpa.class)
        .getResultList();

    var failed = false;
    
    for (final var orderJpa : orders) {
        try {
            final var order = Order.rehydrate(
                orderJpa.getOrderNumber(),
                orderJpa.getRecipientId(),
                orderJpa.getPaymentId()
            );
        } catch (DomainValidationException ex) {
            ex.printStackTrace();
            failed = true;
        }
    }

    assertFalse(failed);
}

This iterates through all orders and attempts to rehydrate each one, logging any failures to the console. Rather than stopping on the first error, it collects all failures so you get a complete picture of what went wrong in one run.

In practice, this test works well as an integration test run against a staging environment after the migration completes, but before switching traffic over to the new system. For larger datasets, it can also be structured as a lightweight script run directly against production in a read-only transaction. Either way, the feedback loop is fast: run the test, see which records failed and why, fix the migration, and run again.

Conclusion

Data migrations are inherently risky, and issues often only surface after the migration is complete. Rich domain models give you a testing interface that is not usually present in a migration setup, letting you verify data integrity at the application layer rather than relying solely on database constraints.

This pattern scales well as the domain grows. As new value objects and rules are added to the application, the rehydration tests automatically cover them — there is no extra test maintenance burden. Combined with other migration strategies like incremental rollouts or dual-write periods, using rich domains for post-migration verification gives you much stronger confidence that the data your application depends on is in the shape it expects.

The source code is available on https://github.com/HHMagnus/Rich-Domain-Generic-Datamodel-Java.

Rich Domain

The example of the domain is BusinessReceipt and PersonalReceipt with the following (simplified) code:

public record Address(String addressLine) { }
public record Period(LocalDate startDate, LocalDate endDate) { }
public record SubscriptionMonth(int value) { }

public record BusinessReceipt(
    Address invoiceAddress,
    Period subscriptionPeriod,
    boolean isRenewal,
    SubscriptionMonth months
) { }

public record PersonalReceipt (
        Address deliveryAddress,
        Address invoiceAddress,
        LocalDate date
) { }

This is a very simple structure where any domain logic is omitted for brevity.

Contract

In order to interface with a generic model, a few contracts are needed. The model needs to know the enum type of both the Receipt and the specific field:

public enum ReceiptType {
    BUSINESS,
    PERSONAL
}

public enum Field {
    INVOICE_ADDRESS_LINE,
    DELIVERY_ADDRESS_LINE,
    DATE,
    SUBSCRIPTION_PERIOD_START_DATE,
    SUBSCRIPTION_PERIOD_END_DATE,
    IS_RENEWAL,
    SUBSCRIPTION_MONTHS;
}

Field stores all possible fields. This is the structure that can be expanded in the future when new fields are added or removed.

In order to save the domain, two additional interfaces are used to specify how to store it:

public interface Receipt {
    ReceiptType save(ReceiptStorage storage);
}
public interface ReceiptStorage {
    void text(Field field, String value);
    void number(Field field, int value);
    void bool(Field field, boolean value);
    void date(Field field, LocalDate date);
}

These allow the domain to specify how to save the data it contains. They are then implemented by the domain classes:

// BusinessReceipt
public ReceiptType save(final ReceiptStorage storage) {
    storage.text(Field.INVOICE_ADDRESS_LINE, invoiceAddress().addressLine());
    storage.date(Field.SUBSCRIPTION_PERIOD_START_DATE, subscriptionPeriod().startDate());
    storage.date(Field.SUBSCRIPTION_PERIOD_END_DATE, subscriptionPeriod().endDate());
    storage.bool(Field.IS_RENEWAL, isRenewal());
    storage.number(Field.SUBSCRIPTION_MONTHS, months.value());

    return ReceiptType.BUSINESS;
}

// PersonalReceipt
public ReceiptType save(final ReceiptStorage storage) {
    storage.text(Field.DELIVERY_ADDRESS_LINE, deliveryAddress().addressLine());
    storage.text(Field.INVOICE_ADDRESS_LINE, invoiceAddress().addressLine());
    storage.date(Field.DATE, date());

    return ReceiptType.PERSONAL;
}

The interface needs to be expanded for any possible type.

Generic Datamodel

The generic datamodel contains two entities: Instance and InstanceField. The Instance represents a Receipt and the InstanceField is a list of all the fields. The Instance class simply contains the ReceiptType and the field list:

@Entity
@Table(name = "instance")
public final class Instance {
    @Id
    @GeneratedValue
    private Long id;

    @Column
    @Enumerated(EnumType.STRING)
    public ReceiptType type;

    @OneToMany(mappedBy = "instance", cascade = CascadeType.ALL, orphanRemoval = true)
    public List<InstanceField> fields = new ArrayList<>();
}

InstanceField is an abstract class that uses inheritance to structure the different value types using BooleanField, DateField, NumberField, and TextField:

@Entity
@Table(name = "instance_field")
@DiscriminatorColumn(name = "field_type")
@Inheritance(strategy = InheritanceType.SINGLE_TABLE)
@ConcreteProxy
public abstract sealed class InstanceField
        permits BooleanField, DateField, NumberField, TextField {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    @ManyToOne(fetch = FetchType.LAZY)
    @JoinColumn(name = "instance_id")
    private Instance instance;

    @Column
    @Enumerated(EnumType.STRING)
    private Field field;

    protected InstanceField(
            Field field,
            Instance instance
    ) {
        this.field = field;
        this.instance = instance;
    }

    protected InstanceField() { }

    public Field field() {
        return field;
    }
}

@Entity
@DiscriminatorValue("boolean")
public final class BooleanField extends InstanceField {
    @Column(name = "boolean_value")
    public Boolean value;

    public BooleanField(
            Field field,
            Instance instance,
            Boolean value
    ) {
        super(field, instance);
        this.value = value;
    }

    // No args constructor required by Hibernate
    protected BooleanField() {
        super();
    }

    public Boolean value() {
        return value;
    }
}

@Entity
@DiscriminatorValue("date")
public final class DateField extends InstanceField {
    @Column(name = "date_value")
    public LocalDate value;

    public DateField(
            Field field,
            Instance instance,
            LocalDate value
    ) {
        super(field, instance);
        this.value = value;
    }

    // No args constructor required by Hibernate
    protected DateField() {
        super();
    }

    public LocalDate value() {
        return value;
    }
}

@Entity
@DiscriminatorValue("number")
public final class NumberField extends InstanceField {
    @Column(name = "number_value")
    public Integer value;

    public NumberField(
            Field field,
            Instance instance,
            Integer value
    ) {
        super(field, instance);
        this.value = value;
    }

    // No args constructor required by Hibernate
    protected NumberField() {
        super();
    }

    public Integer value() {
        return value;
    }
}

@Entity
@DiscriminatorValue("text")
public final class TextField extends InstanceField {
    @Column(name = "text_value")
    public String value;

    public TextField(
            Field field,
            Instance instance,
            String value
    ) {
        super(field, instance);
        this.value = value;
    }

    // No args constructor required by Hibernate
    protected TextField() {
        super();
    }

    public String value() {
        return value;
    }
}

This data structure uses a SINGLE_TABLE inheritance strategy, which means the SQL schema only contains two tables:

create table instance (
	id bigint not null primary key,
	type varchar(255) check ((type in ('BUSINESS','PERSONAL')))
);

create table instance_field (
	id bigint generated by default as identity primary key,
	instance_id bigint references instance,
	
	field_type varchar(31) not null check ((field_type in ('boolean','number','date','text'))),
	field varchar(255) check ((field in ('INVOICE_ADDRESS_LINE','DELIVERY_ADDRESS_LINE','DATE','SUBSCRIPTION_PERIOD_START_DATE','SUBSCRIPTION_PERIOD_END_DATE','IS_RENEWAL','SUBSCRIPTION_MONTHS'))),

	boolean_value boolean,
	date_value date,
	number_value integer,
	text_value varchar(255)
);

In order to construct the Instance, it implements the ReceiptStorage interface and has the following static factory method Instance.of(Receipt receipt):

// Instance class
public static Instance of(final Receipt receipt) {
    final var instance = new Instance();
    instance.type = receipt.save(instance);
    return instance;
}

@Override
public void text(final Field field, final String value) {
    final var textField = new TextField(field, this, value);
    fields.add(textField);
}

@Override
public void number(final Field field, final int value) {
    final var numberField = new NumberField(field, this, value);
    fields.add(numberField);
}

@Override
public void bool(final Field field, final boolean value) {
    final var boolField = new BooleanField(field, this, value);
    fields.add(boolField);
}

@Override
public void date(final Field field, final LocalDate date) {
    final var dateField = new DateField(field, this, date);
    fields.add(dateField);
}

This allows for quickly saving the domain model and can easily be expanded.

Reading

A read model is created to expose later:

public record ReceiptFieldModel(Field field, String displayValue) { }

public record ReceiptModel(
        ReceiptType type,
        List<ReceiptFieldModel> fields
) {
    public static ReceiptModel of(final Instance instance) {
        final var fields = instance.fields()
                .stream()
                .map(field -> switch (field) {
                    case BooleanField booleanField -> new ReceiptFieldModel(booleanField.field(), booleanField.value.toString());
                    case DateField dateField -> new ReceiptFieldModel(dateField.field(), dateField.value.toString());
                    case NumberField numberField -> new ReceiptFieldModel(numberField.field(), numberField.value.toString());
                    case TextField textField -> new ReceiptFieldModel(textField.field(), textField.value);
                }).toList();

        return new ReceiptModel(instance.type(), fields);
    }
}

The sealed part of InstanceField now comes in handy, as it allows for switching over the possible values of the field without a default case. This will give a compile error if it breaks later.

Combining It

In a real application, some business logic will create the Receipt, which can then be stored and even later read. This process is most visible in the ReceiptTest from the code:

@Test
void canSaveAndLaterReadPersonalReceipt() {
    // Arrange
    final var invoiceAddress = "Invoice address";
    final var startDate = LocalDate.of(2026, 1, 1);
    final var endDate = LocalDate.of(2026, 1, 31);
    final var isRenewal = true;
    final var months = 15;

    final var businessReceipt = new BusinessReceipt(
            new Address(invoiceAddress),
            new Period(startDate, endDate),
            isRenewal,
            new SubscriptionMonth(months)
    );

    // Act (persist)
    final var instance = Instance.of(businessReceipt);

    entityManager.persist(instance);

    sync();

    // Assert (read to model)
    final var retrieved = entityManager.find(Instance.class, instance.id());
    final var recipientModel = ReceiptModel.of(retrieved);

    assertEquals(ReceiptType.BUSINESS, recipientModel.type());
    assertEquals(List.of(
            new ReceiptFieldModel(Field.INVOICE_ADDRESS_LINE, invoiceAddress),
            new ReceiptFieldModel(Field.SUBSCRIPTION_PERIOD_START_DATE, startDate.toString()),
            new ReceiptFieldModel(Field.SUBSCRIPTION_PERIOD_END_DATE, endDate.toString()),
            new ReceiptFieldModel(Field.IS_RENEWAL, "" + isRenewal),
            new ReceiptFieldModel(Field.SUBSCRIPTION_MONTHS, "" + months)
    ), recipientModel.fields());
}

From this it is also visible that if the domain ever changes, the code here can be updated, but reading the old domain will still be possible as no structure is enforced when changing.

Conclusion

This example successfully shows that a rich domain model can be persisted and later read by a generic datamodel. This allows a domain to change over time while still allowing previous states to be read in a consistent manner. The code can move forward while still supporting a historical view of the data. This is most useful when storing data that is append-only and never modified. This example can be the basis of other business-complex problem solutions, and the Receipt example is simply one of many possible domains.

The source code is available on https://github.com/HHMagnus/Rich-Domain-Generic-Datamodel-Java.

A database migration is essentially just a SQL file that is executed before an application starts. The idea is that the migration files live as part of the source code, allowing you to migrate databases just before or at the same time as you upgrade the application version. Having the migrations be part of the source also makes it easier to go back to an earlier version of the application and spin up a database as it was when that version existed.

Flyway migrations

Flyway migrations usually exist in the resource folder of the database module of the application. They are usually executed in one of two ways:

• As part of application startup, where the Flyway process is embedded as part of the application.
• As a separate task that must be run before starting the application.

The first is usually preferred, but the second is sometimes required to ensure proper setup. For example, if you base your database on a previous dump of data, you might never want to start up the wrong version.

Each Flyway file follows the format <Prefix><Version>__<Name>.sql:

• <Prefix> is either V for a one-time migration or R for repeatable migrations.
  • It is possible to configure a different prefix, but it is rarely worth the trouble.
• <Version> is free-form text used only to determine the ordering of migrations.
• <Name> is free-form text that can be anything; it is useful for describing what the migration is about.

Note the 2 underscores between <Version> and <Name>. This can easily cost a couple of hours of debugging the first time you forget one.

Flyway files are always executed as a transaction, so there is no need to manually handle transactions (for example BEGIN and COMMIT) inside them — despite what many LLMs tend to recommend.

The main Flyway command is called migrate, and it is what you will usually run to apply all migrations to the database.

Ordering of migrations

The <Version> part of the filename specifies the ordering of migrations. Each version string must be unique across all migration files, so it is important to minimise overlap. Here are a few useful patterns:

• Incrementing integer: 1, 2, 3…
  • V1__initial.sql, V2__updated_column.sql, V3__added_table.sql…
  • This is the simpler structure, but it is hard to prevent overlaps in projects with multiple contributors. Recommended when fewer than five people are actively contributing.
• Date and incrementing integer: 2026_01_01_1, 2026_01_01_2, 2026_01_01_3…
  • V2026_01_01_1__initial.sql, V2026_01_01_2__updated_column.sql, V2026_01_01_3__added_table.sql…
  • Easier to keep unique when fewer migrations happen each day. Good for slow-moving projects.
• Date-time format: 2026_01_01_08_00, 2026_01_01_08_01, 2026_01_01_08_02…
  • V2026_01_01_08_00__initial.sql, V2026_01_01_08_01__updated_column.sql, V2026_01_01_08_02__added_table.sql…
  • The most granular format; very good for large projects as collisions are rare. The downside is that you may encounter more out-of-order problems.

Two special versions exist: beforeMigrate.sql and afterMigrate.sql, which do what their name suggests. They are run before and after a migrate command.

Ordering matters because the SQL files do not necessarily produce the same result depending on when they are run. For example, consider these two migration files:

• V1__add_3_rows_to_table.sql — adds 3 new rows to the table
• V2__change_all_rows_of_table.sql — updates a column value on all rows

When V1 runs first, the 3 new rows will also be updated by V2. However, if V2 ran first, the 3 rows added later would not have the updated value.

Failing migration files (repair)

A Flyway file will fail if the SQL is not executable. In those cases, the database must either be updated manually or the migration file must be fixed. During development you can simply change the content and try again. However, if the migration has already been successfully executed in an environment, you should not change it. If you are forced to change it, see the repair section below.

When you are forced to change an already-applied file, other environments will encounter a checksum mismatch because the file content no longer matches what was applied. The repair command can be used in this case. Note that repair is a separate command from migrate and does not run any migrations — it only updates the stored checksum to match the new file. This means changes will not be applied to that environment. Therefore, changes to existing files should not introduce anything new; they should only fix whatever is needed for the file to be executable.

repair can also be used if you remove a previously added migration — it will remove the corresponding entry from the history table. Importantly, it will not undo the changes made by the removed migration. Migrations should never be removed (see the repeatable migrations section for edge cases).

Out of order

With some versioning strategies, an error can occur when you add a migration with a version number earlier than a migration that has already been applied. For example, suppose you added:

V2026_01_01_10_00__my_change.sql

but your colleague merged first with:

V2026_01_01_10_45__colleague_change.sql

The database may already contain your colleague’s migration, so yours would come earlier in the ordering.

Flyway has a flag for this situation: out-of-order. Enabling it allows migrations to run even if their version is earlier than the latest applied version. You should manually verify that running them out of order does not cause problems.

Note that out-of-order is a flag (used with the migrate command), whereas repair is a standalone command.

On large projects with many daily migrations, you may want to enable out-of-order by default in development environments. However, always remember to disable it in production, as applying migrations out of order can cause the issues illustrated in the example above.

Repeatable migrations

Repeatable migrations use the prefix R, for example R__my_test_users.sql. They can be used to insert test data or verify data integrity, and they always run after all standard versioned migrations.

Repeatable migrations are not run every time. Once executed, they will only run again if their content changes.

Repeatable migrations are not easily deletable. Because their execution is recorded in the history table, a repair is needed if you remove one. If you want to deactivate a repeatable migration, one option is to empty the file or add a comment explaining why it is no longer needed.

If you use a repeatable migration to check data integrity, you can add ${flyway:timestamp} at the top of the file. This inserts the current execution time, which forces Flyway to always re-run the migration since the content appears to change each time.

How does Flyway do it?

Flyway creates a table called flyway_schema_history in your database. This is a normal table that can be queried by any SQL user. It stores the prefix, version, name, and checksum of every migration that has been applied. When Flyway runs, it compares the migration files in your codebase against this history and executes any migrations that are not yet present.

Out-of-order errors occur when a row in that table contains a later version than a migration that has not yet been run.

Purging migrations

In long-running projects, the list of migrations can become very long. Running all of them in order to recreate a database from scratch can take a significant amount of time, which can be a problem in local development. A good recommendation is to create a dump of a fully migrated database that you can use locally to save time.

If you want to remove old migrations entirely, the process is as follows:

• Use a database tool to generate a full SQL schema recreation of the current database.
• Remove all existing Flyway migrations and add a single new migration containing the generated schema.
• Manually clear the flyway_schema_history table and insert a fake entry for the new migration, since the schema is already in place.

This process is risky because it requires a manual step. Additionally, if you have a partially migrated database, you may end up with a schema that differs from the new baseline. This approach is useful before a system goes to production but should never be done afterwards.

Conclusion

The aim of this post is to explain the core concepts behind Flyway. Reach out if you think anything important is missing.