mio

A cross-platform C++17 header-only library for memory mapped file IO.

This is a modernized fork of mandreyel/mio, updated to require C++17 and take advantage of modern C++ features. The original library was created by mandreyel and is licensed under the MIT license.

Features

Header-only with no dependencies
Cross-platform: Windows, Linux, macOS
Supports both read-only and read-write mappings
Move-only semantics with zero-cost abstraction over system APIs
Optional shared ownership via shared_mmap
Works with file paths or existing file handles/descriptors
Automatic page boundary alignment
Support for std::filesystem::path
Support for std::byte via type aliases
Optional std::span conversion (C++20)
Single-header version available

Requirements

C++17 compiler (GCC 8+, Clang 7+, MSVC 2017 15.7+, Apple Clang 11+)
CMake 3.14+ (for building tests)

Why mio?

Memory mapping is a powerful technique that can significantly improve IO performance by allowing direct access to file contents through memory pointers, avoiding explicit read/write system calls.

The primary motivations for mio over alternatives like Boost.Iostreams:

File handle support: mio can establish a memory mapping with an already open file handle/descriptor, not just file paths.
Automatic offset alignment: Boost.Iostreams requires users to pick offsets exactly at page boundaries, which is error-prone. mio manages this internally, accepting any offset and finding the nearest page boundary automatically.
Flexible ownership semantics: Boost.Iostreams uses std::shared_ptr for all mappings, incurring heap allocation overhead even when not needed. mio provides two classes:
- mio::mmap_source / mio::mmap_sink: Move-only, zero-cost abstraction
- mio::shared_mmap_source / mio::shared_mmap_sink: Shared ownership when needed
No dependencies: mio is a standalone header-only library with no external dependencies.

Quick Start

#include <mio/mmap.hpp>
#include <system_error>
#include <cstdio>

int main()
{
    std::error_code ec;

    // Memory map a file for reading
    mio::mmap_source source;
    source.map("data.bin", ec);
    if (ec) {
        std::printf("Error: %s\n", ec.message().c_str());
        return 1;
    }

    // Access data directly
    for (const char& byte : source) {
        // process byte...
    }

    return 0;
}

Usage Examples

Creating a Read-Only Mapping

There are three ways to create a mapping:

Using the constructor (throws on failure):

#include <mio/mmap.hpp>

// Map entire file
mio::mmap_source mmap("path/to/file");

// Map with offset and length
mio::mmap_source mmap("path/to/file", offset, length);

Using the factory function:

std::error_code ec;
auto mmap = mio::make_mmap_source("path/to/file", ec);
if (ec) { /* handle error */ }

// With offset and length
auto mmap = mio::make_mmap_source("path/to/file", offset, length, ec);

Using the map() member function:

mio::mmap_source mmap;
std::error_code ec;
mmap.map("path/to/file", ec);
if (ec) { /* handle error */ }

Creating a Read-Write Mapping

Use mmap_sink instead of mmap_source:

#include <mio/mmap.hpp>
#include <algorithm>

int main()
{
    std::error_code ec;

    // Map file for writing
    mio::mmap_sink mmap = mio::make_mmap_sink("output.bin", ec);
    if (ec) { return 1; }

    // Modify the mapped memory
    std::fill(mmap.begin(), mmap.end(), 0);
    mmap[0] = 'H';
    mmap[1] = 'i';

    // Sync changes to disk (also done automatically on destruction)
    mmap.sync(ec);

    return 0;
}

Using File Handles/Descriptors

// POSIX
#include <fcntl.h>
#include <mio/mmap.hpp>

int fd = open("file.txt", O_RDONLY);
mio::mmap_source mmap(fd, 0, mio::map_entire_file);

// Windows
#include <windows.h>
#include <mio/mmap.hpp>

HANDLE handle = CreateFileA("file.txt", GENERIC_READ, ...);
mio::mmap_source mmap(handle, 0, mio::map_entire_file);

Using std::filesystem::path

#include <mio/mmap.hpp>
#include <filesystem>

std::filesystem::path path = "/data/file.bin";
std::error_code ec;

mio::mmap_source mmap;
mmap.map(path, ec);

Shared Ownership

When multiple owners need access to the same mapping:

#include <mio/shared_mmap.hpp>

std::error_code ec;

// Create a shared mapping using factory function
auto shared1 = mio::make_shared_mmap_source("path/to/file", ec);
if (ec) { /* handle error */ }

// With offset and length
auto shared2 = mio::make_shared_mmap_source("path/to/file", offset, length, ec);

// Copy (both share the same underlying mapping)
mio::shared_mmap_source shared3 = shared1;

// Move from a regular mmap
mio::mmap_source mmap("path/to/file");
mio::shared_mmap_source shared4(std::move(mmap));

Using std::byte (C++17)

#include <mio/mmap.hpp>

std::error_code ec;
mio::bmmap_source byte_map;  // basic_mmap_source<std::byte>
byte_map.map("binary.dat", ec);

for (std::byte b : byte_map) {
    // process bytes...
}

Using std::span (C++20)

#include <mio/mmap.hpp>
#include <span>

mio::mmap_source mmap("data.bin");
std::span<const char> span = mmap.as_span();

Complete Example

#include <mio/mmap.hpp>
#include <system_error>
#include <fstream>
#include <cstdio>
#include <cassert>

int main()
{
    const char* path = "example.txt";

    // Create a file to map (mio requires the file to exist)
    {
        std::ofstream file(path);
        file << std::string(1000, 'x');
    }

    std::error_code ec;

    // Create a read-write mapping
    mio::mmap_sink rw_mmap = mio::make_mmap_sink(path, 0, mio::map_entire_file, ec);
    if (ec) {
        std::printf("Error: %s\n", ec.message().c_str());
        return 1;
    }

    // Use iterator-based algorithms
    std::fill(rw_mmap.begin(), rw_mmap.end(), 'a');

    // Use range-based for loop
    for (auto& byte : rw_mmap) {
        byte += 1;  // 'a' -> 'b'
    }

    // Use subscript operator
    rw_mmap[0] = 'Z';

    // Sync and unmap
    rw_mmap.sync(ec);
    rw_mmap.unmap();

    // Verify with read-only mapping
    mio::mmap_source ro_mmap;
    ro_mmap.map(path, ec);
    if (ec) { return 1; }

    assert(ro_mmap[0] == 'Z');
    assert(ro_mmap[1] == 'b');

    std::printf("Success!\n");
    return 0;
}

CMake Integration

Using FetchContent (Recommended)

The simplest way to integrate mio into your CMake project:

cmake_minimum_required(VERSION 3.14)
project(MyProject)

include(FetchContent)

FetchContent_Declare(
    mio
    GIT_REPOSITORY https://github.com/your-username/mio.git
    GIT_TAG        master  # or a specific tag/commit
)
FetchContent_MakeAvailable(mio)

add_executable(my_app main.cpp)
target_link_libraries(my_app PRIVATE mio::mio)

Using find_package

If mio is installed on your system:

find_package(mio REQUIRED)
target_link_libraries(my_app PRIVATE mio::mio)

As a Subdirectory

Copy or add mio as a git submodule to your project:

add_subdirectory(external/mio)
target_link_libraries(my_app PRIVATE mio::mio)

Windows API Targets

On Windows, mio::mio defines WIN32_LEAN_AND_MEAN and NOMINMAX to minimize the Windows API surface and prevent macro conflicts. If this causes issues, use the alternative targets:

# Full Windows API (no WIN32_LEAN_AND_MEAN or NOMINMAX)
target_link_libraries(my_app PRIVATE mio::mio_full_winapi)

# Minimal Windows API (explicit WIN32_LEAN_AND_MEAN and NOMINMAX)
target_link_libraries(my_app PRIVATE mio::mio_min_winapi)

Single Header

A single-header version is available at single_include/mio/mio.hpp:

#include <mio/mio.hpp>  // single header version

The single header is regenerated automatically during CMake configuration, or manually via:

cmake --build build --target mio_amalgamate

Building and Testing

mkdir build && cd build
cmake -DMIO_BUILD_TESTS=ON ..
cmake --build .
ctest --output-on-failure

CMake Options

Option	Default	Description
`MIO_BUILD_TESTS`	`ON` (standalone)	Build test suite
`MIO_BUILD_SINGLE_HEADER`	`ON` (standalone)	Generate single-header amalgamation
`MIO_WINDOWS_FULL_API`	`OFF`	Disable WIN32_LEAN_AND_MEAN and NOMINMAX
`MIO_INSTALL`	`ON` (standalone)	Enable installation targets

API Reference

Type Aliases

// Read-only mappings
using mmap_source = basic_mmap_source<char>;
using ummap_source = basic_mmap_source<unsigned char>;
using bmmap_source = basic_mmap_source<std::byte>;

// Read-write mappings
using mmap_sink = basic_mmap_sink<char>;
using ummap_sink = basic_mmap_sink<unsigned char>;
using bmmap_sink = basic_mmap_sink<std::byte>;

// Shared ownership variants
using shared_mmap_source = basic_shared_mmap_source<char>;
using shared_mmap_sink = basic_shared_mmap_sink<char>;
// ... and unsigned char / std::byte variants

Factory Functions

// Create mmap from path, handle, or file descriptor
template<typename MappingToken>
mmap_source make_mmap_source(const MappingToken& token, size_type offset,
                              size_type length, std::error_code& error);
template<typename MappingToken>
mmap_source make_mmap_source(const MappingToken& token, std::error_code& error);

template<typename MappingToken>
mmap_sink make_mmap_sink(const MappingToken& token, size_type offset,
                          size_type length, std::error_code& error);
template<typename MappingToken>
mmap_sink make_mmap_sink(const MappingToken& token, std::error_code& error);

// Create shared_mmap (no throwing constructors available)
template<typename MappingToken>
shared_mmap_source make_shared_mmap_source(const MappingToken& token, size_type offset,
                                            size_type length, std::error_code& error);
template<typename MappingToken>
shared_mmap_source make_shared_mmap_source(const MappingToken& token, std::error_code& error);

template<typename MappingToken>
shared_mmap_sink make_shared_mmap_sink(const MappingToken& token, size_type offset,
                                        size_type length, std::error_code& error);
template<typename MappingToken>
shared_mmap_sink make_shared_mmap_sink(const MappingToken& token, std::error_code& error);

Constants

inline constexpr size_t map_entire_file = 0;  // Pass as length to map entire file
inline constexpr auto invalid_handle = ...;   // Platform-specific invalid handle value

Page Size

#include <mio/page.hpp>

size_t page = mio::page_size();  // System page allocation granularity

Error Handling

mio provides two error handling mechanisms:

Exception-based (Constructors)

Constructors for mmap_source and mmap_sink throw std::system_error on failure:

try {
    mio::mmap_source mmap("nonexistent.txt");
} catch (const std::system_error& e) {
    std::cerr << "Mapping failed: " << e.what() << "\n";
    std::cerr << "Error code: " << e.code().value() << "\n";
}

Note: shared_mmap_source and shared_mmap_sink do not have throwing constructors. Use factory functions instead:

std::error_code ec;
auto shared = mio::make_shared_mmap_source("file.txt", ec);
if (ec) { /* handle error */ }

Error code-based (Factory functions and map())

Factory functions and map() report errors via std::error_code:

std::error_code ec;
mio::mmap_source mmap;
mmap.map("file.txt", ec);

if (ec) {
    std::cerr << "Mapping failed: " << ec.message() << "\n";
    // ec.value() contains the system error code
    // ec.category() identifies the error category
}

Common Error Conditions

Condition	Cause
`std::errc::no_such_file_or_directory`	File does not exist
`std::errc::permission_denied`	Insufficient permissions
`std::errc::invalid_argument`	Empty path, null pointer, or invalid offset
`std::errc::bad_file_descriptor`	Invalid file handle
`std::errc::not_enough_memory`	System cannot allocate mapping

Checking Mapping State

mio::mmap_source mmap;
mmap.map("file.txt", ec);

if (mmap.is_open()) {
    // Mapping is valid and can be used
}

if (mmap.empty()) {
    // No data mapped (size == 0)
}

Undefined Behavior

The following operations result in undefined behavior and must be avoided:

Accessing Unmapped Memory

mio::mmap_source mmap;
// UB: mmap is not mapped
char c = mmap[0];        // UB
auto* p = mmap.data();   // Returns nullptr, dereferencing is UB

Always check is_open() before accessing data:

if (mmap.is_open()) {
    char c = mmap[0];  // OK
}

Out-of-Bounds Access

mio::mmap_source mmap("file.txt");  // file has 100 bytes
char c = mmap[100];  // UB: valid indices are 0-99

Using Mapping After Unmap

mio::mmap_source mmap("file.txt");
const char* ptr = mmap.data();
mmap.unmap();
char c = *ptr;  // UB: ptr is dangling

File Modification During Mapping

Modifying a file's size while it is memory mapped leads to undefined behavior:

mio::mmap_source mmap("file.txt");
// UB: truncating or extending the file while mapped
truncate("file.txt", 0);

Mismatched Access Modes

Using a file handle with different permissions than the mapping mode:

int fd = open("file.txt", O_RDONLY);
// Potential failure or UB: handle is read-only but sink requires write
mio::mmap_sink mmap(fd, 0, mio::map_entire_file);

Null Path to Constructor

const char* path = nullptr;
mio::mmap_source mmap(path);  // UB: null pointer

Use factory functions or map() with error codes for safe handling:

std::error_code ec;
mio::mmap_source mmap;
mmap.map(static_cast<const char*>(nullptr), ec);
// ec will be set to std::errc::invalid_argument

Thread Safety

mio mappings are not thread-safe. Concurrent access to the same mmap object from multiple threads without synchronization is undefined behavior. However, multiple threads can safely read from the same mapped memory region if the mmap object itself is not modified.

mio::mmap_source mmap("file.txt");

// OK: Multiple threads reading mapped data
std::thread t1([&]{ auto c = mmap[0]; });
std::thread t2([&]{ auto c = mmap[1]; });

// UB: One thread modifying mmap object while others use it
std::thread t3([&]{ mmap.unmap(); });  // UB if t1/t2 still running

Platform Notes

Windows

Wide character paths are supported via std::filesystem::path
UTF-8 encoded paths are automatically converted
File handles (HANDLE) can be used directly

macOS

Requires macOS 10.15+ (Catalina) for std::filesystem support
Uses standard POSIX mmap APIs

Linux

Uses standard POSIX mmap APIs
Works with any file descriptor

License

MIT License - see LICENSE for details.

Credits

This library is based on mio by mandreyel. The original implementation provided the foundation for this modernized C++17 version.

Changes from the original:

Upgraded minimum C++ standard from C++11 to C++17
Replaced SFINAE patterns with if constexpr and static_assert
Added std::filesystem::path support (required)
Added std::byte type aliases
Added std::span conversion (C++20)
Added [[nodiscard]] attributes
Simplified string handling using std::string_view
Replaced Python amalgamation script with pure CMake solution
Modernized CMake build system
Added GitHub Actions CI for Linux, macOS, and Windows
Changed shared_mmap API to use factory functions (avoids constructor overload ambiguity on Windows)

Name		Name	Last commit message	Last commit date
Latest commit History 153 Commits
.github/workflows		.github/workflows
cmake		cmake
include/mio		include/mio
single_include/mio		single_include/mio
test		test
.gitignore		.gitignore
CMakeLists.txt		CMakeLists.txt
LICENSE		LICENSE
README.md		README.md

Folders and files

Latest commit

History

Repository files navigation

mio