Skip to content
gkostkaPublic

About

ext2/ext3/ext4 filesystem library for microcontrollers

Topics

Resources

Stars

601 stars

Watchers

26 watching

Forks

Repository files navigation

Join the chat at https://gitter.im/gkostka/lwext4 License (GPL v2.0) Build Status

lwext4

About

The main goal of the lwext4 project is to provide ext2/3/4 filesystem for microcontrollers. It may be an interesting alternative for traditional MCU filesystem libraries (mostly based on FAT32). Library has some cool and unique features in microcontrollers world:

  • directory indexing - fast file find and list operations
  • extents - fast big file truncate
  • journaling transactions & recovery - power loss resistance

Lwext4 is an excellent choice for SD/MMC card, USB flash drive or any other wear leveled memory types. However it is not good for raw flash devices.

Feel free to contact me: kostka.grzegorz@gmail.com

Getting started

lwext4 is a library: your program registers a block device (a struct with read/write callbacks for your storage), formats or mounts it, and then uses a file API much like stdio. Everything below runs on a PC first.

  1. Build the library, the tools and the examples (see Compile for the dependencies):
     make generic
     cd build_generic
     make
  2. Run the basic example and check its result with e2fsprogs:
     ./examples/lwext4-example-basic disk.img
     e2fsck -fn disk.img
     debugfs -R 'ls -l /docs' disk.img
    examples/basic/main.c walks through the whole life cycle step by step: block device, ext4_mkfs, mount, journal, write-back cache, directories and files, unmount.
  3. Port it to your hardware by writing a block device: examples/blockdev-template is an annotated skeleton that CI runs on a RAM disk.
  4. Pick the features and buffer sizes you need with the CONFIG_* options of include/ext4_config.h and build for your target with a toolchain file from toolchain/, e.g. make cortex-m4.

The core of a program, as a function (CI compiles this snippet, like every C snippet in this file):

#include <ext4.h>
#include <ext4_mkfs.h>
#include <blockdev/file_dev.h>

/* Format an image file as ext4, then write "hello" to /mp/hello.txt. */
int format_and_write(const char *image)
{
	static struct ext4_fs fs; /* used by ext4_mkfs only; large */
	struct ext4_mkfs_info info = {.block_size = 1024, .journal = true};
	struct ext4_blockdev *bd;
	ext4_file f;
	size_t written;
	int r;

	file_dev_name_set(image);  /* the block device: a file */
	bd = file_dev_get();

	r = ext4_mkfs(&fs, bd, &info, F_SET_EXT4);
	if (r != EOK)
		return r;

	r = ext4_device_register(bd, "disk");
	if (r != EOK)
		return r;
	r = ext4_mount("disk", "/mp/", false);
	if (r == EOK) {
		ext4_recover("/mp/");       /* replay the journal if needed */
		ext4_journal_start("/mp/");

		r = ext4_fopen(&f, "/mp/hello.txt", "wb");
		if (r == EOK) {
			r = ext4_fwrite(&f, "hello\n", 6, &written);
			ext4_fclose(&f);
		}

		ext4_journal_stop("/mp/");
		ext4_umount("/mp/");
	}
	ext4_device_unregister("disk");
	return r;
}

More examples:

  • examples/ - the host examples above and how to check what they write.
  • fs_test/ - the lwext4-generic, lwext4-mkfs and lwext4-mbr tools (see below) are complete programs too.

Credits

The most of the source code of lwext4 was taken from HelenOS:

Some features are based on FreeBSD and Linux implementations.

KaHo Ng (https://github.com/ngkaho1234):

  • advanced extents implementation
  • xattr support
  • metadata checksum support
  • journal recovery & transactions
  • many bugfixes & improvements

Lwext4 could be used also as fuse internals. Here is a nice project which uses lwext4 as a filesystem base:

Some of the source files are licensed under GPLv2. It makes whole lwext4 GPLv2 licensed. To use library as a BSD3, GPLv2 licensed source files must be removed first. At this point there are two files licensed under GPLv2:

  • ext4_xattr.c
  • ext4_extent.c

All other modules and headers are BSD-3-Clause licensed code.

Features

  • filetypes: regular, directories, softlinks
  • support for hardlinks
  • multiple blocksize supported: 1KB, 2KB, 4KB ... 64KB
  • little/big endian architectures supported
  • multiple configurations (ext2/ext3/ext4)
  • only C standard library dependency
  • various CPU architectures supported (x86/64, cortex-mX, msp430 ...)
  • small memory footprint
  • flexible configurations
  • partition tables: MBR including logical partitions, and GPT (ext4_partition_scan in ext4_partition.h)

Memory footprint

Advanced ext4 filesystem features, like extents or journaling require some memory. However most of the memory expensive features could be disabled at compile time. Here is a brief summary for cortex-m4 processor (arm-none-eabi-gcc 14, -O2, debug output disabled; measured by tests/acceptance/test-cortex-m.sh):

  • .text: 48KB - only ext2 fs support , 65KB - full ext4 fs feature set
  • RAM: 13KB - minimum 8 x 1KB block cache (9KB heap, 4.6KB static data), 19KB - when journaling and extents are enabled
  • .stack: 2KB - is enough (about 1KB measured)

Blocks are allocated dynamically. Previous versions of library could work without malloc but from 1.0.0 dynamic memory allocation is required. However, block cache should not allocate more than CONFIG_BLOCK_DEV_CACHE_SIZE blocks.

Supported ext2/3/4 features

incompatible:

  • filetype, recover, meta_bg, extents, 64bit, flex_bg: yes
  • compression, journal_dev, mmp, ea_inode, dirdata, bg_meta_csum, largedir, inline_data: no

A filesystem with an unsupported incompatible feature is not mounted (ENOTSUP). The exception is mmp, which is ignored: such a filesystem is mounted without multi-mount protection.

compatible:

  • has_journal, ext_attr, dir_index: yes
  • dir_prealloc, imagic_inodes, resize_inode: no

read-only:

  • sparse_super, large_file, huge_file, gdt_csum, dir_nlink, extra_isize, metadata_csum: yes
  • quota, bigalloc, btree_dir: no

A filesystem with an unsupported read-only feature is mounted read-only.

e2fsprogs 1.47 and later enable metadata_csum_seed and orphan_file by default, which lwext4 does not support. Create images for lwext4 without them: mke2fs -t ext4 -O ^metadata_csum_seed,^orphan_file ...

Project tree

  • blockdev - block devices set, supported blockdev
  • examples - example programs (see Getting started)
  • fs_test - test suite, mkfs and demo application
  • src - source files
  • include - header files
  • toolchain - cmake toolchain files
  • CMakeLists.txt - CMake config file
  • fs_test.mk - automatic tests definitions
  • Makefile - helper makefile to generate cmake and run test suite
  • README.md - readme file

Compile

Dependencies

  • Windows

Download MSYS-2: https://sourceforge.net/projects/msys2/

Install required packages is MSYS2 Shell package manager:

 pacman -S make gcc cmake
  • Linux

Package installation (Debian):

 apt-get install make gcc cmake

Compile & install tools

 make generic
 cd build_generic
 make
 sudo make install

Installation goes to /usr/local by default. To install somewhere else, pick the prefix at install time or when configuring:

 cmake --install build_generic --prefix $HOME/.local
 cmake -DCMAKE_INSTALL_PREFIX=$HOME/.local build_generic

Using the installed library

Installing puts the library, the file block device library and their headers below the prefix, together with a pkg-config file and a CMake package:

 include/lwext4/                 ext4.h, ext4_mkfs.h, ... generated/ext4_config.h
 include/lwext4/blockdev/        blockdev.h, file_dev.h (file_windows.h on Windows)
 lib/liblwext4.a                 the library (liblwext4.so with -DLWEXT4_BUILD_SHARED_LIB=ON)
 lib/libblockdev.a               file_dev_get(): an image file or device as ext4_blockdev
 lib/pkgconfig/lwext4.pc
 lib/cmake/lwext4/               lwext4Config.cmake, lwext4ConfigVersion.cmake, targets

Both package files find the prefix relative to their own location, so an install tree can be moved or unpacked elsewhere. A program includes <ext4.h> (and <blockdev/file_dev.h> for the file block device) and is built with any of:

 cc app.c $(pkg-config --cflags --libs lwext4)
 cc app.c -I$PREFIX/include/lwext4 -L$PREFIX/lib -lblockdev -llwext4
 find_package(lwext4 1.0 CONFIG REQUIRED)
 target_link_libraries(app lwext4::blockdev lwext4::lwext4)

(set PKG_CONFIG_PATH=$PREFIX/lib/pkgconfig or CMAKE_PREFIX_PATH=$PREFIX for a prefix outside the default search path). The install_package CTest test builds and runs such a program in all three ways; tests/package/ is a complete example.

lwext4-generic demo application

Simple lwext4 library test application:

  • load ext2/3/4 images
  • load linux block device with ext2/3/4 part
  • load windows volume with ext2/3/4 filesystem
  • directory speed test
  • file write/read speed test

How to use for images/blockdevices (make test or make images_small creates the ext_images directory):

 lwext4-generic -i ext_images/ext2 
 lwext4-generic -i ext_images/ext3 
 lwext4-generic -i ext_images/ext4 

Show full option set:

 lwext4-generic --help

Run automatic tests

The tests create their ext2/3/4 images with mke2fs (e2fsprogs), no root access is needed.

Execute tests for autogenerated 128MB images:

 make test

Execute tests for autogenerated 1GB images (only on Linux targets) + fsck:

 make test_all

Run regression tests

Small self-contained tests live in tests/ and run through CTest. They need mke2fs (e2fsprogs), sfdisk (util-linux) and python3 on the host to build their images, and pkg-config for the installed package test; no root access is required:

 make generic
 cd build_generic
 make
 ctest --output-on-failure

To run them with AddressSanitizer/UBSan, configure with -DLWEXT4_SANITIZE=address,undefined.

Other Linux architectures can be tested under qemu-user with the toolchain/linux-cross.cmake toolchain (gcc-<triple> and qemu-user packages). The byte order is taken from the compiler:

 cmake -S . -B build_s390x -DCMAKE_TOOLCHAIN_FILE=toolchain/linux-cross.cmake \
       -DCROSS_TRIPLE=s390x-linux-gnu -DCROSS_EMULATOR=qemu-s390x
 cmake --build build_s390x
 ctest --test-dir build_s390x --output-on-failure
 ci/scripts/fs-roundtrip.sh build_s390x qemu-s390x -L /usr/s390x-linux-gnu

fs-roundtrip.sh formats images with mke2fs and lwext4-mkfs, exercises them with lwext4-generic and checks the result with e2fsck.

The bare-metal and simulator toolchains build tests/baremetal, a test firmware (unit tests, read-only mounts of mke2fs images, ext4_mkfs and read/write tests on a RAM disk where there is RAM for one) that ctest runs on an emulator: cortex-m* on QEMU MPS2 boards, arm-sim on qemu-arm, atmega1284 on simavr and msp430-sim on msp430-elf-run. The MinGW build runs its tests under Wine.

Every CI job runs in a container defined in ci/, so the whole chain runs locally with only docker installed, exactly as in GitHub Actions (.github/workflows/ci.yml):

 ci/run.sh --list
 ci/run.sh qemu-user s390x
 ci/run.sh avr

See ci/README.md.

Using lwext4-mkfs tool

It is possible to create ext2/3/4 partition by internal library tool.

Generate empty file (1GB):

 dd if=/dev/zero of=ext_image bs=1M count=1024

Create ext2 partition:

 lwext4-mkfs -i ext_image -e 2

Create ext3 partition:

 lwext4-mkfs -i ext_image -e 3

Create ext4 partition:

 lwext4-mkfs -i ext_image -e 4

Show full option set:

 lwext4-mkfs --help

Cross compile standalone library

Toolchains needed:

Lwext4 could be compiled for many targets. Here are an examples for 8/16/32/64 bit architectures.

  • generic for x86 or amd64
  • arm-none-eabi-gcc for ARM cortex-m0/m3/m4 microcontrollers
  • avr-gcc for AVR xmega microcontrollers
  • msp430-gcc for msp430 microcontrollers

The library is tested on the host (x86-64, aarch64), on other Linux architectures under qemu-user (including big endian s390x) and on Cortex-M under QEMU, see "Run regression tests". For AVR and MSP430 compilation passes (with warnings somewhere) but tests are not done yet. The byte order is taken from the compiler (__BYTE_ORDER__, or the usual per architecture macros such as __ARMEB__/_MIPSEB). CONFIG_BIG_ENDIAN (1 or 0) overrides it for compilers that define none of them; a value that contradicts the compiler is a build error.

Build avrxmega7 library:

 make avrxmega7
 cd build_avrxmega7
 make lwext4

Build cortex-m0 library:

 make cortex-m0
 cd build_cortex-m0
 make lwext4

Build cortex-m3 library:

 make cortex-m3
 cd build_cortex-m3
 make lwext4

Build cortex-m4 library:

 make cortex-m4
 cd build_cortex-m4
 make lwext4

About

ext2/ext3/ext4 filesystem library for microcontrollers

Topics

Resources

Stars

601 stars

Watchers

26 watching

Forks

Releases

Packages

Used by

Contributors

Languages