magic is a Go package that identifies the type of a file or of bytes, as
the file command does. It is a port of
libmagic, the library of file, to pure Go.
It gives the output of libmagic 5.48, with the commits of file after the
release, to the commit in docs/upstream/upstreams.tsv: the same
description, MIME type, charset, extensions and Apple code. The rules of
file are compiled into the package. You need only Go 1.27 or later. You do
not need a C compiler or libmagic, and a program that imports magic builds
for every target of Go.
go get github.com/xo/magicDetect identifies bytes or a string:
r, err := magic.Detect(ctx, buf)
if err != nil {
return err
}
fmt.Println(r.Description) // GIF image data, version 89a, 1375 x 825
fmt.Println(r.MIME) // image/gif
fmt.Println(r.ContentType()) // image/gif; charset=binary
fmt.Println(r.Extensions) // [gif]A Result holds every form of the answer at once:
| Field | What it holds |
|---|---|
Description |
the text that file prints |
MIME |
the MIME type, such as image/gif |
Charset |
the MIME encoding, such as binary or utf-8 |
Extensions |
the usual extensions, without the dot, or nil |
Apple |
the creator and type code of classic Mac OS, or "" |
Matches |
each match, when WithAll is set, and else the one match |
DetectFile identifies a file. A directory, a link, a pipe or a device is
named by its kind, as file names it. A file that cannot be found or opened
is an error that wraps the error of the os package:
r, err := magic.DetectFile(ctx, "photo.jpg")DetectReader reads from an io.Reader, and DetectReaderAt reads from an
io.ReaderAt of a known size. A rule can read from the end of a file, so
DetectReaderAt gives the same answer as a file, and DetectReader does
not when the data is longer than what it reads.
Each function takes a context.Context first. A detection stops between the
rules when the context is done, and its error wraps magic.ErrStopped and
the error of the context. Every function is safe to call from many
goroutines at once.
The options change how a call detects:
| Option | What it does | Flag of libmagic |
|---|---|---|
WithAll() |
keeps every match, one on each line of Description, and one in each entry of Matches |
MAGIC_CONTINUE |
WithDecompress() |
identifies what is inside compressed data | MAGIC_COMPRESS |
WithLimit(n) |
reads and examines n bytes. The default is 7 MiB | bytes_max |
WithRaw() |
keeps the characters that cannot be printed, which are octal escapes by default | MAGIC_RAW |
WithFollowSymlinks() |
identifies the target of a link | MAGIC_SYMLINK |
WithDevices() |
reads a device | MAGIC_DEVICES |
WithoutCheck(c) |
turns off one built-in test, such as CheckJSON |
MAGIC_NO_CHECK_* |
WithDecompress decodes a format only when the program imports its
decoder, as a program imports the driver of database/sql. Each decoder
adds to the size of the program, so a program imports the ones it needs:
import (
_ "github.com/xo/magic/compress/bzip2" // bzip2
_ "github.com/xo/magic/compress/xz" // xz and lzma
_ "github.com/xo/magic/compress/zlib" // gzip and zlib
_ "github.com/xo/magic/compress/zstd" // zstd
)compress/xz and compress/zstd are Go modules of their own, because they
need other modules. Get them with go get as well. For a format with no
decoder, magic answers as libmagic does when it was built without the
library and finds no program that decodes it. magic never runs a program.
The package walk finds the files that are embedded in a binary file, such
as a firmware image, as binwalk does, and extracts them. It follows
binwalk-ng, the fork of binwalk
that is maintained. It has the 114 signatures of binwalk-ng, with the same
names and descriptions, and gives the same matches, at the commit of
binwalk-ng that docs/upstream/upstreams.tsv names:
for e, err := range walk.WalkFile(ctx, "firmware.bin", walk.WithDepth(8)) {
if err != nil {
return err
}
fmt.Printf("%*s%#x %s\n", 2*e.Depth, "", e.Offset, e.Description)
}A program imports the formats that it needs, or all of them:
import _ "github.com/xo/magic/walk/format/all"Entry.Members gives the files of a match in memory, and walk.Extract
writes them under an os.Root. binwalk extracts many formats with programs,
such as 7-Zip, sasquatch and tsk_recover. The walker runs no program, and
writes what those programs write. The formats of xz, lzma and zstd need the
decoders of compress/xz and compress/zstd, and the keys of D-Link firmware
are in walk/format/encrypted. docs/WALKER.md has the design, and
cmd/walk is an example program.
The formats build only where an int has 64 bits, as binwalk does.
A program that prints the MIME type of a file grows by about 980 KB with
magic, stripped, on linux/amd64. Most of it is the compiled rules. The
decoders add 45 KB for zlib, 74 KB for bzip2, 442 KB for xz and 594 KB
for zstd.
The test module builds libmagic from the source of file, at the commit
that the port follows, and compares the package with it, on 2,471 files and
damaged copies of them, with 14 combinations of flags, as bytes and as files.
Every answer is the same, except 50 where a decoder of Go reads damaged
compressed data in another way than the library of C. All 113 tests of
file pass. CONTRIBUTING.md says how to run the tests.
Up to v0.4.1, magic wrapped libmagic with cgo, and carried static archives
of it for eight targets. The port is the next minor version, and its API is
new (D39). docs/API.md gives the reasons.
To move from v0.4.1:
v0.4.1 |
The port |
|---|---|
m, err := magic.New(magic.MimeType), then m.Buffer(buf) |
r, err := magic.Detect(ctx, buf), then r.MIME |
m.File(name) |
magic.DetectFile(ctx, name) |
magic.None |
r.Description |
magic.MimeEncoding |
r.Charset |
magic.Mime |
r.ContentType() |
magic.Extension |
r.Extensions |
magic.Apple |
r.Apple |
magic.Continue |
magic.WithAll() |
magic.Compress |
magic.WithDecompress(), and a decoder from compress/ |
m.Close() |
nothing. The port has no handle |
magic.Version() |
magic.FileVersion |
magic.MagicDB() |
nothing. The port has no .mgc (D47) |
A file that cannot be opened is an error now, where libmagic gave the text "cannot open".
| Document | What it holds |
|---|---|
README.md |
this page, for a person who uses the package |
CONTRIBUTING.md |
how to change the package and test it |
AGENTS.md |
the rules for a coding agent. CLAUDE.md imports it |
docs/PLAN.md |
the plan of the port and its phases |
docs/API.md |
the design of the API |
docs/WALKER.md |
the design of the binary walker, and the formats of binwalk |
docs/decisions/ |
every decision, one file each, with an index |
docs/UPSTREAM.md |
the rules for following file, the upstream of the port |
docs/BACKLOG.md |
the work that is known and not done, and the binary walker |
magic is one of the xo projects. Several of them, as magic does, port a
library of C or Rust to pure Go. Each one is a separate repository:
| Project | What it is |
|---|---|
| okage | A Go port of tiny-skia and resvg, which draws 2D graphics and SVG images |
| transit | A Go port of tree-sitter, a parser of source code |
| rline | A readline package for Go, which reads a line of text that a person edits. It started as a port of isocline |
| zuzu | Draws DOT graphs, Mermaid diagrams, and Vega and Vega-Lite charts |
| ox | Parses the flags and arguments of a command line |
| usql | A command line client for SQL and NoSQL databases |
iv, an image viewer for the terminal, uses
magic to find the type of each file that it shows.
The code of magic that is not a translation, such as the API and the tests,
is under the MIT license, in LICENSE. The port translates file, which is
under its own BSD license, in LICENSE.file. Each translated Go file opens
with the notice of the file of file that it translates, and the compiled
rules carry the notices of file and of its rule files, in
internal/table/LICENSE. The binary walker in walk/ translates binwalk-ng, which is under the MIT license, in LICENSE.binwalk,
and holds copies of two decoders of the Go standard library, under the BSD
license of Go, in walk/internal/flate/LICENSE and
walk/internal/bzip2/LICENSE.
A program that ships magic must reproduce these notices in its
documentation.