Sheltie

Sheltie is the leaner, nimbler cousin of lassie that knows a few more tricks.

• Changes from Lassie
• Overview
• Installation
• Methods of Retrieval
  • Command Line Interface
    • Extracting Content from a CAR
    • Fetch Example
  • HTTP API
    • Daemon Example
  • Golang Library
  • Roots, pieces and payloads
• Contribute
• License

• Sheltie is HTTP-only (no Bitswap, no Graphsync)
• Sheltie uses delegated routing V1 API to find providers instead of legacy IPNI, see filecoin-project/lassie#489
• Sheltie reconstructs DAGs across HTTP providers via frontier traversal when a provider returns an incomplete CAR
• Sheltie fully implements the client side of the trustless gateway spec
• Sheltie supports streaming extraction (--extract) to write UnixFS content directly to disk during retrieval

See below for more details.

This project is a fork of Protocol Labs' Lassie (https://github.com/filecoin-project/lassie) under Apache 2.0/MIT.

Sheltie is a retrieval client for Filecoin/IPFS using the HTTP Trustless Gateway protocol. It attempts whole-DAG retrieval first, then continues with frontier traversal when providers have partial content. This allows fetching data split across multiple providers (e.g., one has directory nodes, another has leaf blocks).

Download the sheltie binary from the latest release based on your system architecture, or install via the Go package manager:

$ go install github.com/parkan/sheltie/cmd/sheltie@latest

Optionally, download the go-car binary from the latest release based on your system architecture, or install the go-car package using the Go package manager:

$ go install github.com/ipld/go-car/cmd/car@latest

The go-car package makes it easier to work with files in the content-addressed archive (CAR) format, which is what sheltie uses to return the content it fetches. For the sheltie use-case, go-car can be used to extract the contents of a CAR into usable files -- though sheltie fetch --extract can do this directly during retrieval (see below).

The sheltie command line interface (CLI) is the simplest way to retrieve content from the Filecoin/IPFS network. The CLI is best used when needing to fetch content from the network on an ad-hoc basis. The CLI is also useful for testing and debugging purposes, such as making sure that a CID is retrievable from the network or from a specific provider.

The CLI can be used to retrieve content from the network by passing a CID to the sheltie fetch command:

$ sheltie fetch [-o <output file>] [--global-timeout <duration>] <CID>[/path/to/content]

The sheltie fetch command will return the content of the CID to a file in the current working directory by the name of <CID>.car. If the -o output flag is used, the content will be written to the specified file. Use --global-timeout to set an overall time limit for the entire retrieval.

fetch will also take as input IPFS Trustless Gateway style paths. If the CID is prefixed with /ipfs/, the remainder will be interpreted as a URL query, accepting query parameters that the Trustless Gateway spec accepts, including dag-scope=, entity-bytes=. For example, sheltie fetch '/ipfs/<CID>/path/to/content?dag-scope=all' will fetch the CID, the blocks required to navigate the path, and all the content at the terminus of the path.

More information about available flags can be found by running sheltie fetch --help.

These options can also be set via environment variables prefixed with SHELTIE_ (e.g., SHELTIE_DELEGATED_ROUTING_ENDPOINT). Legacy LASSIE_-prefixed variables are also still honored.

Let's grab some content from the Filecoin/IPFS network using the sheltie fetch command:

$ sheltie fetch -o fetch-example.car -p bafybeic56z3yccnla3cutmvqsn5zy3g24muupcsjtoyp3pu5pm5amurjx4

This will fetch the bafybeic56z3yccnla3cutmvqsn5zy3g24muupcsjtoyp3pu5pm5amurjx4 CID from the network and save it to a file named fetch-example.car in our current working directory.

The -p flag shows verbose provider events (indexer queries, connections, etc). Progress is shown by default; use -q to suppress it.

Note: try fetching bafybeibxtfn2zibw4olisv3aajmrqnozxtm2h4q6zvh7ea6s2foul4kkgm for a little easter egg.

To extract the contents of the fetch-example.car file we created in the previous example, we would run:

$ car extract -f fetch-example.car

To fetch and extract at the same time (recommended), use the --extract flag:

$ sheltie fetch --extract -p bafybeic56z3yccnla3cutmvqsn5zy3g24muupcsjtoyp3pu5pm5amurjx4

This streams content directly to disk as it arrives, without intermediate CAR files. Use --extract-to to specify a different output directory.

Note: The car extract tool requires seekable input and cannot process streamed CAR data. The --extract flag provides integrated extraction that works with sheltie's streaming retrieval.

You should now have a birb.mp4 file under bafybeic56z3yccnla3cutmvqsn5zy3g24muupcsjtoyp3pu5pm5amurjx4/ in your current working directory. Feel free to play it with your favorite video player!

The sheltie HTTP API allows one to run a web server that can be used to retrieve content from the Filecoin/IPFS network via HTTP requests. It's primarily retained for backwards compatibility with lassie.

The API server can be started with the sheltie daemon command:

$ sheltie daemon

Sheltie daemon listening on address 127.0.0.1:41443
Hit CTRL-C to stop the daemon

The port can be changed by using the -p port flag. Any available port will be used by default.

More information about available flags can be found by running sheltie daemon --help.

To fetch content using the HTTP API, make a GET request to the /ipfs/<CID>[/path/to/content] endpoint:

$ curl http://127.0.0.1:41443/ipfs/<CID>[/path/to/content]

By default, this will output the contents of the CID to stdout.

To save the output to a file, use the filename query parameter:

$ curl http://127.0.0.1:41443/ipfs/<CID>[/path/to/content]?filename=<filename> --output <filename>

CURL Note: With curl we need to also specify the --output <filename> option. However, putting the above URL into a browser will download the file with the given filename parameter value upon a successful fetch.

More information about HTTP API requests and responses, as well as the numerous request parameters that can be used to control fetch behavior on a per request basis, can be found in the HTTP Specification document.

The original lassie design aimed to accomodate the plurarlity of protocols in the IPFS/Filecoin ecosystems, simultaneously attempting retrievals over graphsync, bitswap, and http from multiple providers. This approach meaningfully bridged the two ecosystems and (theoretically) maximized the odds of successful retrieval, but at the cost of significant complexity (in general and due to the inter-protocol architectural differences specifically). As delegated routing and trustless gateway/HTTP based retrievals have gained prominence, sheltie aims to be the best possible retrieval client within a more focused conceptual scope. Think of it as a "trustless gateway aware cURL" (cf. https://curl.se/docs/ipfs.html)

The sheltie model focuses entirely on resolving providers (which may be individual filecoin SPs, kubo nodes, etc) via delegated routing endpoints and retrieving via the trustless gateway protocol. Collecting and forwarwding subgraphs from libp2p peers may be handled by specific gateways, however this is completely abstracted at the client level (apart from verifying blocks received).

The removal of bitswap does come at a cost: the original implementation of HTTP retrieval assumed only complete graphs would be returned for the requested root CID, and bailed out otherwise. Luckily, the trustless gateway (hereafter TG) spec indicates that an incomplete DAG should still be served, and it's up to the client to handle it. Therefore, we can recursively attempt to re-discover missing subgraphs, thereby achieving a similar behavior to the graph stitching that we get "for free" in bitswap (though efficient CAR stream retrieval of the subgraph is always preferred).

The other major change is moving to a fully streaming-by-default architecture; instead of keeping a temporary on-disk blockstore for the duration of the retrieval, we only keep a "frontier" of blocks seen and needed; this allows for retrieving potentially TiB-to-PiB scale datasets. Furthermore, the --extract option unpacks the car stream on the fly (assuming unixfs input) and constructs the files without needing to flush the CAR.

Use the sheltie library to fetch content programmatically from Go:

$ go get github.com/parkan/sheltie@latest

A minimal fetch:

package main

import (
	"context"
	"fmt"
	"os"

	"github.com/parkan/sheltie/pkg/sheltie"
	"github.com/parkan/sheltie/pkg/storage"
	"github.com/parkan/sheltie/pkg/types"
	"github.com/ipfs/go-cid"
	trustlessutils "github.com/ipld/go-trustless-utils"
)

func main() {
	ctx := context.Background()

	s, err := sheltie.NewSheltie(ctx)
	if err != nil {
		panic(err)
	}

	rootCid := cid.MustParse("bafybeic56z3yccnla3cutmvqsn5zy3g24muupcsjtoyp3pu5pm5amurjx4")
	store := storage.NewDeferredStorageCar(os.TempDir(), rootCid)
	request, err := types.NewRequestForPath(store, rootCid, "", trustlessutils.DagScopeAll, nil)
	if err != nil {
		panic(err)
	}

	stats, err := s.Fetch(ctx, request)
	if err != nil {
		panic(err)
	}

	fmt.Printf("Fetched %d blocks in %d bytes\n", stats.Blocks, stats.Size)
}

Notes:

• NewSheltie accepts variadic SheltieOptions (e.g. WithGlobalTimeout). Full list: pkg.go.dev/github.com/parkan/sheltie/pkg/sheltie.
• NewRequestForPath(store, cid, path, dagScope, byteRange) -- path is an optional path under the CID; dagScope controls DAG traversal at the path terminus (see HTTP spec).
• Fetch takes variadic types.FetchOptions.

Early days PRs are welcome!

This library is dual-licensed under Apache 2.0 and MIT terms.

Copyright 2025 Arkadiy Kukarkin Derived from Lassie, © 2022 Protocol Labs, Inc. Modifications licensed under Apache License 2.0 / MIT