A distributed object storage system built in Go, using JuiceFS for file storage and FoundationDB for metadata storage. The system implements a CRAQ-like (Chain Replication with Apportioned Queries) architecture for high availability and consistency.

The system consists of the following components:

• Manages file metadata and chunk information
• Uses FoundationDB as the underlying storage backend
• Provides a consistent view of the file system structure
• Tracks replication chains for each chunk
• Implements CRAQ-like consistency protocols

• Stores and retrieves chunk data
• Uses JuiceFS for actual data storage
• Implements chain replication for data consistency
• Provides parallel read capabilities from any node in the chain
• Ensures write operations go through the tail node

• Provides a high-level API for applications
• Optimizes read operations through parallel chunk reading
• Automatically selects the best node for reading based on preferences
• Handles chunk calculations and coordination

1. Metadata Querying

   • Client operations request metadata from MDS
   • MDS returns chunk information and replication chain details
   • CRAQ-like consistency protocol ensures metadata is consistent and highly available

2. Data Block Location

   • Clients compute chunk IDs based on offset and length requirements
   • Extract node information from the metadata
   • Can read from any node in the replication chain for flexibility

3. Parallel Data Reading

   • Clients send parallel requests to selected storage nodes
   • Node selection prefers local or low-latency nodes
   • Utilizes CRAQ properties to distribute load across nodes

• Go 1.21 or higher
• FoundationDB 6.2 or higher
• JuiceFS setup with object storage

1. Clone the repository:

   git clone https://github.com/lizhongxuan/jos.git
   cd jos
   

2. Install dependencies:

   go mod download
   

3. Build the components:

   go build -o bin/mds cmd/mds/main.go
   go build -o bin/sn cmd/sn/main.go
   go build -o bin/client-example cmd/client-example/main.go
   

./bin/mds --port=8000 --cluster=/path/to/fdb.cluster --log-level=info

./bin/sn --port=8001 --hostname=node1 --base-dir=/path/to/storage --object-uri=<juicefs-object-uri> --meta-uri=<juicefs-meta-uri> --log-level=info

./bin/client-example --mds=localhost:8000 --action=read --path=/myfile.txt --out=output.txt

./bin/client-example --mds=localhost:8000 --action=write --path=/myfile.txt --in=input.txt

./bin/client-example --mds=localhost:8000 --action=delete --path=/myfile.txt

• --port: The port to listen on (default: 8000)
• --cluster: Path to the FoundationDB cluster file
• --log-level: Log level (debug, info, warn, error)

• --port: The port to listen on (default: 8001)
• --hostname: The hostname of this storage node
• --base-dir: Base directory for storage (default: /tmp/distsys-storage)
• --object-uri: JuiceFS object storage URI
• --meta-uri: JuiceFS metadata URI
• --log-level: Log level (debug, info, warn, error)

• --mds: Address of the metadata service (default: localhost:8000)
• --action: Action to perform (read, write, delete)
• --path: Path of the file to operate on
• --out: Path to output file for read operation
• --in: Path to input file for write operation

Each chunk has a replication chain consisting of multiple storage nodes. The chain has a head node and a tail node:

• Writes are always sent to the tail node and propagate through the chain
• Reads can be served by any node in the chain
• The system maintains consistency through a CRAQ-like protocol

The client can read multiple chunks in parallel, optimizing throughput for large files:

1. Client requests file metadata from MDS
2. Calculates which chunks are needed based on offset and length
3. Issues parallel read requests to optimal nodes for each chunk
4. Combines chunks into a unified response

FoundationDB is used to store:

• File metadata (name, size, timestamps, etc.)
• Path to file ID mapping
• Chunk information for each file
• Replication chain information for each chunk

MIT License