中文文档

GS3D (Generic S3 Downloader) is a versatile Python tool designed specifically for downloading entire folders and their contents from AWS S3 buckets. It provides a simple yet powerful command-line interface that enables users to easily retrieve files from S3 buckets.

• Multi-threaded parallel downloads: Utilize multi-threading technology to download multiple files simultaneously, significantly improving download speed
• Anonymous access support: Access public buckets without AWS credentials
• Directory structure management: Flexibly choose whether to preserve the complete S3 directory structure
• Real-time progress display: Visually track download progress through a progress bar
• Multiple authentication methods: Support for AWS configuration profiles, access keys, and default credentials

• Python 3.6+
• pip (Python package manager)

pip install boto3 tqdm

Clone this repository or download the script directly:

git clone https://github.com/MEKXH/gs3d.git
cd gs3d

python GS3D.py s3://my-bucket/my-folder/ --output-dir ./downloads

python GS3D.py s3://public-bucket/folder/ --anonymous

python GS3D.py s3://my-bucket/folder/ --keep-structure --output-dir ./downloads

python GS3D.py s3://my-bucket/folder/ --profile my-profile-name

python GS3D.py s3://geos-chem/GEOS_2x2.5/MERRA2/2024/02/ --anonymous --region us-east-1 --output-dir ./climate-data

python GS3D.py s3://my-company/project-assets/ --profile work --output-dir ./backup --keep-structure

When running on an EC2 instance with an IAM role configured, no credentials are needed:

python GS3D.py s3://internal-data/reports/ --output-dir /mnt/data

Complete documentation is available on our official documentation site.

If you want to run the documentation site locally:

# Install dependencies
pnpm install

# Start development server
pnpm docs:dev

Then visit http://localhost:5173 in your browser.

This project provides two deployment scripts for deploying documentation to GitHub Pages:

The full version script provides detailed logging and error handling:

.\scripts\deploy.ps1

The quick version script is more concise but requires the repository URL:

.\scripts\quick-deploy.ps1 -RepoUrl "https://github.com/MEKXH/gs3d.git"

Both scripts support the following parameters:

• -RepoUrl: GitHub repository URL
• -BranchName: Deployment branch name (default: gh-pages)
• -Force: Force push (overwrite remote history)

gs3d/
├── GS3D.py              # Main script file
├── docs/                # Documentation source files
│   ├── .vitepress/      # VitePress configuration
│   ├── public/          # Static resources
│   ├── guide/           # Guide documentation
│   ├── introduction/    # Introduction documentation
│   ├── api/             # API reference
│   └── index.md         # Homepage
├── scripts/             # Utility scripts
│   ├── deploy.ps1       # Full deployment script
│   └── quick-deploy.ps1 # Quick deployment script
├── package.json         # Project configuration
└── README.md            # Project description

Contributions are welcome! Feel free to submit issues or pull requests. For major changes, please open an issue first to discuss what you would like to change.

This project is released under the MIT License.

• boto3 - Python SDK for interacting with AWS services
• tqdm - For displaying progress bars
• VitePress - For building the documentation site

If you have any questions or suggestions, please open an issue