A lightweight Python tool for managing YUM/DNF and DEB package repositories hosted on Amazon S3 (or S3-compatible storage). Designed for efficient package publishing without requiring full repository downloads.

• Efficient Updates: Add or remove packages without downloading existing RPMs/DEBs
• Metadata Manipulation: Direct XML manipulation for fast operations
• SQLite Database Support: Creates and maintains SQLite metadata for faster DNF/YUM queries
• S3 Native: Built for S3-backed repositories with pluggable local backend for testing
• Auto-Detection: Automatically detects architecture and distro version from packages
• Validation: Built-in repository integrity checks (XML + SQLite databases)
• Type-Specific Config: Separate configuration per repo type (RPM vs DEB) with shared fallbacks
• Package Replication: Replicate packages between distro versions (e.g., el9 to el10)
• Automatic Backups: Metadata is backed up before changes and restored on failure

• Python 3.6+
• AWS credentials configured (via ~/.aws/credentials, environment variables, or IAM role)

make setup-venv
source ~/.venv/yums3/bin/activate

This installs system dependencies (createrepo_c, rpm, etc.) using the appropriate package manager for your OS (brew on macOS, apt on Debian/Ubuntu, dnf on RHEL/Rocky), then creates a Python virtual environment and installs the Python dependencies (boto3, lxml).

Configuration uses a flat JSON format with dot-notated keys. Files are searched in order:

1. ./dg-repos.conf (local directory)
2. ~/.dg-repos.conf (user home)
3. /etc/dg-repos.conf (system-wide)

./yums3.py config backend.type s3
./yums3.py config backend.s3.bucket my-company-yum-repo
./yums3.py config backend.s3.profile production
./yums3.py config repo.cache_dir /tmp/yum-cache

./yums3.py config --list

Config keys support per-type overrides. For a key like backend.s3.bucket, the lookup order is:

1. backend.rpm.s3.bucket (type-specific, when running yums3)
2. backend.s3.bucket (shared fallback)
3. Default value

This lets RPM and DEB repos use different S3 buckets, AWS profiles, cache directories, etc. while sharing a single config file:

{
  "backend.type": "s3",
  "backend.s3.bucket": "shared-bucket",
  "backend.rpm.s3.bucket": "rpm-only-bucket",
  "backend.deb.s3.bucket": "deb-only-bucket",
  "backend.s3.profile": "production"
}

Command-line arguments override config file values:

./yums3.py --bucket other-bucket add my-package.rpm
./yums3.py --cache-dir /tmp/custom-cache add my-package.rpm
./yums3.py --profile production add my-package.rpm
./yums3.py --s3-endpoint-url https://s3.us-west-2.amazonaws.com add my-package.rpm

# Single package
./yums3.py add my-package-1.0.0-1.el9.x86_64.rpm

# Multiple packages
./yums3.py add package1.rpm package2.rpm package3.rpm

# Glob patterns
./yums3.py add /tmp/rpmbuild/RPMS/x86_64/*.rpm

# Skip confirmation (CI/CD)
./yums3.py add -y my-package.rpm

./yums3.py remove my-package-1.0.0-1.el9.x86_64.rpm
./yums3.py remove -y old-package1.rpm old-package2.rpm

./yums3.py validate el9/x86_64

./yums3.py replicate --src el9 --dst el10 libtorch myapp
./yums3.py replicate --src el9 --dst el10 --arch aarch64 -y myapp

./yums3.py add --no-validate my-package.rpm
./yums3.py remove --no-validate old-package.rpm

./debs3.py add my-package_1.0.0_amd64.deb
./debs3.py add -y package1.deb package2.deb
./debs3.py add --distribution noble --component main my-package.deb

./debs3.py remove my-package
./debs3.py remove --distribution focal --component main --architecture amd64 my-package

./debs3.py validate focal main amd64

./debs3.py replicate --src focal --dst noble my-package
./debs3.py replicate --src focal --dst noble --component main --arch amd64 -y my-package

s3://your-bucket/
├── el9/
│   ├── x86_64/
│   │   ├── package1.rpm
│   │   ├── package2.rpm
│   │   └── repodata/
│   │       ├── repomd.xml
│   │       ├── <checksum>-primary.xml.gz
│   │       ├── <checksum>-primary_db.sqlite.bz2
│   │       ├── <checksum>-filelists.xml.gz
│   │       ├── <checksum>-filelists_db.sqlite.bz2
│   │       ├── <checksum>-other.xml.gz
│   │       └── <checksum>-other_db.sqlite.bz2
│   └── aarch64/
│       └── ...
└── el8/
    └── ...

s3://your-bucket/
├── dists/
│   └── focal/
│       ├── Release
│       └── main/
│           └── binary-amd64/
│               ├── Packages
│               ├── Packages.gz
│               └── Packages.bz2
└── pool/
    └── main/
        └── m/
            └── myapp/
                └── myapp_1.0.0_amd64.deb

Instead of downloading all packages to regenerate metadata, the tool directly manipulates metadata XML/control files:

Adding: Downloads metadata only, generates metadata for new packages in a temp dir, merges XML trees, recalculates checksums, creates SQLite databases, and uploads.

Removing: Downloads metadata only, parses XML, removes package entries, updates checksums and SQLite databases, uploads modified metadata, deletes packages from S3.

Before any metadata changes, a timestamped backup is created in S3. On success the backup is cleaned up. On failure, metadata is automatically restored from the backup.

When adding packages, the tool checks checksums against existing packages. Exact duplicates are skipped, same-name packages with different checksums are updated.

sudo tee /etc/yum.repos.d/my-repo.repo <<EOF
[my-repo]
name=My Repository
baseurl=https://your-bucket.s3.amazonaws.com/el\$releasever/\$basearch
enabled=1
gpgcheck=0
EOF

sudo dnf makecache

sudo dnf install python3-dnf-plugin-s3

sudo tee /etc/yum.repos.d/my-repo.repo <<EOF
[my-repo]
name=My Private Repository
baseurl=s3://your-bucket/el\$releasever/\$basearch
enabled=1
gpgcheck=0
s3_enabled=1
EOF

• Not concurrent-safe: Multiple simultaneous updates can corrupt metadata. Use external locking (e.g., CI/CD pipeline serialization).
• Brief inconsistency window: During metadata updates, clients may see transient checksum mismatches. They will retry and succeed once the update completes.
• No repository signing: Individual RPM signing is supported and recommended; repository metadata signing (repomd.xml) is not implemented.

yums3/
├── core/
│   ├── __init__.py          # Exports and Colors utility
│   ├── backend.py           # Storage backend abstraction (S3 + local)
│   ├── cli.py               # Generic CLI interface
│   ├── config.py            # Configuration management (RepoConfig)
│   ├── constants.py         # Defaults, config file locations
│   ├── deb.py               # Debian repository manager
│   ├── sqlite_metadata.py   # SQLite database generation
│   └── yum.py               # YUM repository manager
├── scripts/                  # Utility scripts
├── tests/                    # Test suite
├── yums3.py                  # YUM CLI entry point
├── debs3.py                  # DEB CLI entry point
├── Makefile                  # setup-venv target
├── requirements.txt          # Python dependencies (boto3, lxml)
└── README.md

source ~/.venv/yums3/bin/activate
python3 tests/test_config.py
python3 tests/test_storage_backend.py
python3 tests/test_sqlite_integration.py

MIT License. See LICENSE for details.