SAM3 Training Repository

This repository is dedicated to training and fine-tuning the Meta Segment Anything Model 3 (SAM 3) with comprehensive automation scripts and orchestration tools.

SAM 3 is a powerful segmentation model that can segment objects in images and videos based on text prompts, geometry, and image exemplars. This repository contains the SAM 3 codebase as a submodule and provides a dedicated workspace for training and fine-tuning tasks with automated setup, validation, and execution workflows.

• sam3/ - SAM 3 codebase (git submodule from facebookresearch/sam3)
• scripts/ - Training orchestration scripts and individual task scripts
• plan/ - Implementation plan and task breakdown
• experiments/ - Training logs, configs, and outputs
• rf100-vl/ - Roboflow 100-VL dataset utilities
• pyproject.toml - Project dependencies and configuration (managed with uv)
• .venv/ - Virtual environment (created by uv venv)

• Python 3.9-3.12 (preferably 3.12)
• uv package manager (installation guide)
• CUDA-capable GPU (recommended for training)
• Git with submodule support

1. Clone this repository:

   git clone --recurse-submodules <your-repo-url>

   Or if you've already cloned without submodules:

   git submodule update --init --recursive

2. Create virtual environment and install dependencies:

   uv venv
   source .venv/bin/activate  # or use: uv run python ...
   uv sync

   This will install all dependencies defined in pyproject.toml, including:

   • SAM3 core dependencies (timm, numpy, tqdm, huggingface_hub, etc.)
   • Training dependencies (hydra-core, submitit, tensorboard, etc.)
   • Dataset utilities (roboflow)
   • Configuration management (python-dotenv)
   • Experiment tracking (wandb)

3. Install local packages in editable mode:

   # Install SAM3 with training extras
   uv pip install -e "sam3/[train]"
   
   # Install rf100-vl dataset utilities
   uv pip install -e rf100-vl/

4. Install PyTorch (choose based on your CUDA version):

   # For CUDA 12.6 (recommended)
   uv pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
   
   # For CUDA 12.1
   uv pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
   
   # For CPU only (not recommended for training)
   uv pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu

5. Optional: Install additional extras:

   # For development tools (pytest, black, ruff, etc.)
   uv sync --extra dev
   
   # For Jupyter notebook support
   uv sync --extra notebooks

This repository uses uv for dependency management with a pyproject.toml file at the root. The file includes:

• All SAM3 dependencies (core + training)
• Dataset utilities (roboflow)
• Development and notebook dependencies as optional extras

To add new dependencies, edit pyproject.toml and run uv sync to update the environment.

The recommended way to run training is using the master orchestration script:

bash scripts/run_training.sh \
    -c configs/roboflow_v100/roboflow_v100_full_ft_100_images.yaml \
    --mode local \
    --num-gpus 1

For detailed usage instructions, see scripts/README_RUN_TRAINING.md.

For manual training without orchestration, refer to the SAM 3 training documentation.

This repository includes a comprehensive training infrastructure organized into five main categories:

• Virtual environment setup and verification
• Python environment checks
• SAM3 dependency installation

• Data root variable management
• Roboflow dataset download automation
• Data directory structure validation
• Configuration file existence checks

• Mode selection (local vs. cluster)
• Resource allocation flags (GPUs, nodes, partitions)
• Task type selection (train vs. eval)

• Local training command construction
• Cluster training command construction
• Job array configuration for dataset sweeps

• Log directory feedback
• TensorBoard launch helper

All tasks have been implemented as individual scripts in the scripts/ directory. For a detailed breakdown of all tasks, see plan/plan.md.

The scripts/ directory contains modular scripts for each task:

• task_10_environment_preparation.sh - Environment setup
• task_11_verify_python_environment.sh - Python environment verification
• task_12_install_sam3_dependencies.sh - SAM3 dependency installation
• task_20_data_validation_and_configuration.sh - Data validation orchestration
• task_21_define_data_root_variables.sh - Data root variable setup
• task_22_download_roboflow_dataset.sh - Roboflow dataset download
• task_24_validate_data_directory_structure.sh - Directory structure validation (validates both rf100vl and ODinW datasets)
• task_25_config_file_existence_check.sh - Config file validation
• task_30_script_argument_parsing.sh - Argument parsing
• task_31_implement_mode_selection.sh - Mode selection logic
• task_32_implement_resource_allocation_flags.sh - Resource allocation
• task_33_implement_task_type_selection.sh - Task type selection
• task_40_execution_logic_construction.sh - Execution logic
• task_41_construct_local_training_command.sh - Local command construction
• task_42_construct_cluster_training_command.sh - Cluster command construction
• task_43_job_array_configuration.sh - Job array setup
• task_50_post_execution_and_monitoring.sh - Post-execution tasks
• task_51_log_directory_feedback.sh - Log directory extraction
• task_52_tensorboard_launch_helper.sh - TensorBoard helper

• SAM 3 Official Website
• SAM 3 Paper
• Original SAM 3 Repository
• Training Documentation
• Implementation Plan

This repository follows the SAM License. See the LICENSE file in the submodule for details.