Instella-VL-1B: First AMD Vision Language Model

Welcome to the official repository for Instella-VL-1B, AMD's first ever Vision-Language Model (VLM). This repository provides a detailed guide for training and inference with Instella-VL-1B. Developed from AMD's Instella-1B (previously known as AMD OLMo 1B SFT LLM), this model is fully open-source, with both model weights and training code available for AMD GPUs (MI300). Its compact size aims to make it accessible to a broad spectrum of researchers, developers, and enthusiasts, enabling them to build upon, modify, and integrate it into their own projects.

In this repository, you'll find everything you need to get started with Instella-VL-1B, including detailed documentation, example code, and best practices for both training and inference.

We compare our model with models which partially releases the training details (with * in the below table) and also models which releases weights, data curation and all training details.

Use the Dockerfile as well as .dockerignore to build the image as shown below.

# Assuming you are inside the InstellaVL repo
# Save build logs at build_instella.log. Use --no-cache before -t to build fresh.
docker build -t instellavl_image:latest . > build_instella.log 2>&1
# Save the image at some common/shared storage space
docker save -o /path/to/common-space/instellavl.rar instellavl_image:latest

Load the above image as interactive docker container.

# Load docker
docker load -i /path/to/common-space/instellavl.rar
# Now 
docker run -it \ 
  --gpus all \ 
  --network=host \ 
  --device=/dev/kfd \ 
  --device=/dev/dri \ 
  --device=/dev/infiniband \ 
  --group-add=video \ 
  --ipc=host \ 
  --cap-add=SYS_PTRACE \ 
  --security-opt seccomp=unconfined \ 
  --privileged -v /cephfs:/cephfs \ 
  --shm-size 8G \ 
  --name="instellavl_container" \
  rocm/instellavl:latest \
  /bin/bash

This assist us with multinode docker training, ROCm powered bitsandbytes, flash-attention-v2, and apex.

pip install -e . --no-deps

import torch

# Import essential modules
from instellavl.constants import DEFAULT_IMAGE_TOKEN, IMAGE_TOKEN_INDEX
from instellavl.conversation import conv_templates, SeparatorStyle
from instellavl.model.builder import load_pretrained_model
from instellavl.utils import disable_torch_init
from instellavl.mm_utils import process_images, tokenizer_image_token, get_model_name_from_path

from PIL import Image

import requests
from io import BytesIO

# Login into HF Hub
from huggingface_hub import login
login(token = "<Your HFtoken id>") # Enter your token 

def load_image(image_file):
    if image_file.startswith("http") or image_file.startswith("https"):
        response = requests.get(image_file)
        image = Image.open(BytesIO(response.content)).convert("RGB")
    else:
        image = Image.open(image_file).convert("RGB")
    return image

#
# ========= CHANGE IMAGE and Query only HERE ============
image_file = '/path/to/InstellaVL-repo/assets/images/example2_dog.jpg' # Enter the test image path
query = 'Describe this image.'
# =======================================================

disable_torch_init()
conv_mode = 'instella'

# Model loading
model_path = '<path/to/model-checkpoint-saved-locally>' # Enter your model path, should contain instellavl substring in the name.
model_name = get_model_name_from_path(model_path)
tokenizer, model, image_processor, context_len = load_pretrained_model(model_path, None, model_name, False, False)
model.eval()
model = model.to('cuda') # change to 'cpu' if not 'cuda'

# Image pre-processing
image = load_image(image_file)
image_tensor = process_images([image], image_processor, model.config)
image_tensor = image_processor.preprocess(image, return_tensors="pt")["pixel_values"].to(model.dtype)

# Text pre-processing - follow the below logic too when there is no Image:
# if images is not None and len(image_tensor) != 0 and DEFAULT_IMAGE_TOKEN not in text:
#     question = DEFAULT_IMAGE_TOKEN + "\n" + text
# else:
#     question = text
query = query.replace(DEFAULT_IMAGE_TOKEN, "").strip()
question = DEFAULT_IMAGE_TOKEN + "\n" + query
conv = conv_templates[conv_mode].copy()
conv.append_message(conv.roles[0], question)
conv.append_message(conv.roles[1], None)
prompt_question = conv.get_prompt()

# Final arrangements required
input_ids = tokenizer_image_token(prompt_question, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0)
keywords = [conv.sep]
image_sizes = [image.size]
stopping_criteria = [KeywordsStoppingCriteria(keywords, tokenizer, input_ids)]
terminators = [tokenizer.eos_token_id, tokenizer.convert_tokens_to_ids("|||IP_ADDRESS|||")]

with torch.inference_mode():
    output_ids = model.generate(input_ids.to(model.device), images=image_tensor.to(model.device), image_sizes=image_sizes, do_sample=True, num_beams=1, temperature=0.2, max_new_tokens=1024, use_cache=True, stopping_criteria=stopping_criteria, eos_token_id=terminators)

outputs = tokenizer.decode(output_ids[0, input_ids.shape[1] :]).strip()
print("InstellaVL: ", outputs)

import os
import torch
from transformers import AutoTokenizer, AutoProcessor, AutoConfig, AutoModelForCausalLM

from PIL import Image
import requests
from io import BytesIO

def load_image(image_file):
    if image_file.startswith("http") or image_file.startswith("https"):
        response = requests.get(image_file)
        image = Image.open(BytesIO(response.content)).convert("RGB")
    else:
        image = Image.open(image_file).convert("RGB")
    return image


config = AutoConfig.from_pretrained("amd/Instella-VL-1B", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("amd/Instella-VL-1B", config=config, trust_remote_code=True)
processor = AutoProcessor.from_pretrained("amd/Instella-VL-1B", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("amd/Instella-VL-1B", trust_remote_code=True).to('cuda') # or 'cpu'
model.eval()

query="Describe the image."
image=load_image("path/to/your_image")
out = processor.encode(query, image, model.get_vision_tower().image_processor, tokenizer, config)
inputs = {k: v.to(model.device) for k, v in out.items() if isinstance(v, torch.Tensor)}
with torch.inference_mode():
    output_ids = model.generate(inputs["input_ids"], images=inputs['image_tensor'], image_sizes=out['image_sizes'], do_sample=True, num_beams=1, temperature=0.2, max_new_tokens=1024, use_cache=True, stopping_criteria=out['stopping_criteria'], eos_token_id=out['eos_token_id'])
outputs = processor.decode(output_ids)
print("InstellaVL: ", outputs)

Each node has 8 AMD MI300 GPU cards.
MLP warmup: 1 node
Pretraining: 2 nodes
Finetune: 4 nodes

We have 3 stages of training:

1. MLP Warmup: Warmimg up the 2-layer MLP projector (otherwise known as Adapter) while keeping other components frozen. To do the same, we used BLIP558k dataset.
2. Pretraining: The entire model is pre-trained in NTP (Next-Token Prediction) fashion on various Image-Caption datasets. Details to datasets used can be found in the table below.
3. SFT: In this final stage of Supervised Fine-Tuning, we emphasized upon instruction following ability of the model using several vision instruction datasets spanning across various domains. Further details mentioned below.

1. First, construct playground/data/ directory. In this we have two options of loading data,
   1. Directly from Image folders

      1. First, we need to construct respective data folders involved in each stage of training.

         mkdir -p playground/data/warmup_data
         bash scripts/1B_release/prepare_warmup_data_folders.sh
         mkdir -p playground/data/pretraining_data
         bash scripts/1B_release/prepare_pretrain_data_folders.sh
         mkdir -p playground/data/instruction_tuning_data
         bash scripts/1B_release/prepare_sft_data_folders.sh

         TL;DR: Folder structure after creation

         playground
         └──data
             ├──warmup_data
             │  └──instella_alignment
             │     ├──blip_laion_cc_sbu_558k.json
             │     └──images
             ├──pretrain_data
             │  ├──json_files_processed
             │  ├──LLaVA-Stage1.5
             │  ├──ArT
             │  ├──chart2text_pew
             │  ├──chart2text_statista
             │  ├──coco_text
             │  ├──docvqa
             │  ├──hiertext
             │  ├──mlt
             │  ├──mmc_instruction_arxiv
             │  ├──mmc_instruction_non-arxiv
             │  ├──M-Paper-analysis
             │  ├──M-Paper-cap
             │  ├──M-Paper-outline
             │  ├──openvino
             │  ├──Paper2Fig100k
             │  ├──PlotQA
             │  ├──Rico_ScreenQA
             │  ├──RICO-Screen2Words
             │  ├──SciGraphQA-295K-train
             │  ├──st_vqa
             │  ├──textocr
             │  ├──Uber-Text
             │  ├──UniChart
             │  ├──DocStruct4M_struct_aware_parse
             │  ├──DocStruct4M_multi_grained_text_localization
             │  ├──pixmo_cap_part_1
             │  ├──pixmo_cap_part_2
             │  ├──pixmo_cap_part_3
             │  ├──pixmo_cap_part_4
             │  └──pixmo_cap_part_5
             └──instruction_tuning_data
                ├──json_files_processed
                ├──VQA-RAD
                ├──infographic_vqa
                ├──iconqa
                ├──TabMWP 
                ├──scienceqa_nona_context
                ├──scienceqa
                ├──ureader_kg
                ├──aokvqa
                ├──k12_printing
                ├──tqa
                ├──mavis_math_metagen
                ├──magpie_pro_qwen2_72b_st
                ├──textocr
                ├──geo3k
                ├──TQA
                ├──magpie_pro_l3_80b_mt
                ├──clevr
                ├──hitab
                ├──VQA-AS
                ├──robut_wtq
                ├──ureader_ie
                ├──hme100k
                ├──MapQA
                ├──IconQA
                ├──diagram_image_to_text
                ├──geo170k_qa
                ├──rendered_text
                ├──websight
                ├──GEOS
                ├──llavar
                ├──chartqa
                ├──multihiertt
                ├──cambrian_selection
                ├──ureader_cap
                ├──lrv_normal
                ├──robut_sqa
                ├──CLEVR-Math
                ├──chrome_writting
                ├──raven
                ├──robut_wikisql
                ├──FigureQA
                ├──infographic_vqa_llava_format
                ├──hateful_memes
                ├──geo170k_align
                ├──ureader_qa
                ├──vsr
                ├──dvqa
                ├──magpie_pro_l3_80b_st
                ├──image_textualization
                ├──lrv_chart
                ├──iam
                ├──docvqa
                ├──infographic_gpt4v
                ├──vizwiz
                ├──visual7w
                ├──llava_next
                ├──iiit5k
                ├──sharegpt4v
                ├──ai2d_gpt4v
                ├──ai2d_internvl
                ├──Super-CLEVR
                ├──vqarad
                ├──evol_instruct
                ├──visualmrc
                ├──ai2d_original
                ├──geomverse
                ├──allava_instruct
                ├──mathqa
                ├──Geometry3K_MathV360K
                ├──vision_flan
                ├──GeoQA+
                ├──st_vqa
                ├──figureqa
                ├──chart2text
                ├──sharegpt4o
                ├──PMC-VQA
                ├──screen2words
                ├──mapqa
                ├──intergps
                ├──UniGeo
                ├──textcaps
                ├──vistext
                ├──mavis_math_rule_geo
                ├──tallyqa
                ├──tabmwp
                ├──arxivqa
                ├──kvqa
                ├──visual_dialog
                ├──coco-goi
                ├──coco-itm
                ├──image-paragraph-captioning
                ├──imagenet
                ├──iqa
                ├──localized_narratives
                ├──mocheg
                ├──shapes
                ├──snli-ve
                ├──SVIT_core_150K
                ├──vcr
                ├──viquae
                ├──finqa
                ├──web-landmark
                ├──web-celebrity
                ├──alfworld
                ├──DocDownstream-1.0
                ├──DocReason25K
                ├──DocStruct4M_struct_aware_parse
                ├──DocStruct4M_multi_grained_text_localization
                ├──design2code
                ├──hfdata
                ├──lnqa
                ├──TinyChartData
                ├──sam
                ├──Q-Instruct-DB
                ├──mathvision
                ├──oodvqa
                ├──idk
                ├──gpt4v-dataset
                ├──pixmo_cap_qa_part_1
                ├──pixmo_cap_qa_part_2
                ├──pixmo_cap_qa_part_3
                ├──pixmo_cap_qa_part_4
                ├──pixmo_cap_qa_part_5
                ├──pixmo_count_shards
                ├──pixmo_points_part_1
                ├──pixmo_point_explanations_shards
                ├──pixmo_docs_tables_shards
                ├──pixmo_docs_other_shards
                ├──pixmo_docs_diagrams_shards
                ├──pixmo_docs_charts_shards
                └──pixmo_ask_model_anything_shards
         

      2. Next, download datasets from links provided above which compulsorily involves the image folder and then convert the dataset annotation with the below format saved as a json file .

         {
             "id": string,
             'image": <image_path>,
             "conversations': 
                 [
                 {"from": "human", "value": "<image>\n" + question/prompt},
                 {"from": "gpt", "value": answer},
                 .......
                 ]
         }

         We have provided some of our pre-processing script at playground/support. Do check it out and adapt it with datasets of different original format. Let us know if any issue you are facing with the tranformation.

   2. From mosaic shards
      1. Create a folder mosaic_shards as: playground/data/mosaic_shards.

         mkdir playground/data/mosaic_shards

      2. Now execute the script prepare_shards.sh to obtain mosaic shards at playground/data/mosaic_shards/.

         bash scripts/1B_release/data/prepare_shards.sh

To upload the above constructed shards (locally at playground/data/mosaic_shards) into S3 bucket, run the script upload_shards.sh:

bash scripts/1B_release/data/upload_shards.sh

Set the flag --online_training True to ensure training from S3 storage including dataloading from S3 and reading/writing checkpoints to S3.

We use lmms-eval ToolKit to evaluate the model performance. But before that do the required setup mentioned at 👉 here. The command is as follows:

# Assuming you are inside lmms_eval repo which is adjacent to InstellaVL repo
# With the same docker provided above, for MI300 GPUs, one can do the evaluation as well.
# Set the working directory, while loading docker, to the parent directory of lmms-eval where InstellaVL resides too.

export HF_TOKEN="<Your HF Token>"
accelerate launch --num_processes=8 \
 -m lmms_eval \
 --model instellavl \
 --model_args pretrained=path/to/sft-checkpoint-folder,conv_template=instella,model_name=instellavl \
 --task gqa,scienceqa_img,textvqa_val,pope,mmbench_en_dev,ai2d,chartqa,docvqa_val,infovqa_val,mmmu_val,realworldqa,mmstar,ocrbench,seedbench \
 --batch_size 1 \
 --log_samples \
 --log_samples_suffix instellavl \
 --output_path logs/

Core contributors: Ximeng Sun, Aditya Kumar Singh, Gowtham Ramesh, Zicheng Liu

Contributors: Pratik Prabhanjan Brahma, Ze Wang, Jiang Liu, Jialian Wu, Prakamya Mishra, Xiaodong Yu, Yusheng Su, Sudhanshu Ranjan, Emad Barsoum

This model is made accessible without any safety guarantees. Users should be aware that the model may generate outputs that are sensitive, inaccurate, harmful, biased, or otherwise objectionable based on user prompts. It is crucial for users to conduct comprehensive safety evaluations, implement safety filtering, and verify the model's outputs to mitigate these risks.

See Files for license and any notices.

If you find our code useful in your research, please consider giving a star ⭐ and citation 📝 :)

@misc{Instella-VL-1B, 
    title = {Instella-VL-1B: First AMD Vision Language Model},
    url = {https://huggingface.co/amd/Instella-VL-1B},
    author = {Ximeng Sun, Aditya Singh, Gowtham Ramesh, Jiang Liu, Ze Wang, Sudhanshu Ranjan, Pratik Prabhanjan Brahma, Prakamya Mishra,  Jialian Wu, Xiaodong Yu, Yusheng Su, Emad Barsoum, Zicheng Liu},
    month = {March},
    year = {2025}
}