• Summary
• Pretrained model
• Requirements
• Installation
  • Conda environment
  • Data
  • DINOv2 weights
  • Environment variables
• Training Kinaema models
• Evaluating Kinaema models on Mem-RPE
  • Results with the released checkpoint
• Citation

One key aspect of spatially aware robots is the ability to “find their bearings”, i.e. to correctly situate themselves in previously seen spaces. In this work, we focus on this particular scenario of continuous robotics operations, where information observed before an actual episode start is exploited to optimize efficiency. We introduce a new model, Kinaema, and agent, capable of integrating a stream of visual observations while moving in a potentially large scene, and upon request, processing a query image and predicting the relative position of the shown space with respect to its current position. Our model does not explicitly store an observation history, therefore does not have hard constraints on context length. It maintains an implicit latent memory, which is updated by a transformer in a recurrent way, compressing the history of sensor readings into a compact representation. We evaluate the impact of this model in a new downstream task we call “Mem-Nav”. We show that our large-capacity recurrent model maintains a useful representation of the scene, navigates to goals observed before the actual episode start, and is computationally efficient, in particular compared to classical transformers with attention over an observation history.

A pretrained Kinaema checkpoint is available here:

curl -LO https://download.europe.naverlabs.com/kinaema/kinaema.ckpt

The checkpoint can be evaluated directly — see Evaluating Kinaema models on Mem-RPE. For example:

bash scripts/eval_rpe.sh kinaema.ckpt test

Training and evaluation scripts are written for SLURM clusters. Running locally is possible (see Training) but requires manually managing environment variables.

env_name="kinaema_pth290_cu128"
conda create -y -n ${env_name} python=3.12
conda activate ${env_name}
pip install -e .

After installing the environment, configure the required paths in scripts/job_env.sh — see Environment variables below.

We use habitat-sim to render images at specific camera poses just-in-time when data is getting loaded. If you follow recommended installation instructions, you will end up with python version 3.9 or lower. We have chosen to circumvent this limitation and have been able to successfully install habitat-sim=0.3.3 with python>=3.12 and up-to-date dependencies using the following command:

conda install cmake
pip install -U attrs gitpython imageio imageio-ffmpeg matplotlib numba numpy numpy-quaternion pillow scipy tqdm
CMAKE_POLICY_VERSION_MINIMUM=3.5 pip install --ignore-requires-python --no-deps 'habitat-sim @ git+https://github.com/facebookresearch/habitat-sim@v0.3.3'

The numpy archives containing camera poses are available here as a tar file. You can un-tar it to the folder of your choice, then set DATA_ROOT_DIR in scripts/job_env.sh to point to the root kinaema directory where the data for the different splits are stored (see Environment variables). For example:

download_dir=/path/to/download
cd ${download_dir}
curl -LO https://download.europe.naverlabs.com/kinaema/kinaema_data.tar
tar -xf kinaema_data.tar
rm kinaema_data.tar
export DATA_ROOT_DIR=${download_dir}/kinaema  # and/or set it in job_env.sh

Additionally, the simulator needs the 3D assets of the different scenes where data have been generated. We used the Gibson database for Habitat-sim and the Habitat-Matterport 3D Research Dataset. Please follow these links to get detailed download instructions for each scenes dataset (license agreement for the former and matterport account creation for the latter). Set SCENES_ROOT_DIR in scripts/job_env.sh to point to the root directory where scenes datasets are stored (see Environment variables). For example:

curl -Lo gibson_habitat_trainval.zip <link-obtained-from-stanford-after-accepting-gibson-dataset-agreement>
mkdir -p data/scene_datasets
unzip gibson_habitat_trainval.zip -d data/scene_datasets
python -m habitat_sim.utils.datasets_download --data-path=data --username <matterport-api-token-id> --password <matterport-api-token-secret> --uids hm3d_train_v0.2 hm3d_val_v0.2
export SCENES_ROOT_DIR=data/scene_datasets  # and/or set it in job_env.sh

The image encoders in Kinaema are initialized from the pretrained DINOv2 weights, which are automatically downloaded from the official repository during the first training run. Set LOCAL_MODEL_DIR in scripts/job_env.sh to a directory with sufficient space (see Environment variables).

All paths are configured in scripts/job_env.sh. Set the following before running any script:

All cache/data directories should have sufficient free disk space (scene assets alone can be several hundred GB).

The training script is available at scripts/train_kinaema.sh. It is designed to be submittable via SLURM (sbatch), using 2× A100/H100/H200 GPUs by default.

# SLURM jobs (SLURM logs are written to out/slurm)
mkdir -p out/slurm
sbatch scripts/train_kinaema.sh /path/to/output [args]

# Local runs
bash scripts/train_kinaema.sh /path/to/output [args]

Checkpoints and logs are saved to <output_dir>/<SLURM_JOB_ID>/ (or <output_dir>/test_<timestamp>/ for local runs). Checkpoints are saved every 25 epochs by default.

Some of the key arguments passed to pretrain.py:

This repository contains the code for training and evaluating the Kinaema memory model on the Mem-RPE task (relative pose estimation from memory). The code for training the downstream navigation agent used in the Mem-Nav experiments is not released.

Evaluating the RPE performance of a trained Kinaema model can be done via the script scripts/eval_rpe.sh:

bash scripts/eval_rpe.sh /path/to/checkpoint.pth <dstype>

Supported values for <dstype>:

You can set evaluate on trajectories of certain length with the --seq-len argument, e.g.:

bash scripts/eval_rpe.sh /path/to/checkpoint.pth test --seq-len=200

Results (an accuracy.md table and a log.txt) are saved next to the checkpoint, under <checkpoint_dir>/eval_rpe/<checkpoint_name>/<dstype>/.

The table below reports the Mem-RPE accuracy of the released checkpoint on the test set (HM3D val scenes). Each <dist> - <angle> column reports the percentage of predictions with position error below <dist> and orientation error below <angle>, taken from the ALT table in the generated accuracy.md.

source ./scripts/job_env.sh
ckpt="<path/to/kinaema.ckpt>"
export DATA_ROOT_DIR="<path/to/kinaema/data>"  # see the Data section above

for seq_len in 200 800; do
    python -m kinaema.tools.eval_rpe \
        -K="${ckpt}" \
        -o="out/eval_rpe/${seq_len}" \
        -d="${DATA_ROOT_DIR}/test" \
        --seq-len=${seq_len} \
        -n=1000 \
        -b=2
done

The released checkpoint obtains slightly higher results than those reported in the paper for sequences of length 200, likely due to the retraining we performed after cleaning up the code for this release.

@inproceedings{sariyildiz2025kinaema,
      title={Kinaema: a recurrent sequence model for memory and pose in motion},
      author={Mert Bulent Sariyildiz and Philippe Weinzaepfel and Guillaume Bono and Gianluca Monaci and Christian Wolf},
      year={2025},
      booktitle={{Advances in Neural Information Processing Systems (NeurIPS)}},
      url={https://arxiv.org/abs/2510.20261},
}