Welcome to the ollama-for-amd wiki!

This wiki aims to extend support for AMD GPUs that Ollama Official doesn't currently cover due to limitations in official ROCm on Windows. The goal is to remove these GPU limitations and include support for more AMD graphics card models. Remember, this help is provided voluntarily by community.

(For those who prefer another language, right-click and use the auto-translate feature in your browser or device. Most platforms have this built-in!.)

Need Help?

issues? Please share your server logs in the issue section .also check out the official ollama issue tracker: (ollama/issues) for more in-depth assistance or get help from similiar issue.If you suspect the issue originates from the Ollama program itself, please hold off on further troubleshooting for now. ollama teams are working on updates to address this problem. Keep an eye out for a new release!"

If you're not comfortable compiling or build the software yourself, no problem! You can skip the development steps and jump straight to using the ** Step 7 ,Demo Release Version**.

To use the demo, simply replace the rocblas files in the Rocblas Support Section with those provided in the demo release.

（"This guide outlines how the repository builds the program. You don't need to follow these steps unless you're interested in customizing the build process. One benefit of following these steps is that the compiled program may take up less space, as it will only be optimized for your specific GPU architecture instead of a wide range of architectures.").

Follow the guide on development.md for Windows development setup.

1. Clone the Ollama Repository:

   git clone https://github.com/ollama/ollama.git

or simpley clone this repo.

2. Edit the CMakePresets.json File:

• Locate this file:CMakePresets.json

  • Add your GPU model to the existing list at line 59. Here's an example of the official supported list:

  "AMDGPU_TARGETS": "gfx900;gfx940;gfx941;gfx942;gfx1010;gfx1012;gfx1030;gfx1100;gfx1101;gfx1102;gfx906:xnack-;gfx908:xnack-;gfx90a:xnack+;gfx90a:xnack-"
  

  • Throughout the examples AMDGPU_TARGETS is all the arches,to speed up the build. You can build with your need arches to reduce build times. eg, DAMDGPU_TARGETS="gfx1031;"
  • Add your desired GPU models to this list (see examples in the original text). Example Additional GPUs for This Repo:（AMD-GPU-Arches-lists-Info) or get info from techpowerup.com

  "gfx803"(rocm5),"gfx900:xnack-"(rocm5), "gfx902" (rocm5),"gfx90c:xnack-"(rocm5,only available at self-build),//"gfx906:xnack-","gfx1010:xnack-", "gfx1011", "gfx1012:xnack-", "gfx1031", "gfx1032", "gfx1034", "gfx1035", "gfx1036","gfx1103", "gfx1150"

while also need to edit CMakeLists.txt in line 99

official support lists .

list(FILTER AMDGPU_TARGETS INCLUDE REGEX "^gfx(900|94[012]|101[02]|1030|110[012])$")

meaning : gfx900 gfx940、gfx941、gfx942 gfx1010、gfx1012 gfx1030 gfx1100、gfx1101、gfx1102

update list on this repo

list(FILTER AMDGPU_TARGETS INCLUDE REGEX "^gfx(803|900(:xnack-)|902|90c(:xnack-)|1010(:xnack-)|1011|1012(:xnack-)|103[0-6]|110[0-3]|1150)$")

meaning ：gfx803 ；gfx900:xnack-；gfx902；gfx906:xnack-；gfx90c:xnack-；gfx1010:xnack-； gfx1011；gfx1012:xnack-；gfx1030 to gfx1036； gfx1100 to gfx1103； 1150：gfx1150；

• Ensure you have a compatible Hip SDK installed. Install HIP SDK 6.1.2/HIP SDK 6.2.4 from hip-sdk.

Official AMD ROCm Supported GPUs: You can find a complete list of officially supported GPUs here: rocm.docs.amd.com

For supported GPUs, please use the official ROCm version from AMD also the Ollama Official. You don't need to build anything unless you want to learn how. (Examples:gfx900 gfx940 gfx941 gfx942 gfx1010 gfx1012 gfx1030 gfx1100 gfx1101 gfx1102)

If your GPU is not on the list:

• Try pre-built ROCm libraries:
  • Some pre-built rocblas libraries are available for specific GPUs at:

ROCmLibs for HIP SDK 6.1.2

ROCmLibs for HIP SDK 6.2.4

• Choose the appropriate library based on your GPU model (e.g., gfx902, gfx1031).

Building ROCm libraries:

• You may need to build rocblas yourself if a pre-built version is not available for your GPU.
• Refer to the Rocm documentation for detailed building instructions. learn how to build rocblas

Installing and Replacing Libraries:

1. Install HIP SDK: This will create a C:\Program Files\AMD\ROCm\6.1\bin folder.
2. Place rocblas.dll in the C:\Program Files\AMD\ROCm\6.1\bin folder. Replace any existing rocblas.dll.
3. Replace library within rocblas\library: This ensures your code uses the correct library files.
4. Add bin in the PATH: eg , C:\Program Files\AMD\ROCm\6.1\bin\ to Path( systeme envrimental , PATH , most people forgot this)

(change the path to 6.2 if you want to work on HIP 6.2 )

Community Support:

• If you encounter issues or need assistance in rocblas and library, please leave a message on the project's issues page for community support.

cmake -B build -G Ninja -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++

Then run

cmake --build build --config Release

or

ninja -C build

• Install Inno Setup: To build an installer for Ollama, you'll need to install Inno Setup: https://jrsoftware.org/isinfo.php.
• Run PowerShell Script: In the top directory of the Ollama repository, run this PowerShell script:

  powershell -ExecutionPolicy Bypass -File .\scripts\build_windows.ps1
  

• The Installer: After the build is complete, you'll find the OllamaSetup.exe installer in the dist folder. This will work exactly like the official release.
• If The Installer Build Broken in recent update: OllamaSetup.exe installer in the dist folder has not package all the build libs in build\lib\ollama and rocmlibs. if that's a necessary steps for you . simply manuly copy it in the Ollama program folder after installed the OllamaSetup.exe (ie,C:\Users\UsrName\AppData\Local\Programs\Ollama\lib\ollama\.. and rocm as show in demo release ollama-windows-amd64.7z on v0.5.9 or ollama official ) .

• Test Drive Ollama: If you want to test before building from source, download and install a demo release from ollama-for-amd/releases.

1. Install the Base Ollama Setup: Start by running OllamaSetup.exe. This sets up the basic framework for Ollama.(optional,If you skip these steps, make sure to remove any existing Ollama clients to avoid version conflicts. remove the ollama program directory or delete it not uninstall . As the uninstall will also remove ollama download models)

2. (optional)Unzip and Replace Libraries( update this steps no longer necessary if you had do the first steps unless you do not have rocm directory inside your ollama directory: Download ollama-windows-amd64.7z and unzip it. Then, replace the files within C:\Users\usrname\AppData\Local\Programs\Ollama\lib\ollama with those from the extracted archive.

3. Match ROCm to Your GPU: Download the appropriate ROCm libraries for your GPU architecture from ROCmlibs for 6.1.2. or ROCmLibs for HIP SDK 5.7 (based on each release package)

   • Important: Ensure the rocblas.dll and rocblas/library folder you choose match your GPU architecture.
   • Replace the existing files in your Ollama program folder (ie,C:\Users\UsrName\AppData\Local\Programs\Ollama\lib\ollama\rocm) with these downloaded libraries .

4. Start Serving: Once everything is installed, run ollama run llama3.1 in CMD OR Powershell to start your model or Skip step1 , navigate to your ollama-windows-amd64 program directory and replace with your target rocmlibs and run ./ollama serve.

This repository serves as an example, and won't be regularly updated.

Try pre-built ROCm libraries:

• Some pre-built rocblas libraries are available for specific GPUs at:

ROCmLibs for HIP SDK 6.1.2

ROCmLibs for HIP SDK 5.7

ROCmLibs for HIP SDK 6.2.4

• Choose the appropriate library based on your GPU model (e.g., gfx902, gfx1032).

HIP SDK 6.2.4 Notes: The ollama build with hip 6.1 should also works with rocmlibs for 6.2.4.

Multi gpu Notes: if your has multi gpu and the arches not on the support lists . please consider build your rocmlibs and olloma as the guide in development-guide .

We aim to simplify this by breaking it into two manageable steps.

For an even easier solution, consider using ByronLeeeee's Ollama-For-AMD-Installer. It streamlines the update and replacement of libraries with a simple click.

gfx803 hip memory error fixed by this rocm 5.7 libsrocm.gfx803.optic.test.version.7z for rocm5.7 in rocm 5.7 ,received feedback for gfx900 series ( gfx803;gfx900:xnack,gfx90c:xnack-...) can not detacted in Hip SDK 6.1.2 and forward . please consider build ollama with rocm5.7 yourself if there is no ollama-windows-amd64-rocm5.7.7z package available at release page .the ROCmlibs for 5.7 available at HIP SDK 5.7

• Start by trying the pre-built rocblas and library files for your GPU. *If you encounter issues, try setting the environment variable HSA_OVERRIDE_GFX_VERSION=9.0.12（for gfx90c），HSA_OVERRIDE_GFX_VERSION=10.1.2（ gfx1012） . Refer to the Windows documentation for guidance on setting environment variables: https://www.computerhope.com/issues/ch000549.htm.

• Running Your Models: After setting up ROCm and Ollama, run your models in a separate shell using the command: ./ollama run [model name], replacing "[model name]" with the actual name of your model.

and start the server in C:\Users\usrname\AppData\Local\Programs\Ollama\:

./ollama serve

Finally, in a separate shell, run a model(eg):

./ollama run llama3

• Update Carefully: If you are using a demo release, DO NOT click the "Update" button if Ollama prompts you. Download updates manually from this release page: ollama-for-amd/releases

• or update via Ollama-For-AMD-Installer created by ByronLeeeee, a simply click will do the update and replace libs process .

error 0xc0000005

1. make sure get the latest drivers.and replace the files match your gpu.
2. some proven solutions as shows in 8#,14#,18#,by mannuly move the all the .dll files in Ollama\lib\ollama and rocblas folder into Ollama\lib\ollama\runners\rocm_v6.1.

No GPU detected

evel=INFO source=gpu.go:386 msg="no compatible GPUs were discovered

install HIPSDK , add hipinfo in path , ie, rocm/bin in the path,refer to hipinfo#, that's usually happens in ROCm5.7 package or Release.

Hide Nvida GPU detected

CUDA_VISIBLE_DEVICES=-1,0,1,2... can be get info by run nvidia-smi -L

Hide AMD GPU detected

HIP_VISIBLE_DEVICES=-1,0,1,2... can be get info by run hipinfo

Multiple AMD GPU OVERRIDE Have mulitiple OVERRIDE_GFX_VERSION can add extra tag 0,1 behidn version

HSA_OVERRIDE_GFX_VERSION_0=10.3.0 HSA_OVERRIDE_GFX_VERSION_1=11.0.0

For users who prefer a self-build approach or use ollama official release on unsupported arches, setting environment variables like HSA_OVERRIDE_GFX_VERSION=9.0.0 or 9.0.6, 10.1.0, 10.3.0, 11.0.0 can enable GPU support on Linux systems. Configuring Environment Variables

Setting on Linux For Linux, set variables using systemd: 1, Edit your systemd service:

sudo systemctl edit ollama.service

2, Add the following line under the [Service]

Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0"

Have mulitiple GPUs ,can add extra tag 0,1 behidn version

HSA_OVERRIDE_GFX_VERSION_0=10.3.0 HSA_OVERRIDE_GFX_VERSION_1=11.0.0

3, Reload the service and restart:

$ sudo systemctl daemon-reload
$ sudo systemctl restart ollama.service

However, some users may encounter issues such as:

• "amdgpu too old gfx%d%x%x"
• "unsupported Radeon iGPU detected skipping"
• "unable to verify rocm library: %w", err)

These errors due to incompatibility between the installed AMDGPU drivers and rocm supports issue. To potentially resolve these issues, consider modifying the discover/amd_linux.go file within your ollama directory if the HSA_OVERRIDE_GFX_VERSIONnot works for you.

Caution: Editing this file can have unintended consequences. It's recommended to back up the original file before making any changes.

Specifically, focus on lines 295 to 320 and either delete or edit the relevant content. The exact modifications required will depend on your specific hardware and driver version. Refer to online forums and documentation for guidance on tailoring these edits for your situation.