Thanks @AndreasKunar ! I've adjusted to use the clang compiler so we can target armv8.7-a

I've switched to draft until I can test this on more permutations to make sure this new binary is generally compatible with arm64 systems the prior build worked on.

@dhiltgen I tested your current changes on Windows for ARM on a Surface 11 Pro base model, and they require MinGW, gcc, which are not available for arm. For ARM we use clang(LLVM) - best installed via Visual Studio Community Edition 2022 (note clang on Windows requires the MSVC backend), which also can install CMake, Git, ... you then only need to additionally install Go via winget install --id=GoLang.Go -e. The same toolset can also be used for Windows x64 - but there the MinGW alternative is available.

My build (in a Developer Powershell for VS 2022):

$env:CGO_ENABLED="1"
go generate ./...        -> generates error
go build . 

Build- Error (sorry for the German Windows messages):
Checking for MinGW...
get-command : Die Benennung "gcc" wurde nicht als Name eines Cmdlet, einer Funktion, einer Skriptdatei oder eines
ausführbaren Programms erkannt. Überprüfen Sie die Schreibweise des Namens, oder ob der Pfad korrekt ist (sofern
enthalten), und wiederholen Sie den Vorgang.
In C:\Users\andre\Projects\ollama\llm\generate\gen_windows.ps1:189 Zeichen:9

•     get-command gcc
  

•     ~~~~~~~~~~~~~~~
  

  • CategoryInfo : ObjectNotFound: (gcc:String) [Get-Command], CommandNotFoundException
  • FullyQualifiedErrorId : CommandNotFoundException,Microsoft.PowerShell.Commands.GetCommandCommand

Also please be aware that armv8.7-a might not work correctly for Macs, running Windows on ARM in Parallels. Currently, there probably are more Mac/Parallels/Windows on ARM users than the new Copilot+PCs (hopefully changing in the future). I will try and test both, since I'm mainly using Macs, only have one Snapdragon X/Windows machine.

Note: There also is a problem with current ollama and the new, fast Q4_0_4_8 quantization acceleration of modern ARM CPUs. ollama PR #6126 tries to address this and I'm trying to test/verify that it's running correctly.

Windows arm64 tools do seem to be a bit challenging, but I found it via msys2 via the mingw-w64-clang-aarch64-gcc-compat package

get-command gcc

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Application     gcc.exe                                            0.0.0.0    C:\msys64\clangarm64\bin\gcc.exe     

I haven't decided, but I'm contemplating moving this to a cross-compilation model instead of trying to build native though.

Windows arm64 tools do seem to be a bit challenging, but I found it via msys2 via the mingw-w64-clang-aarch64-gcc-compat package

get-command gcc

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Application     gcc.exe                                            0.0.0.0    C:\msys64\clangarm64\bin\gcc.exe     

I haven't decided, but I'm contemplating moving this to a cross-compilation model instead of trying to build native though.

Since you need MSVC (clang on Windows normally depends on the MSVC backend), I still suggest best to directly install CMake via the Visual Studio 2022 Community Edition/... Installer. You can also directly install CMake and Git with it in just one installation step. Just search for these components (Note: Clang is 2 components, the Clang compiler and the MSBuild support vor LLVM (clang-cl) toolset). And this works for native and cross-compile, you could then use different Developer powersehell/command-prompt settings.

I copied your changed files into the current ollama version. When trying to go generate ./... with Visual Studio 2022 installed clang (LLVM and not MinGW), it still generates the following error:

Checking for MinGW...
get-command : The term 'gcc' is not recognized as the name of a cmdlet, function, script file, or operable program.
Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At C:\Users\andi\Projects\ollama\llm\generate\gen_windows.ps1:189 char:9
+         get-command gcc
+         ~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (gcc:String) [Get-Command], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException,Microsoft.PowerShell.Commands.GetCommandCommand

llm\generate\generate_windows.go:3: running "powershell": exit status 1

I'm currently on Windows 11 for ARM, running on a M2 MacBook Air in Parallels because I'm moving from my Surface Pro to a Surface Laptop and have not yet got my new machine.

@AndreasKunar thanks for giving it a shot. I'm still tinkering to find the best combination, but you might want to try commenting out that check and try setting something like $env:CC="clang"; $env:CXX="clang++" to see if Go can pick up clang instead of gcc.

@AndreasKunar thanks for giving it a shot. I'm still tinkering to find the best combination, but you might want to try commenting out that check and try setting something like $env:CC="clang"; $env:CXX="clang++" to see if Go can pick up clang instead of gcc.

Sorry, does not work for me. Tested in powershell developer prompt and via set-commands in developer command prompt.

In line 189 in lllm\generate\gen_windows.ps1 it checks for gcc with get-command gcc and fails. It would also fail with the following line get-command mingw32-make. mingw32 and gcc are not (yet?) available for WoA, its clang LLVM.

@AndreasKunar if you install the gcc-compat MSYS2 package then you don't have to do anything (gcc becomes an alias to clang, and so on). Did you try building with these instructions? #5268 (See https://github.com/hmartinez82/ollama/blob/win_arm64_docs/docs/development.md#windows-on-arm-arm64)

I even get the installer working (the installer itself will still be non ARM64, but all the binaries inside it will be).

@AndreasKunar I'm waiting for Go to be able to detect the presence of IMM8 support by the CPU at runtime (this should be in the next release of Go https://go-review.googlesource.com/c/sys/+/595678). Once that's done, I want to create a new PR here that will have two builds of the arm64 llama.cpp binary (one armv8.2 and one armv8.7) that can be chosen at runtime just like cpu, cpu_avx, and cpu_avx2 are in x64 builds.

@AndreasKunar if you install the gcc-compat MSYS2 package then you don't have to do anything (gcc becomes an alias to clang, and so on). Did you try building with these instructions? #5268 (See https://github.com/hmartinez82/ollama/blob/win_arm64_docs/docs/development.md#windows-on-arm-arm64)

I even get the installer working (the installer itself will still be non ARM64, but all the binaries inside it will be).

Yes, I build natively on Windows on ARM (WoA), and with nearly all arm64 toolsets. But no, like with the recommendations for building llama.cpp (which I understand ollama uses/compiles), I use Visuals Studio 2022 and its included clang (LLVM), CMake installation as a fully released / supported compiler for WoA (see the build recommendations for llama.cpp on WoA), not the preliminary MSYS2 arm64 support.

So my bad, sorry! I will try and build it with these instructions, but see if I still can use my llama.cpp tooling (e.g. adding aliases to my installed clang) with just adding go.

@AndreasKunar I'm waiting for Go to be able to detect the presence of IMM8 support by the CPU at runtime (this should be in the next release of Go https://go-review.googlesource.com/c/sys/+/595678). Once that's done, I want to create a new PR here that will have two builds of the arm64 llama.cpp binary (one armv8.2 and one armv8.7) that can be chosen at runtime just like cpu, cpu_avx, and cpu_avx2 are in x64 builds.

Cool, thanks a lot!!! For Snapdragon X, llama.cpp's new Q4_0_4_4/Q4_0_4_8 quantization provides an up to 2-3x acceleration (for the compute-bound prompt-processing tasks, token-generation is mostly memory-bandwidth constrained).

@AndreasKunar #6379

No runtime dynamic detection yet of course. But that's the blunt of the changes.

@dhiltgen Can you try these steps ? https://github.com/hmartinez82/ollama/blob/win_arm64_docs/docs/development.md#windows-on-arm-arm64
With those steps, I can build an ARM64 installer without any changes to build_windows.ps1

I just realized that I repeated a bunch of things you were already doing here @dhiltgen :(

@dhiltgen I'm going ahead and close #6379 since your PR is much more comprehensive (it includes installer, etc).