Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support configuring whether to load the entire model into memory or use mmap #4

Closed
xISSAx opened this issue Apr 12, 2023 · 2 comments
Closed

Support configuring whether to load the entire model into memory or use mmap #4

xISSAx opened this issue Apr 12, 2023 · 2 comments
Assignees
@alexrozanski
Labels
enhancement New feature or request
Milestone
v1.2

Comments

@xISSAx
Copy link

xISSAx commented Apr 12, 2023

Greetings,
Love the application and UX!

I noticed Llama cpp running on my M1 was flushing the memory during and after each generation causing slower-than-expected outputs.
This can be fixed by passing "-mlock" argument, which massively boosts Mac M1 performance by locking the model into the memory.

However, currently, LlamaChat has a similar issue, and I believe it can be fixed by passing a simple '-mlock' argument. In fact, I suggest leaving it ON by default for a seamless beginner's experience for M1s.

Moreover, please also consider an advanced feature to allow users to change the parameters.
@alexrozanski alexrozanski added the enhancement New feature or request label Apr 12, 2023
@alexrozanski
Copy link
Owner

Thanks @xISSAx. You're right, LlamaChat sets the mlock parameter to false always, since this was touted as a big performance improvement over the previous versions (which for large models I think is true)?

I need to do some more investigation into this, but I was definitely thinking of adding a switch for this. Perhaps you're right, maybe this should be enabled by default for a good FTUE, but configurable if people need it.

@alexrozanski alexrozanski added this to the v1.2 milestone Apr 17, 2023
@alexrozanski alexrozanski changed the title Mac M1 Memory Flush - Llama cpp Support configuring whether to load the entire model into memory or use mmap Apr 17, 2023
@alexrozanski alexrozanski mentioned this issue Apr 17, 2023
Open
@alexrozanski alexrozanski self-assigned this Apr 17, 2023
@alexrozanski
Copy link
Owner

Added in v1.2.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alexrozanski alexrozanski

Labels
enhancement New feature or request
Projects
LlamaChat
Status: Done
Milestone
v1.2
Development

No branches or pull requests
2 participants
@alexrozanski @xISSAx