Replies: 1 comment 1 reply
-
|
I get your point, but in many situations we simply can't change the fact that the model isn't available in safetensors. A case in point is the EuroLLM models, which aren't stored in safetensors. We could write to them and mention it, and we could even open a PR with the change, but it sometimes takes weeks or months for the model authors to respond to such requests, and sometimes never do. So it's a compromise. In the newer versions I include a warning when you're benchmarking a non-safetensors model, and urging the user to convert it to safetensors if they are the model owner. But I think that may be as good as we can get? |
Beta Was this translation helpful? Give feedback.
-
|
In my case, I am just concerned that I will forget to check if there is a safetensors version of the model when I run benchmarks. We have an internal policy in Telenor of safetensors-only, and I (of course) dont want to violate that. What if the user was prompted to continue when using CLI, rather than just being warned? |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
🚀 The feature, motivation and pitch
Saw the most recent update (c1d3266), and realized that models arent necessarily required to be in .safetensors format.
My pitch is the following:
To avoid the various safety risks involved with other formats, I think ScandEval should make it mandatory that the model must be in safetensors format. It is very low effort for the model developer, and I think this is a standard we should adopt as a community.
Alternatives
No response
Additional context
No response
Beta Was this translation helpful? Give feedback.
All reactions