Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature Request: Support Jina V3 arch #9585

Open
4 tasks done
abhishekbhakat opened this issue Sep 21, 2024 · 5 comments
Open
4 tasks done

Feature Request: Support Jina V3 arch #9585

abhishekbhakat opened this issue Sep 21, 2024 · 5 comments
Labels
enhancement New feature or request stale

Comments

@abhishekbhakat
Copy link

Prerequisites

  • I am running the latest code. Mention the version if possible as well.
  • I carefully followed the README.md.
  • I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
  • I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

I was trying to convert https://huggingface.co/jinaai/jina-embeddings-v3 to GGUF and it seems like it doesn't support it yet:

INFO:hf-to-gguf:Loading model: jina-embeddings-v3
INFO:gguf.gguf_writer:gguf: This GGUF file is for Little Endian only
INFO:hf-to-gguf:Exporting model...
INFO:hf-to-gguf:gguf: loading model part 'model.safetensors'
Traceback (most recent call last):
  File "/Volumes/AI/llama.cpp/convert_hf_to_gguf.py", line 4330, in <module>
    main()
  File "/Volumes/AI/llama.cpp/convert_hf_to_gguf.py", line 4324, in main
    model_instance.write()
  File "/Volumes/AI/llama.cpp/convert_hf_to_gguf.py", line 425, in write
    self.prepare_tensors()
  File "/Volumes/AI/llama.cpp/convert_hf_to_gguf.py", line 294, in prepare_tensors
    for new_name, data in ((n, d.squeeze().numpy()) for n, d in self.modify_tensors(data_torch, name, bid)):
  File "/Volumes/AI/llama.cpp/convert_hf_to_gguf.py", line 2704, in modify_tensors
    return super().modify_tensors(data_torch, name, bid)
  File "/Volumes/AI/llama.cpp/convert_hf_to_gguf.py", line 2568, in modify_tensors
    return [(self.map_tensor_name(name), data_torch)]
  File "/Volumes/AI/llama.cpp/convert_hf_to_gguf.py", line 214, in map_tensor_name
    raise ValueError(f"Can not map tensor {name!r}")
ValueError: Can not map tensor 'roberta.emb_ln.bias'

Motivation

Jina V3 has been one of the top performing embedding model. And it might be expected to see more models in the future.

Possible Implementation

No response
@abhishekbhakat abhishekbhakat added the enhancement New feature or request label Sep 21, 2024
@abhishekbhakat abhishekbhakat changed the title Feature Request: Support roberta.emb_ln.bias Feature Request: Support Jina V3 arch Sep 21, 2024
@abhishekbhakat
Copy link
Author

#6826 added the support for Jina V2. Perhaps something similar is needed.

@enthermo
Copy link

Jina V2 is based off of 'JinaBERT', V3 is based off of Jina-XLM-RoBERTa

@abhishekbhakat
Copy link
Author

Okay, so the script already supports XLMRobertaModel. But the Flash implementation has different layer names or structures compared to the standard XLMRobertaModel, I believe.

@ggerganov
Copy link
Owner

Maybe a fix similar to the one in #9510 would be needed:

    def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:
        # if name starts with "roberta.", remove the prefix
        # e.g. https://huggingface.co/BAAI/bge-reranker-v2-m3/tree/main
        if name.startswith("roberta."):
            name = name[8:]

@abhishekbhakat
Copy link
Author

That might mitigate the immediate error, but later it will fall into another.

For example, XLMRobertaModel looks for sentencepiece.bpe.model in the hf repo. But this repo only has a tokenizer.json.

I made a very naive attempt of a whole lot of re-writing set_vocab() but in the end got stuck in GGUFWriter getting a boolean in the array expecting <GGUFValueType.STRING: 8> type 🥲.

@github-actions github-actions bot added the stale label Oct 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ggerganov @abhishekbhakat @enthermo