Clarification: Does PARD require an in-family small AR draft model? How to train a draft model when no such model exists? #8

Hi PARD team,

Thanks for the great work. I am studying PARD / PARD-2 and would like to clarify the assumptions behind draft model training.

From my reading, PARD adapts small autoregressive draft models into parallel draft models, and one of its advantages is that a single draft model can be reused across a family of target models. This makes me wonder whether PARD assumes the existence of an in-family small AR model as the starting point.

My questions are:

1. Does PARD / PARD-2 require an existing small AR model from the same model family as the target model, e.g. a small Qwen model for Qwen targets or a small Llama model for Llama targets?

2. If the target model does not have an official or suitable in-family small model, what is the recommended way to train a PARD draft model?

   Possible options I am considering:

   • train a small AR draft model from scratch with the target tokenizer;
   • initialize from a generic small LM with a compatible tokenizer;
   • initialize a shallow target-like drafter from a subset of target layers;
   • use target logits / hidden states for knowledge distillation before or during PARD training.

3. For PARD-2 target-dependent mode, can target hidden feature injection and knowledge distillation compensate for the lack of an in-family small AR backbone, or is a pretrained small AR draft model still expected?

4. Are there plans to release training scripts, configs, or guidance for building PARD draft models for targets without a smaller sibling model?

It would be very helpful if the documentation could clarify the assumptions about tokenizer sharing, draft model initialization, and whether an in-family small model is required.

Thanks!