[template] Support HunyuanMT1.5-1.8B and HunyuanMT1.5-7B templates#7351
[template] Support HunyuanMT1.5-1.8B and HunyuanMT1.5-7B templates#7351rinne1998 wants to merge 1 commit intomodelscope:mainfrom
Conversation
- Add hunyuan_mt1_5_1_8b and hunyuan_mt1_5_7b template types to LLMTemplateType - Move hunyuan_moe and hunyuan template definitions from llm.py to tencent.py
Summary of ChangesHello @rinne1998, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request enhances the system's compatibility with new large language models by integrating specific chat templates for HunyuanMT1.5-1.8B and HunyuanMT1.5-7B. This ensures that these models can process and generate text accurately according to their unique formatting requirements. Concurrently, the change improves the codebase's structure by relocating existing Hunyuan-related template definitions to a more appropriate module, streamlining maintenance and future expansions. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
| class HunYuanMT15_18B_Template(Template): | ||
| def format_prompt(self, messages, add_generation_prompt: bool = False, **kwargs): | ||
| s = "<｜hy_begin▁of▁sentence｜>" | ||
| for m in messages: | ||
| if m["role"] == "user": | ||
| s += "<｜hy_User｜>" + m["content"] | ||
| elif m["role"] == "assistant": | ||
| s += "<｜hy_Assistant｜>" + m["content"] + "<｜hy_place▁holder▁no▁2｜>" | ||
| if add_generation_prompt: | ||
| s += "<｜hy_Assistant｜>" | ||
| else: | ||
| s += "<｜hy_place▁holder▁no▁8｜>" | ||
| return s | ||
|
|
||
|
|
||
| register_template( | ||
| TemplateMeta( | ||
| LLMTemplateType.hunyuan_mt1_5_1_8b, | ||
| template_cls=HunYuanMT15_18B_Template, | ||
| prefix=["<｜hy_begin▁of▁sentence｜>"], | ||
| prompt=["<｜hy_User｜>{{QUERY}}<｜hy_Assistant｜>"], | ||
| chat_sep=[""], | ||
| suffix=["<｜hy_place▁holder▁no▁2｜><｜hy_place▁holder▁no▁8｜>"], | ||
| ) | ||
| ) |
There was a problem hiding this comment.
HunYuanMT15_18B_Template 类及其 format_prompt 方法似乎未被 Swift 模板框架使用，该框架依赖 TemplateMeta 进行编码。同时，当前为 hunyuan_mt1_5_1_8b 提供的 TemplateMeta 在处理多轮对话时也是不正确的，因为它会遗漏对话轮次之间的分隔符。
为了修复这个问题，应该移除未使用的类，并修正 TemplateMeta 以符合模型的官方 jinja 模板。具体来说，需要正确设置 chat_sep，并且 suffix 也需要调整以正确处理对话结束。
register_template(
TemplateMeta(
LLMTemplateType.hunyuan_mt1_5_1_8b,
prefix=["<｜hy_begin▁of▁sentence｜>"],
prompt=["<｜hy_User｜>{{QUERY}}<｜hy_Assistant｜>"],
chat_sep=["<｜hy_place▁holder▁no▁2｜>"],
suffix=["<｜hy_place▁holder▁no▁2｜>", "<｜hy_place▁holder▁no▁8｜>"],
)
)| register_template( | ||
| TemplateMeta( | ||
| LLMTemplateType.hunyuan_mt1_5_7b, | ||
| prefix=["<|startoftext|>"], | ||
| prompt=["{{QUERY}}<|extra_0|>"], | ||
| chat_sep=[""], | ||
| suffix=["<|eos|>"], | ||
| ) | ||
| ) |
There was a problem hiding this comment.
hunyuan_mt1_5_7b 模板的 chat_sep 当前为空字符串，这对于多轮对话是不正确的。根据模型的官方 jinja 模板，每个 assistant 的回复之后都应该有一个 <|eos|> 令牌作为分隔。这应该在 chat_sep 和 suffix 中都得到体现。
register_template(
TemplateMeta(
LLMTemplateType.hunyuan_mt1_5_7b,
prefix=["<|startoftext|>"],
prompt=["{{QUERY}}<|extra_0|>"],
chat_sep=["<|eos|>"],
suffix=["<|eos|>"],
)
)1 participant
参考对应模型的chat template编写：
https://huggingface.co/tencent/HY-MT1.5-1.8B/blob/main/chat_template.jinja
https://huggingface.co/tencent/HY-MT1.5-7B/blob/main/chat_template.jinja
如果HY-MT1.5-7B使用默认的hunyuan的模板，会生成不正常的结束符/重复片段，训练时需指定
--template hunyuan_mt1_5_7b，HY-MT1.5-1.8B需指定--template hunyuan_mt1_5_1_8。