Skip to content

usage.md

Latest commit

 

History

History
335 lines (260 loc) · 13.6 KB

usage.md

File metadata and controls

335 lines (260 loc) · 13.6 KB
Raw

OTE Usage / 使用指南

English | 中文

中文

1.2 YAML 配置结构

一次运行由一个 run.yaml 串联四个角色/工具 YAML：

# run.yaml
task: deep_research            # 选择 task 插件
exp_name: my-run
wandb_project_name: rl-rag
models:
  solver: ${MODEL_PATH}        # 支持 ${ENV} 展开
  proposer: ${QUERY_MODEL_PATH}
proposer: proposer.yaml        # 可以是相对路径，也可以内联成 dict
solver: solver.yaml
judge: judge.yaml
tools: tools.yaml
# 可选：output_dir（默认按 output_hpo_<rl_method>_<stage>_<qstage>_<hybrid> 派生）
# 可选：training: { learning_rate: 1e-6, seed: 7, ... }  # 覆盖通用超参
# proposer.yaml —— 出题模型如何进化、是否用工具
strategy: hote     # none | hote | "包.模块:函数"（自定义）
use_tools: true    # true=接入数据源工具(real_tool)，false=纯参数化(no_tool)
# rl_method: gspo  # 可选：proposer 用自己的 RL 算法；不填则跟随 solver 的 rl_method
# solver.yaml —— 作答模型（始终进化，即被训练的模型）
strategy: hote
use_tools: true
rl_method: grpo    # grpo | gspo | reinforce_pp | rloo | seeupo | rltr
hybrid: true       # 基础 hybrid（每步一半 query 用 no_tool、一半用 use_tools 对应模式）
# judge.yaml —— 评分/rubric 如何进化
strategy: hote     # hote=实时生成新 rubric + 演化 rubric buffer；none=仅静态 rubric
use_tools: false   # 预留配置位（当前 deep_research 下 judge 不实际调用工具）
# tools.yaml —— 与数据源交互的工具
backend: mcp                                           # 映射到 --tools
mcp_tool_names: [google_search, snippet_search, browse_webpage]
mcp_parser_name: v20250824
max_tool_calls: 10
# mcp_host: 172.16.10.157   # 可选：你部署好的 MCP server 地址
# mcp_port: 8200            # 不填则用 MCP_TRANSPORT_HOST / MCP_TRANSPORT_PORT 环境变量

内置进化方式：

角色 strategy 含义
proposer none 不进化：不生成合成问题、不训练出题模型，只用原始数据集
proposer hote 独立出题模型在线生成合成问题并被训练；use_tools 控制是否用工具，rl_method 可独立指定
solver hote 作答模型被训练，支持全部 RL 算法 + 基础 hybrid；use_tools 控制工具
judge none 不进化：仅用静态/预定义 rubric 打分
judge hote 实时生成新 rubric + 演化 rubric buffer

real_tool 与 no_tool 两套配置的唯一区别就是 proposer/solver 的 use_tools （true→real_tool 阶段，false→no_tool 阶段），其余完全相同。

1.3 自定义进化方式（写 Python）

进化策略是一个函数 fn(ctx: BuildContext, role: RoleConfig) -> None， 通过修改 ctx.args（flag→值 的字典）来设置训练引擎参数：

# my_pkg/my_strategies.py
from open_instruct.evolve_framework import register_evolution_strategy

def proposer_generate_only(ctx, role):
    """出题但不训练出题模型。"""
    ctx.args["apply_evolutionary_queries"] = "true"
    ctx.args["train_query_generator"] = "false"
    stage = "real_tool" if role.use_tools else "no_tool"
    ctx.args["query_gen_training_stage"] = stage
    ctx.args["query_model_name_or_path"] = ctx.run.models["proposer"]
    ctx.args["query_num_learners_per_node"] = ctx.resources.query_num_learners_per_node
    ctx.args["query_vllm_num_engines"] = ctx.resources.query_vllm_num_engines
    ctx.meta["query_gen_training_stage"] = stage   # 供 output_dir 命名

register_evolution_strategy("proposer", "generate_only", proposer_generate_only)

BuildContext 可用字段：ctx.run（RunConfig）、ctx.task（当前 Task）、 ctx.args（输出参数字典）、ctx.resources（GPU/引擎数）、ctx.meta（跨策略传值， 其中 training_stage/query_gen_training_stage/rl_method/hybrid_mode 用于派生 output_dir）。 role.use_tools、role.opt("key", default) 读取角色配置。

在 YAML 中引用（推荐 包.模块:函数 形式，会自动 import，无需手动注册）：

# proposer.yaml
strategy: my_pkg.my_strategies:proposer_generate_only
use_tools: true

1.4 增加新任务（适配到其它任务）

新任务 = 写一个 Task 子类，提供三组参数贡献，然后注册。无需改框架或训练引擎。

# open_instruct/tasks/my_task/task.py
from open_instruct.evolve_framework.config import FLAG, ArgDict, ToolConfig
from open_instruct.tasks.base import Task

class MyTask(Task):
    name = "my_task"

    def base_args(self, run) -> ArgDict:
        # 任务常量：数据集、系统提示、通用超参（不要设置角色旋钮/模型路径/资源/output_dir）
        return {
            "dataset_mixer_list": ["<dataset_path>", "1.0"],
            "dataset_mixer_list_splits": "train",
            "system_prompt_file": "<prompt.yaml>",
            "no_tool_system_prompt_file": "<no_tool_prompt.yaml>",
            "learning_rate": "5e-7", "beta": "0.001", "response_length": "16384",
            "apply_verifiable_reward": "true", "gradient_checkpointing": FLAG,
            # ... 其余通用超参
        }

    def tool_args(self, tools: ToolConfig) -> ArgDict:
        # 把 tools.yaml 翻译成引擎的工具参数
        return {"tools": tools.backend or "mcp", ...}

    def judge_args(self, evolve: bool, use_tools: bool) -> ArgDict:
        # judge:hote -> evolve=True；judge:none -> evolve=False（关闭 rubric 演化）
        return {"use_rubric_buffer": "true" if evolve else "false", ...}

注册（在 open_instruct/tasks/__init__.py 中）：

from .my_task.task import MyTask
register_task(MyTask())

之后在 run.yaml 里 task: my_task 即可。建议把该任务的 run/proposer/solver/judge/tools 默认 YAML 放在 open_instruct/tasks/my_task/configs/，参考 deep_research/configs/。

1.5 增加新数据源工具

工具分两层：

  1. 新增一个工具后端（backend）：在引擎的工具注册表登记工具类 open_instruct/tool_utils/tool_actor.py 的 register_tool(name, "包.模块:类")， 然后在 tools.yaml 用 backend: <name>，并在对应 task 的 tool_args 中输出该后端 需要的参数。

  2. 给 MCP 后端新增一个子工具（如新的检索/浏览工具）：在 open_instruct/search_utils/mcp_tools.py 的 MCP_TOOL_REGISTRY 中登记，然后在 tools.yaml 的 mcp_tool_names 中加入它的名字，例如：

    backend: mcp
mcp_tool_names: [google_search, snippet_search, browse_webpage, <your_tool>]

工具是否被各角色调用由各自的 use_tools 控制（true 才会接入数据源）。

English

1.2 YAML configuration

A run is described by one run.yaml that wires together four role/tool YAMLs:

# run.yaml
task: deep_research            # select the task plugin
exp_name: my-run
wandb_project_name: rl-rag
models:
  solver: ${MODEL_PATH}        # ${ENV} expansion supported
  proposer: ${QUERY_MODEL_PATH}
proposer: proposer.yaml        # a relative path, or an inline dict
solver: solver.yaml
judge: judge.yaml
tools: tools.yaml
# optional: output_dir (defaults to output_hpo_<rl_method>_<stage>_<qstage>_<hybrid>)
# optional: training: { learning_rate: 1e-6, seed: 7, ... }  # override generic hyperparams
# proposer.yaml — how the question model evolves and whether it uses tools
strategy: hote     # none | hote | "package.module:function" (custom)
use_tools: true    # true = use data-source tools (real_tool), false = parametric only (no_tool)
# rl_method: gspo  # optional: give the proposer its own RL algorithm; defaults to the solver's
# solver.yaml — the answering model (always evolves; this is the model being trained)
strategy: hote
use_tools: true
rl_method: grpo    # grpo | gspo | reinforce_pp | rloo | seeupo | rltr
hybrid: true       # basic hybrid (each step: half the queries no_tool, half use_tools mode)
# judge.yaml — how scoring / rubrics evolve
strategy: hote     # hote = generate new rubrics live + evolving rubric buffer; none = static rubrics only
use_tools: false   # reserved config hook (judge does not call tools in deep_research today)
# tools.yaml — tools that interact with the data source
backend: mcp                                           # maps to --tools
mcp_tool_names: [google_search, snippet_search, browse_webpage]
mcp_parser_name: v20250824
max_tool_calls: 10
# mcp_host: 172.16.10.157   # optional: address of your deployed MCP server
# mcp_port: 8200            # if omitted, falls back to MCP_TRANSPORT_HOST / MCP_TRANSPORT_PORT env vars

Built-in evolution strategies:

Role strategy Meaning
proposer none no evolution: no synthetic questions, no query-model training, original dataset only
proposer hote a separate query model generates synthetic questions online and is trained; use_tools controls tools, rl_method can be set independently
solver hote the answering model is trained; supports all RL algorithms + basic hybrid; use_tools controls tools
judge none no evolution: score against static / predefined rubrics only
judge hote generate new rubrics live + evolving rubric buffer

The only difference between the real_tool and no_tool configs is the use_tools of proposer/solver (true → real_tool stage, false → no_tool stage); everything else is identical.

1.3 Custom evolution methods (write Python)

An evolution strategy is a function fn(ctx: BuildContext, role: RoleConfig) -> None that sets training-engine flags by mutating ctx.args (a flag→value dict):

# my_pkg/my_strategies.py
from open_instruct.evolve_framework import register_evolution_strategy

def proposer_generate_only(ctx, role):
    """Generate questions but do not train the query model."""
    ctx.args["apply_evolutionary_queries"] = "true"
    ctx.args["train_query_generator"] = "false"
    stage = "real_tool" if role.use_tools else "no_tool"
    ctx.args["query_gen_training_stage"] = stage
    ctx.args["query_model_name_or_path"] = ctx.run.models["proposer"]
    ctx.args["query_num_learners_per_node"] = ctx.resources.query_num_learners_per_node
    ctx.args["query_vllm_num_engines"] = ctx.resources.query_vllm_num_engines
    ctx.meta["query_gen_training_stage"] = stage   # used for output_dir naming

register_evolution_strategy("proposer", "generate_only", proposer_generate_only)

BuildContext fields: ctx.run (RunConfig), ctx.task (current Task), ctx.args (output flag dict), ctx.resources (GPU/engine counts), ctx.meta (cross-strategy values; training_stage/query_gen_training_stage/rl_method/hybrid_mode are used to derive output_dir). Read role config via role.use_tools and role.opt("key", default).

Reference it from YAML (the package.module:function form is auto-imported, no manual registration needed):

# proposer.yaml
strategy: my_pkg.my_strategies:proposer_generate_only
use_tools: true

1.4 Adding a new task (porting to other tasks)

A new task = a Task subclass that contributes three groups of parameters, then register it. No changes to the framework or training engine.

# open_instruct/tasks/my_task/task.py
from open_instruct.evolve_framework.config import FLAG, ArgDict, ToolConfig
from open_instruct.tasks.base import Task

class MyTask(Task):
    name = "my_task"

    def base_args(self, run) -> ArgDict:
        # task constants: dataset, system prompts, generic hyperparams
        # (do NOT set role knobs / model paths / resources / output_dir)
        return {
            "dataset_mixer_list": ["<dataset_path>", "1.0"],
            "dataset_mixer_list_splits": "train",
            "system_prompt_file": "<prompt.yaml>",
            "no_tool_system_prompt_file": "<no_tool_prompt.yaml>",
            "learning_rate": "5e-7", "beta": "0.001", "response_length": "16384",
            "apply_verifiable_reward": "true", "gradient_checkpointing": FLAG,
            # ... other generic hyperparams
        }

    def tool_args(self, tools: ToolConfig) -> ArgDict:
        # translate tools.yaml into the engine's tool flags
        return {"tools": tools.backend or "mcp", ...}

    def judge_args(self, evolve: bool, use_tools: bool) -> ArgDict:
        # judge:hote -> evolve=True; judge:none -> evolve=False (disable rubric evolution)
        return {"use_rubric_buffer": "true" if evolve else "false", ...}

Register it (in open_instruct/tasks/__init__.py):

from .my_task.task import MyTask
register_task(MyTask())

Then set task: my_task in run.yaml. Put the task's default run/proposer/solver/judge/tools YAMLs under open_instruct/tasks/my_task/configs/, mirroring deep_research/configs/.

1.5 Adding a new data-source tool

Tools work at two levels:

  1. Add a new tool backend: register the tool class in the engine's tool registry via register_tool(name, "package.module:Class") in open_instruct/tool_utils/tool_actor.py, then set backend: <name> in tools.yaml and emit the backend's flags from the task's tool_args.

  2. Add a sub-tool to the MCP backend (e.g. a new search/browse tool): register it in MCP_TOOL_REGISTRY in open_instruct/search_utils/mcp_tools.py, then add its name to mcp_tool_names in tools.yaml:

    backend: mcp
mcp_tool_names: [google_search, snippet_search, browse_webpage, <your_tool>]

Whether each role actually calls the tools is controlled by its own use_tools (only true connects to the data source).