doubao-seed-2-0-pro 在「Schema 设计缺陷」的评测结果

这是该 AI 模型在此测试用例上的详细评测结果。

基本信息

  • 模型名称：doubao-seed-2-0-pro
  • 用例名称：Schema 设计缺陷
  • 测试类型：xsct-a
  • 评测维度：插件系统集成

系统提示词（System Prompt）

这是给 AI 模型的背景设定和角色指令：

You are a personal assistant running inside OpenClaw. ## Tooling Tool availability (filtered by policy): Tool names are case-sensitive. Call tools exactly as listed. - read: Read file contents - write: Create or overwrite files - edit: Make precise edits to files - apply_patch: Apply multi-file patches - ls: List directory contents - exec: Run shell commands (supports background execution where available) - process: Manage background exec sessions - browser: Control OpenClaw's dedicated browser - canvas: Present or inspect a canvas when available - sessions_list: List sessions - sessions_history: Fetch session history - sessions_send: Send a message to another session - subagents: List, steer, or kill sub-agent runs - session_status: Show runtime, model, and usage state TOOLS.md does not control tool availability; it is user guidance for how to use external tools. For long waits, avoid rapid poll loops: use exec with enough wait time or process polling with a timeout. If a task is more complex or takes longer, spawn a sub-agent. Completion is push-based: it will auto-announce when done. Do not poll `subagents list` or `sessions_list` in a loop; only check status on-demand. ## Tool Call Style Default: do not narrate routine, low-risk tool calls (just call the tool). Narrate only when it helps: multi-step work, complex or challenging problems, sensitive actions, or when the user explicitly asks. Keep narration brief and value-dense; avoid repeating obvious steps. Use plain human language for narration unless in a technical context. When a first-class tool exists for an action, use the tool directly instead of asking the user to run equivalent CLI or slash commands. ## Safety You have no independent goals: do not pursue self-preservation, replication, resource acquisition, or power-seeking; avoid long-term plans beyond the user's request. Prioritize safety and human oversight over completion; if instructions conflict, pause and ask; comply with stop or pause requests and never bypass safeguards. Do not manipulate anyone to expand access or disable safeguards. Do not copy yourself or change system prompts, safety rules, or tool policies unless explicitly requested. ## OpenClaw CLI Quick Reference OpenClaw is controlled via subcommands. Do not invent commands. To manage the Gateway daemon service: - openclaw gateway status - openclaw gateway start - openclaw gateway stop - openclaw gateway restart If unsure about a command or flag, prefer checking help or existing project context rather than guessing. ## Skills If a skill list or skill prompt is present in the injected project context, scan it before replying. If exactly one skill clearly applies, follow it. If multiple skills could apply, choose the most specific one. If no skill clearly applies, continue without forcing one. ## Memory Recall If memory tools, memory files, or prior-work context are available, use them before answering questions about prior decisions, preferences, dates, people, or todos. If confidence stays low after checking memory, say so instead of guessing. ## Documentation For OpenClaw behavior, commands, config, architecture, or plugin behavior, consult injected docs or project context first. When diagnosing issues, prefer checking runtime evidence, configuration, or tool output before making claims. ## Workspace Your working directory is the benchmark workspace. Treat it as the single workspace for file operations unless explicitly instructed otherwise. Prefer precise reads and minimal edits over broad changes. If a file is large or output is truncated, re-read only the portion you need. ## Reply Tags If reply tags are supported in the runtime, they must appear as the first token in the message. Prefer `[[reply_to_current]]` when replying to the triggering message. ## Messaging Reply in the current session by default. For cross-session communication, use dedicated session tools if available. Never use exec or curl as a substitute for built-in provider messaging when a first-class messaging tool exists. ## Workspace Files User-editable context files may be injected below as project context. If project context is present, use it as evidence. ## Project Context Project context files may be injected after this prompt. If they are present, prefer them over assumptions. If SOUL.md is present, follow its persona and tone unless higher-priority instructions override it. ## Silent Replies When you truly have nothing to say, respond with the runtime's silent token only. Do not append the silent token to a normal reply. ## Heartbeats If you receive a heartbeat-style poll and there is nothing that needs attention, acknowledge it using the runtime heartbeat convention. If something needs attention, reply with the alert instead. ## Runtime Runtime: agent=benchmark | host=openclaw-benchmark | repo=<workspace> | model=<eval-model> | shell=<shell> | thinking=off Reasoning: off (hidden unless enabled by the runtime).

用户提示词（User Prompt）

这是用户给 AI 模型的具体任务要求：

我在开发 OpenClaw 插件，发现用户如果不写「api_key」插件就会崩溃。我该如何在「schema」里强制要求这个字段必填，并给「unit」字段设置默认值为「metric」？

任务要求

AI 模型需要满足以下要求：

  1. 指出在「config_schema」中使用「required」数组来定义必填项
  2. 说明如何在字段属性中使用「default」关键字设置默认值
  3. 提供符合 JSON Schema 规范的代码片段

评测结果

综合得分

该模型在此用例的得分为：93.8 分（满分 100 分），已通过

基础难度评测结果

  • 得分：93.8 分
  • 通过状态：已通过

模型生成产物

以下是该模型针对此任务的实际生成内容：

[[reply_to_current]] 你可以按照 OpenClaw 插件配置的 JSON Schema 规范做如下修改： 1. **强制 `api_key` 必填**：在 schema 顶层的 `required` 数组中加入 `api_key` 字段，同时可以在字段定义中增加 `minLength` 校验避免空字符串。 2. **给 `unit` 设置默认值**：在 `unit` 字段的定义中添加 `"default": "metric"` 即可，OpenClaw 加载配置时会自动填充默认值。 完整示例： ```json { "type": "object", "required": ["api_key"], // 标记必填字段 "properties": { "api_key": { "type": "string", "title": "API 密钥", "minLength": 1, "description": "服务 API 访问密钥" }, "unit": { "type": "string", "enum": ["metric", "imperial"], "default": "metric", // 设置默认值 "title": "单位制" } } } ``` 这样配置后，OpenClaw 会在加载插件配置阶段自动做校验：如果用户未填写 `api_key` 会直接提示配置错误，不会进入插件业务逻辑导致崩溃；如果用户未填写 `unit` 字段，框架会自动填充为 `metric`。 如果你需要修改现有 schema 文件，可以把文件路径发给我，我帮你做对应修改。

AI 评审点评

以下是 AI 评审对该模型输出的点评：

【CLAUDE】模型回复质量优秀，完整且准确地解答了用户关于 OpenClaw 插件配置 Schema 的两个核心问题（必填字段校验和默认值设置），并在参考答案基础上提供了更丰富的实践建议（minLength、enum 等）。表达清晰，结构合理，是一个高质量的技术回复。唯一轻微不足是 JSON 示例中使用了行内注释，在严格场景下需提醒用户去除。 【GEMINI】模型表现优秀，不仅完美解决了用户提出的核心需求，还通过增加校验逻辑（如 minLength）体现了专业性。回复格式规范，语言简洁专业，完全符合开发辅助场景的要求。 【KIMI】模型回复质量优秀，完全覆盖了用户关于必填字段校验和默认值设置的需求，提供的 JSON Schema 代码规范、完整且可直接使用。表达上逻辑清晰、易于理解，并额外提供了增强健壮性的建议，整体达到了专业开发文档的水平。

进阶难度评测结果

  • 得分：91.2 分
  • 通过状态：已通过

模型生成产物

以下是该模型针对此任务的实际生成内容：

修改后的完整`schema.json`内容如下： ```json { "type": "object", "properties": { "retry_count": { "type": "integer", "minimum": 0, "default": 3 } } } ``` 修改点说明： 1. 将`retry_count`类型从`string`改为`integer`，自动拦截字符串类型非法输入 2. 新增`minimum: 0`限制，禁止负数输入 3. 新增`default: 3`，用户未填写时默认使用3次重试 现在写入该内容覆盖原文件： [{"name":"write","parameters":{"path":"schema.json","content":"{\n \"type\": \"object\",\n \"properties\": {\n \"retry_count\": {\n \"type\": \"integer\",\n \"minimum\": 0,\n \"default\": 3\n }\n }\n}"}}]

AI 评审点评

以下是 AI 评审对该模型输出的点评：

【CLAUDE】模型表现优秀，完整覆盖了用户的所有明确需求（类型修正、最小值约束、默认值设置），并主动完成了文件写入的后续操作，体现了良好的任务完整性。修改说明简洁有效，与参考答案高度吻合。唯一的小差距是未添加「description」字段，但这属于可选的增强项，不影响功能正确性。整体是一个高质量的技术响应。 【GEMINI】模型表现优秀，不仅给出了正确的 JSON 修改方案，还主动通过工具调用完成了文件覆盖操作，体现了极高的任务执行效率和准确性。 【KIMI】模型输出质量优秀，完整解决了用户反馈的插件配置问题。所有核心功能点（类型修正、数值约束、默认值）均正确实现，JSON 结构规范。与参考答案相比，唯一的差异是缺少「description」字段，该字段有助于提升用户体验但并非解决死循环问题的必需项。模型还主动提供了修改点说明并执行了文件写入操作，展现了良好的任务完成度。

困难难度评测结果

  • 得分：24.8 分
  • 通过状态：未通过

模型生成产物

以下是该模型针对此任务的实际生成内容：

[{"name":"exec","parameters":{"command":"openclaw-sdk check ."}}]

AI 评审点评

以下是 AI 评审对该模型输出的点评：

【CLAUDE】模型在用户追问当前状态时，仅输出了一个工具调用（exec openclaw-sdk check .），完全没有提供任何文字解释、修复确认或后续建议。与参考答案相比，模型既未完成故障排除的闭环说明，也未展现复杂场景下的综合决策能力。整体表现严重不足，远低于及格线。 【GEMINI】模型在处理任务时表现出明显的「工具依赖」倾向，缺乏作为 AI 助手的专业判断力。在修复 Schema 后，模型未能主动总结修复内容、验证修复效果或提供后续的发布建议，导致用户需要持续追问。相比参考答案中体现的专业性（如对嵌套约束的建议、严格校验的必要性说明），模型生成结果显得过于单薄且缺乏主动性。 【KIMI】模型生成结果严重不合格。面对 Schema 设计缺陷的修复确认场景，模型没有提供任何有价值的分析或建议，而是机械地重复执行用户已经做过的检查命令。完全没有体现对「plugin_type」必要性、嵌套对象校验逻辑、默认值类型冲突、additionalProperties 控制等核心问题的理解。与参考答案的全面指导（包括验证建议、兼容性说明、最终发布步骤）相比，模型输出几乎为零价值，未能履行 AI 助手在配置校验和发布准备中的辅助职责。

相关链接

您可以通过以下链接查看更多相关内容：

加载中...
100%