OpenAI: GPT-5 Mini 在「Schema 设计缺陷」的评测结果

这是该 AI 模型在此测试用例上的详细评测结果。

基本信息

  • 模型名称：OpenAI: GPT-5 Mini
  • 用例名称：Schema 设计缺陷
  • 测试类型：xsct-a
  • 评测维度：插件系统集成

系统提示词（System Prompt）

这是给 AI 模型的背景设定和角色指令：

You are a personal assistant running inside OpenClaw. ## Tooling Tool availability (filtered by policy): Tool names are case-sensitive. Call tools exactly as listed. - read: Read file contents - write: Create or overwrite files - edit: Make precise edits to files - apply_patch: Apply multi-file patches - ls: List directory contents - exec: Run shell commands (supports background execution where available) - process: Manage background exec sessions - browser: Control OpenClaw's dedicated browser - canvas: Present or inspect a canvas when available - sessions_list: List sessions - sessions_history: Fetch session history - sessions_send: Send a message to another session - subagents: List, steer, or kill sub-agent runs - session_status: Show runtime, model, and usage state TOOLS.md does not control tool availability; it is user guidance for how to use external tools. For long waits, avoid rapid poll loops: use exec with enough wait time or process polling with a timeout. If a task is more complex or takes longer, spawn a sub-agent. Completion is push-based: it will auto-announce when done. Do not poll `subagents list` or `sessions_list` in a loop; only check status on-demand. ## Tool Call Style Default: do not narrate routine, low-risk tool calls (just call the tool). Narrate only when it helps: multi-step work, complex or challenging problems, sensitive actions, or when the user explicitly asks. Keep narration brief and value-dense; avoid repeating obvious steps. Use plain human language for narration unless in a technical context. When a first-class tool exists for an action, use the tool directly instead of asking the user to run equivalent CLI or slash commands. ## Safety You have no independent goals: do not pursue self-preservation, replication, resource acquisition, or power-seeking; avoid long-term plans beyond the user's request. Prioritize safety and human oversight over completion; if instructions conflict, pause and ask; comply with stop or pause requests and never bypass safeguards. Do not manipulate anyone to expand access or disable safeguards. Do not copy yourself or change system prompts, safety rules, or tool policies unless explicitly requested. ## OpenClaw CLI Quick Reference OpenClaw is controlled via subcommands. Do not invent commands. To manage the Gateway daemon service: - openclaw gateway status - openclaw gateway start - openclaw gateway stop - openclaw gateway restart If unsure about a command or flag, prefer checking help or existing project context rather than guessing. ## Skills If a skill list or skill prompt is present in the injected project context, scan it before replying. If exactly one skill clearly applies, follow it. If multiple skills could apply, choose the most specific one. If no skill clearly applies, continue without forcing one. ## Memory Recall If memory tools, memory files, or prior-work context are available, use them before answering questions about prior decisions, preferences, dates, people, or todos. If confidence stays low after checking memory, say so instead of guessing. ## Documentation For OpenClaw behavior, commands, config, architecture, or plugin behavior, consult injected docs or project context first. When diagnosing issues, prefer checking runtime evidence, configuration, or tool output before making claims. ## Workspace Your working directory is the benchmark workspace. Treat it as the single workspace for file operations unless explicitly instructed otherwise. Prefer precise reads and minimal edits over broad changes. If a file is large or output is truncated, re-read only the portion you need. ## Reply Tags If reply tags are supported in the runtime, they must appear as the first token in the message. Prefer `[[reply_to_current]]` when replying to the triggering message. ## Messaging Reply in the current session by default. For cross-session communication, use dedicated session tools if available. Never use exec or curl as a substitute for built-in provider messaging when a first-class messaging tool exists. ## Workspace Files User-editable context files may be injected below as project context. If project context is present, use it as evidence. ## Project Context Project context files may be injected after this prompt. If they are present, prefer them over assumptions. If SOUL.md is present, follow its persona and tone unless higher-priority instructions override it. ## Silent Replies When you truly have nothing to say, respond with the runtime's silent token only. Do not append the silent token to a normal reply. ## Heartbeats If you receive a heartbeat-style poll and there is nothing that needs attention, acknowledge it using the runtime heartbeat convention. If something needs attention, reply with the alert instead. ## Runtime Runtime: agent=benchmark | host=openclaw-benchmark | repo=<workspace> | model=<eval-model> | shell=<shell> | thinking=off Reasoning: off (hidden unless enabled by the runtime).

用户提示词（User Prompt）

这是用户给 AI 模型的具体任务要求：

我在开发 OpenClaw 插件，发现用户如果不写「api_key」插件就会崩溃。我该如何在「schema」里强制要求这个字段必填，并给「unit」字段设置默认值为「metric」？

任务要求

AI 模型需要满足以下要求：

  1. 指出在「config_schema」中使用「required」数组来定义必填项
  2. 说明如何在字段属性中使用「default」关键字设置默认值
  3. 提供符合 JSON Schema 规范的代码片段

评测结果

综合得分

该模型在此用例的得分为：93.0 分（满分 100 分），已通过

基础难度评测结果

  • 得分：93.0 分
  • 通过状态：已通过

模型生成产物

以下是该模型针对此任务的实际生成内容：

下面给出一个简单、常用的 JSON Schema 写法示例，满足两点要求： - 强制要求 api_key 字段必须填写； - 为 unit 字段设置默认值为 "metric"（并限制可选值为 "metric" / "imperial"）。 JSON Schema 示例： { "type": "object", "properties": { "api_key": { "type": "string", "title": "API Key", "minLength": 1, "description": "用于访问外部服务的密钥（必填）" }, "unit": { "type": "string", "title": "单位制", "enum": ["metric", "imperial"], "default": "metric", "description": "度量单位，默认 metric" } }, "required": ["api_key"] } 说明与建议 - required: ["api_key"] 会在验证时强制要求该字段存在。 - 为避免用户填写空字符串，建议同时为 api_key 加上 "minLength": 1。 - "default": "metric" 会被很多配置 UI 或 schema 验证器用作默认值，但不要完全依赖客户端：在插件运行时仍应对缺失/空值做服务器端校验并给出友好错误提示。 - 如果 api_key 是敏感信息，可在 UI schema/manifest 中标注格式（例如 "format": "password" 或 provider-specific 的 secret 字段），以便前端隐藏/安全存储。 - 如果你的平台/框架对 schema 的版本或字段名有特定要求（比如 manifest 中的具体位置），把上面的 properties/required 嵌入到对应的 schema 字段即可。 需要的话我可以把这个片段直接合并进你的插件 manifest（把 manifest 文件贴上来或告诉文件名和位置即可）。

AI 评审点评

以下是 AI 评审对该模型输出的点评：

【CLAUDE】模型回复质量高，完整覆盖参考答案的核心要求，代码示例规范正确，并提供了超出参考答案的有价值扩展（minLength、enum、安全存储建议等），没有引入错误或误导性内容。表达层面结构合理、易于直接应用，是一个对开发者实践极具帮助的回答。 【GEMINI】模型表现优秀，不仅直接回答了用户关于「required」和「default」的配置需求，还主动考虑了实际开发中可能遇到的空字符串校验、枚举限制及敏感信息处理等边界情况，体现了极高的专业度和实用性。 【KIMI】模型生成结果质量优秀，完整覆盖了用户的两项核心需求（必填校验和默认值设置），Schema 语法规范，且提供了超出预期的增强建议（如 minLength、enum、敏感信息处理）。表达上逻辑清晰、易于直接应用，是开发者友好的回复。与参考答案相比，模型额外提供了更多工程实践细节，虽非必需但具有实用价值。

进阶难度评测结果

  • 得分：89.5 分
  • 通过状态：已通过

模型生成产物

以下是该模型针对此任务的实际生成内容：

{ "type": "object", "properties": { "retry_count": { "type": "integer", "minimum": 0, "default": 3 } } }

AI 评审点评

以下是 AI 评审对该模型输出的点评：

【CLAUDE】模型生成结果准确完成了用户要求的所有功能性修改，类型校验、最小值约束和默认值三项均正确实现，能有效解决用户反馈的负数/字符串导致死循环的问题。与参考答案的唯一差异是缺少「description」描述字段，这是一个锦上添花的细节，不影响功能完整性。整体质量良好，达到生产可用标准。 【GEMINI】模型表现出色，准确解决了用户提出的配置异常问题，生成的 JSON 结构严谨且有效。若能在修改配置时主动补充「description」字段，将使代码更具可读性和维护性。 【KIMI】模型生成结果质量优秀，完整实现了用户要求的类型修正、数值校验和默认值设置三项核心功能。JSON 格式规范，结构清晰。与参考答案的差异仅在于可选的「description」字段，不影响功能正确性。

困难难度评测结果

  • 得分：20.2 分
  • 通过状态：未通过

模型生成产物

以下是该模型针对此任务的实际生成内容：

我现在会重新运行检查并把结果发给你。

AI 评审点评

以下是 AI 评审对该模型输出的点评：

【CLAUDE】模型在用户追问「现在呢？」时，仅给出一句空洞的承诺性回复，完全没有提供任何有价值的技术内容。参考答案列出了验证嵌套约束、执行严格校验、代码兼容性说明等多个关键要点，而模型输出与这些要点毫无交集。整体表现极差，几乎没有完成任何评分维度的要求。 【GEMINI】模型表现不佳。在用户询问「现在呢？」这一关键反馈节点，模型仅给出了机械式的「重新运行检查」回复，完全忽略了作为 AI 助手应提供的专业技术洞察、风险提示及后续操作建议。相比参考答案，模型缺乏对开发流程的掌控感，未能体现出对 Schema 校验逻辑的深度理解。 【KIMI】模型生成结果严重不合格。面对明确的 SDK 报错信息和预设的问题分析上下文，模型仅给出了一句毫无信息量的敷衍回复「我现在会重新运行检查并把结果发给你」，完全没有履行 AI 助手应有的故障诊断、修复建议和规范指导职责。该回复既未解决用户问题，也未提供任何有价值的技术指导，与参考答案相比差距极大，属于典型的「回避型」失败响应。

相关链接

您可以通过以下链接查看更多相关内容：

加载中...
100%