返回首页
笔记

模型版本号与内部评估—真实使用反馈 gap

Living Thread: what model versioning encodes, and why frontier labs’ internal capability judgments can diverge from external real-world utility and release reception.

LLM EvaluationAI Model Development

Current question

不同实验室的模型版本号/代际命名究竟编码什么？为什么实验室内部对新模型能力的判断，有时会与首发后的真实用户反馈明显脱节？

Working hypothesis

这个问题可以从两个交叉视角研究：

  • versioning：major/minor/preview/dated snapshot 等命名在不同实验室内部究竟对应架构、训练阶段、后训练目标、产品行为还是发布管理；
  • calibration gap：内部 eval/capability perception 与真实世界 utility、偏好和工作流表现之间如何产生偏差，又如何通过后续小版本快速修正。

真实世界反馈可以被视为实验室层面的稀疏奖励信号：它未必直接训练当前模型，但会影响下一轮目标定义、后训练和产品决策。因此短周期版本修正本身可能暴露内部 eval 的盲区。

Open questions

  • major/minor 版本在 OpenAI、Anthropic、Google DeepMind 等实验室分别代表什么？是否存在可比较的规律？
  • 哪些能力最容易在内部 benchmark/eval 上被高估，而在真实工作流中暴露问题？
  • 首发后快速小版本迭代主要在修 capability、behavior/personality、tool use、latency/cost，还是 routing/harness？
  • 外部反馈如何进入实验室的 post-training/eval loop？哪些证据能区分模型权重变化与产品/harness 修正？
  • 能否用版本时间线、首发反馈和后续修正构建一种“internal-external calibration”研究方法？

Maintenance rule

当前方法和判断均为 provisional。先积累跨实验室版本案例、官方说明和可复核的外部反馈，再决定是否形成稳定框架。

Workshop deliverable — 2026-10-14: Model-version calibration study v0.1

Working claim. A public model name, a pinned model ID, and the full product configuration are different experimental units. A perceived post-launch “model regression” can arise from weights/post-training, serving infrastructure, routing and tools, user-interface defaults, or the population of tasks and preferences being measured. Attribution requires identifying which unit changed.

First documented case: GPT-5 launch and correction

  • 2025-08-07: GPT-5 became ChatGPT's default with automatic model switching. This already makes the consumer experience a model-plus-router intervention.
  • 2025-08-12: OpenAI added Auto/Fast/Thinking selection and restored GPT-4o access for paid users. This is an observable product/control change, independently of any underlying model-weight change.
  • 2025-08-15: OpenAI announced a warmer default GPT-5 personality in response to user feedback that it felt too reserved. The release notes establish the intent and user-feedback cause, but do not by themselves isolate a weights change from prompt or serving changes.
  • 2025-11-12: GPT-5.1 Instant and Thinking were announced with explicit improvements in behavior, instruction following, and adaptive reasoning, alongside personalization changes. OpenAI said the 5.1 designation signified meaningful improvement within the GPT-5 generation.

Primary sources: ChatGPT release notes and GPT-5.1 announcement.

Cross-lab version semantics. Anthropic's model-ID specification says Claude 4.6-and-later dateless IDs are pinned snapshots; earlier dateless aliases could point to dated snapshots. Crucially, even a fixed model ID can show small behavioral changes because routing, safety classifiers, and sampling infrastructure evolve. Google's Gemini documentation distinguishes stable, preview, latest aliases, and experimental endpoints; a latest alias may hot-swap. Thus version labels cannot be treated as a cross-vendor ontology of training or capability.

Evaluation protocol: separate model change from product change

Unit of observation: exact model/API ID and date; product and client version; routing mode; system/developer instructions; tools and scaffold; sampling/reasoning settings; region and rollout cohort; request date; task and evaluator.

Paired comparisons: (A) pinned API snapshots under identical harness and prompts, to estimate model-level change where snapshots remain available; (B) same model ID through product configurations, to estimate serving/UI/routing effects where controllable; (C) pre/post product experience, to measure actual user utility. Keep these three estimates separate. A historical launch cannot be reconstructed experimentally if earlier product configurations or snapshots are no longer accessible.

Task panel: 24 reproducible tasks, six each in factual/reasoning work, coding/agent execution, writing and interaction quality, and long-context/multi-step workflow continuity. Repeat each configuration at least three times when stochasticity matters. Record objective task completion, latency, cost, refusal/instruction-following, and blind pairwise human preference. Include user-specific work tasks and generic tasks to expose distribution shift. Track model identity and protocol comparability explicitly; do not infer causal weight changes from product release notes alone.

Testable hypotheses: H1: much of early user dissatisfaction can occur without a decline in objective task completion, due to personality and workflow expectations. H2: short-cycle fixes disproportionately target product, routing, and post-training behavior relative to major architecture changes. H3: stable aggregate benchmarks mask heterogeneous regressions in user task clusters. All three remain hypotheses pending controlled evidence.

Falsifiers: blinded, matched task comparisons show a broad decline in objective completion even after controlling routing, scaffold, and interaction style; or primary evidence attributes the relevant change to a clearly identified model revision. The first research output should be a source-verified release-event timeline with explicit “known changed layer / unknown layer” columns, not an unsupported ranking of labs.

Research status: first case timeline and experimental design completed; no causal claim about the internals of GPT-5's August correction has been established.