一代人就是一次 rollout：进化论是种群的强化学习？

核心抓手

进化论，本质上来说是种群的强化学习。

一代人就是一次 rollout。

先保留这两个具有传播力的表述，后续讨论再逐步提高严格性，不急着形成定稿。

Working model

可以先把自然选择抽象成：

variation → environment interaction → differential fitness → selection → inheritance → next iteration

它和强化学习/搜索中的：

policy variation → rollout → reward → update/select → next rollout

在结构上高度相似。优化对象不是单一个体，而是种群中的遗传策略/参数分布；fitness 最终由相对繁殖成功率体现。

一个需要保留的精度问题

“一代人就是一次 rollout”很抓眼球，但严格映射可能是：

• 一个个体的一生更像一次 rollout：从初始遗传参数出发，与环境长期交互，最后得到一个高度延迟、稀疏的 fitness 结果；

• 一代种群更像一批并行 rollouts / 一个 training iteration；

• 下一代的基因频率变化对应 update。

因此后续可以研究：传播表达是否保留“一代人就是一次 rollout”，而正文再解释 batch/iteration 的严格对应。

目前最有意思的延伸

1. 极端长 horizon + 稀疏 reward：自然选择几乎没有显式 credit assignment，却仍能通过海量并行试验优化。

1. reward 不是固定函数：fitness 依赖环境、其他个体和种群频率，是典型的 multi-agent、non-stationary optimization。

1. 进化最终进化出了学习本身：外层遗传进化塑造能够在生命周期内学习的神经系统，于是出现 evolution → individual learning 的两层优化。

1. 文化可能是第三层优化：信息可以跨个体传播而不等待遗传世代，显著提高更新频率。

1. 与 AI 实验室的类比：模型/训练目标变体 → 大规模 rollout → benchmark/用户/商业反馈 → 下一轮目标与训练选择，可能属于同一个更一般的 variation → selection → inheritance 框架。

Open questions

• 自然选择更适合类比 reinforcement learning、evolution strategies、population-based training，还是更一般的 search/optimization？

• “reward”对应 fitness 到什么程度会开始误导？繁殖成功率是否足以作为统一目标函数？

• 性选择、亲缘选择、群体选择、生态位构建等现象如何改变这个 RL 类比？

• 个体学习会反过来改变选择压力；这是否更接近 meta-RL / learned optimizer，而非简单的双层优化？

• 从进化视角看，现代 AI 的 gradient-based RL 相比自然选择究竟压缩了哪些搜索成本：credit assignment、sample efficiency、generation time，还是全部？

• 这套类比能否反过来帮助理解多 Agent、群体智能、持续学习和实验室层面的训练目标演化？

Maintenance rule

这是一个开放 Thread。后续优先补充能够强化、削弱或迫使我们修改上述类比的机制、数学对应和反例；普通的“进化算法像 RL”材料不单独堆积。

Workshop deliverable — 2026-10-21: Where the population-RL analogy holds and breaks

Status: rigorous argument map and worked example; still an exploratory Thread, not a finished article.

Revised claim. Biological evolution, evolution strategies, population-based training and reinforcement learning share a general feedback structure: generate variation, expose alternatives to an environment, measure differential outcomes, and change the distribution of future strategies. In mathematical and mechanistic terms, natural selection is closer to replicator–mutator dynamics / population-level black-box search than to modern gradient-based policy optimization. The rhetorical headline “进化论是种群的强化学习” can work if the body immediately explains this distinction.

Correct mapping. One individual's life is a trajectory/episode (though not necessarily one neatly bounded task); one cohort of individuals is a batch of parallel trajectories; differential reproductive success weights inheritance; change in hereditary-type frequencies is the update. In overlapping-generation species even the batch boundary is approximate. Individual learning changes behavior within a lifetime; evolution changes inherited variation across generations.

Worked example: an update with no gradient. In a simple asexual haploid population, strategy A starts at 60% and leaves 1.5 surviving descendants per individual; B starts at 40% and leaves 2.0. After reproduction, A's expected frequency is (0.60 × 1.5) / (0.60 × 1.5 + 0.40 × 2.0) = 52.9%; B rises to 47.1%. No individual needs to infer why B reproduced better, compute an advantage, or update its policy. This is the simplified replicator recurrence p'ᵢ = pᵢwᵢ / Σⱼpⱼwⱼ; sexual recombination, mutation, drift and population structure complicate the biological case.

Three important non-equivalences.

• Optimization target: fitness is realized relative reproductive contribution under ecological and social conditions, not a fixed external scalar reward supplied by a trainer. It can depend on the population's own composition; mean fitness need not monotonically increase under frequency-dependent selection.

• Learning signal: PPO/GRPO-style RL estimates how action probabilities should change from sampled returns; evolution can shift hereditary frequencies without action-level credit assignment. OpenAI's evolution-strategies work compares these black-box and policy-gradient methods, while warning that modern ES is mathematically abstracted away from actual biology.

• Inheritance and agency: genes, development, cultural transmission, lifetime learning, mutation, drift and recombination interact. Evolution has no single designer, no global optimality guarantee, and no requirement that individual organisms become better at learning.

Useful deeper thesis. Evolution can shape an organism's ability to learn; lifetime learning can in turn alter selective pressures; culture transmits successful behavior much faster than genetic generations. This suggests interacting optimization timescales (genetic, individual, cultural) rather than a single loop. It also makes a careful analogy to AI laboratories possible: model variants, training recipes, scaffolds and evaluators can be selected across releases, while within-run RL updates weights. The lab analogy remains a hypothesis about organization-level optimization, not a literal evolutionary identity.

Potential falsifier of the strong slogan: if the explanation needs a fixed reward function, explicit reward delivery, policy-gradient credit assignment or improvement of the same individual policy to remain coherent, it no longer describes much of natural selection. The defensible generalization is feedback-driven selection over a population of heritable variants.

Next output: a short Chinese essay that opens with “一代人就是一次 rollout,” immediately corrects this to “一代种群是一批并行 rollouts,” works through the two-strategy example, then ends with the three nested timescales. Keep the provocative headline as a question until the mechanism is established.

Primary/reference sources: Nature Education: selection, drift and gene flow; OpenAI: Evolution strategies as an alternative to RL; Population Based Training, original paper; Fitness and its role in evolutionary genetics; Frequency-dependent selection and mean fitness.