<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://trayfouryu.github.io/wam_rl/feed.xml" rel="self" type="application/atom+xml" /><link href="https://trayfouryu.github.io/wam_rl/" rel="alternate" type="text/html" /><updated>2026-10-09T22:45:23+08:00</updated><id>https://trayfouryu.github.io/wam_rl/feed.xml</id><title type="html">Trayfour Yu</title><subtitle>Research notes on embodied AI, vision-language-action models, and reinforcement learning for robot manipulation.</subtitle><entry><title type="html">Dual-Branch Reinforcement Learning for World-Action Models</title><link href="https://trayfouryu.github.io/wam_rl/embodied-ai/reinforcement-learning/2026/09/02/dual-branch-rl-for-wam.html" rel="alternate" type="text/html" title="Dual-Branch Reinforcement Learning for World-Action Models" /><published>2026-09-02T10:00:00+08:00</published><updated>2026-09-02T10:00:00+08:00</updated><id>https://trayfouryu.github.io/wam_rl/embodied-ai/reinforcement-learning/2026/09/02/dual-branch-rl-for-wam</id><content type="html" xml:base="https://trayfouryu.github.io/wam_rl/embodied-ai/reinforcement-learning/2026/09/02/dual-branch-rl-for-wam.html"><![CDATA[<!-- MathJax v3. Remove this block if your theme already loads MathJax. -->
<script>
  MathJax = {
    tex: {
      inlineMath:  [['$', '$'], ['\\(', '\\)']],
      displayMath: [['$$', '$$'], ['\\[', '\\]']],
      processEscapes: true,
      tags: 'ams'
    },
    options: { skipHtmlTags: ['script', 'noscript', 'style', 'textarea', 'pre'] }
  };
</script>

<script id="MathJax-script" async="" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>

<p><strong>TL;DR.</strong> Reinforcement learning (RL) recipes for vision-language-action (VLA) models update <em>one</em> decision variable: the action. A world-action model (WAM) has <em>two</em> — the action chunk <strong>and</strong> the imagined future. We argue that action-only RL leaves the world branch task-blind, and propose <strong>dual-branch RL</strong>, a PPO-style objective in which the video branch is also made accountable for task return. On LIBERO-Long with Fast-WAM as the base policy, dual-branch RL lifts the success rate from $91.1\%$ to $\mathbf{96.0}\%$ in 1k steps, versus $94.5\%$ for action-only Flow-SDE.</p>

<hr />

<h2 id="1-introduction">1. Introduction</h2>

<p><strong>From VLA to RL post-training.</strong> Vision-language-action models have become the dominant recipe for generalist robot policies. RT-2 [1] casts robot actions as text tokens and co-fine-tunes a vision-language model on web and robot data; OpenVLA [2] provides an open 7B discrete-action instantiation; $\pi_0$ [3] replaces action tokens with a <em>flow-matching action expert</em>, yielding continuous action chunks with high-frequency control. Imitation learning alone, however, is bounded by demonstration coverage. This has motivated a wave of RL post-training methods that convert a supervised VLA into a return-maximizing policy: <strong>SimpleVLA-RL</strong> [4] scales online RL for discrete-action VLAs, while <strong>$\pi$RL</strong> [5] targets flow-based VLAs with two flow-compatible estimators — <strong>Flow-Noise</strong>, which treats denoising as a discrete-time MDP with a learnable noise network for exact log-likelihoods, and <strong>Flow-SDE</strong>, which couples a denoising MDP with the environment MDP through an ODE-to-SDE conversion.</p>

<p><strong>World-action models.</strong> In parallel, <strong>world-action models (WAMs)</strong> [6] have emerged as a response to the <em>supervision deficit</em> of pure VLA training: instead of predicting actions alone, the model also predicts how the visual world evolves, and the predicted future is made <em>action-facing</em> [6]. A representative instantiation is <strong>Fast-WAM</strong> [7], a mixture-of-transformers (MoT) that couples a video diffusion transformer with an action expert under shared attention, and shows that video co-training is essential while test-time future imagination can largely be skipped.</p>

<p><strong>The gap.</strong> RL methods for WAMs are, to date, inherited verbatim from the VLA literature: they apply a flow-compatible estimator such as Flow-SDE to the <em>action branch only</em>. We contend that this under-uses the architecture. In Fast-WAM, action tokens are <strong>not</strong> allowed to attend to future video tokens, and the clean first-frame latent is the only shared anchor between the two branches [7]. Consequently the world branch can influence control <em>only</em> through the shared representation — and under action-only RL that representation receives no gradient from the task return. The video branch keeps minimizing a flow-matching reconstruction loss that is indifferent to whether the imagined future corresponds to a successful grasp or a dropped object. The capacity spent on future prediction is, in control terms, never credited.</p>

<p><strong>Our approach.</strong> We propose <strong>dual-branch RL</strong>: both branches of a WAM are treated as stochastic policies over their own decision variables and are updated with a <em>shared</em> advantage estimate. Concretely,</p>

<ol>
  <li><strong>Action branch.</strong> We adopt Flow-SDE [5] and derive a closed-form per-step Gaussian importance ratio, so the likelihood of the whole denoising chain is available without numerical integration (Sec. 3.2).</li>
  <li><strong>Video branch.</strong> Following the treatment of continuous latent tokens in LaST-R1 [8], we model the video latent as an isotropic Gaussian centered at the current policy output and obtain a lightweight step-level ratio over subsampled latent tokens (Sec. 3.3).</li>
  <li><strong>Mixed ODE–SDE rollout.</strong> Integrating likelihoods over the entire video denoising chain is prohibitive. We therefore randomize a single denoising index per rollout and compute the video-branch RL loss only there, keeping memory and compute close to a pure-ODE rollout (Sec. 3.4).</li>
</ol>

<p>On LIBERO-Long, starting from the same Fast-WAM checkpoint, dual-branch RL reaches $96.0\%$ success after 1k RL steps versus $94.5\%$ for action-only Flow-SDE — a $+1.5$ point margin that opens up early and persists.</p>

<hr />

<h2 id="2-preliminaries">2. Preliminaries</h2>

<h3 id="21-flow-based-action-generation">2.1 Flow-based action generation</h3>

<p>Let $\mathbf{a}_t \in \mathbb{R}^{H \times d}$ denote an action chunk of horizon $H$ predicted at environment step $t$, and let $\tau \in [0, 1]$ be the continuous flow time. Conditional flow matching [9, 10] defines a probability path from noise to data by linear interpolation,</p>

\[\mathbf{a}_t^{\tau} = \tau \, \mathbf{a}_t + (1 - \tau) \, \boldsymbol{\epsilon}, \qquad \boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I}),\]

<p>and regresses a velocity field $\mathbf{v}<em>\theta(\mathbf{a}_t^{\tau}, \tau, \mathbf{c}_t)$ toward $\dot{\mathbf{a}}_t^{\tau} = \mathbf{a}_t - \boldsymbol{\epsilon}$, where $\mathbf{c}_t$ is the shared observation–language context. Sampling integrates the ODE $\mathrm{d}\mathbf{a}^{\tau} = \mathbf{v}</em>\theta \, \mathrm{d}\tau$ from $\tau = 0$ to $\tau = 1$. Under this parameterization the one-step estimate of the clean action is</p>

\[\hat{\mathbf{a}}_t^{\tau} = \mathbf{a}_t^{\tau} + (1 - \tau) \, \mathbf{v}_\theta(\mathbf{a}_t^{\tau}, \tau, \mathbf{c}_t).\]

<p>Because sampling is deterministic, a pure-ODE policy has no tractable, non-degenerate action likelihood — which is precisely why RL on flow policies requires an explicit change of measure [5].</p>

<h3 id="22-world-action-models">2.2 World-action models</h3>

<p>A WAM augments the action expert with a <em>world branch</em> that predicts future visual latents $\mathbf{Z}<em>t = (\mathbf{z}</em>{t,1}, \dots, \mathbf{z}_{t,N})$ of the next $T$ frames. Fast-WAM [7] implements this as an MoT over a pretrained video DiT and a lightweight action DiT, partitioning tokens into (i) clean latents of the first observation frame, which act as the shared visual anchor, (ii) noisy latents of future frames, used <strong>only during training</strong>, and (iii) action tokens. The attention mask is deliberately asymmetric: future video tokens are invisible to action tokens, and first-frame tokens attend to nothing. Training minimizes</p>

\[\mathcal{L}_{\text{SFT}} = \underbrace{\mathcal{L}_{\text{FM}}(\mathbf{a}_{1:H})}_{\text{action branch}} \;+\; \lambda \underbrace{\mathcal{L}_{\text{FM}}(\mathbf{Z}_{1:T})}_{\text{video branch}} .\]

<p>Two properties of this design motivate our method. First, the video branch is <em>load-bearing</em>: removing video co-training degrades performance far more than removing test-time imagination [7]. Second, the branch is <em>architecturally decoupled</em> at inference: with action tokens masked from future video tokens, the only channel through which world modeling can shape control is the shared representation. Dual-branch RL is designed to make that channel carry return information.</p>

<hr />

<h2 id="3-dual-branch-rl">3. Dual-Branch RL</h2>

<h3 id="31-a-shared-advantage-objective-over-two-branches">3.1 A shared-advantage objective over two branches</h3>

<p>Let $\mathbf{a}_t$ be the action chunk and $\mathbf{Z}_t$ the future latent produced at environment step $t$. We write the WAM policy as a product over branches conditioned on the shared context $\mathbf{c}_t$,</p>

\[\pi_\theta(\mathbf{a}_t, \mathbf{Z}_t \mid \mathbf{c}_t) \;=\; \pi_\theta^{a}(\mathbf{a}_t \mid \mathbf{c}_t) \; \pi_\theta^{z}(\mathbf{Z}_t \mid \mathbf{c}_t),
\tag{1}\]

<p>i.e. we treat the branches as conditionally independent given $\mathbf{c}_t$. This is an approximation — the two branches interact through shared attention — but it is the same factorization adopted by LAPO in LaST-R1 [8] for jointly optimizing latent reasoning and action generation, and it keeps each ratio computable in closed form.</p>

<p>Both branches are then updated with the <strong>same</strong> advantage estimate $\hat{A}_t$ (GAE [11]) computed from the environment return:</p>

\[\mathcal{L}_{\text{policy}}(\theta)
= -\,\mathbb{E}_t \Bigg[ \sum_{k \in \{a, z\}}
\min\Big( r_t^{k}(\theta) \, \hat{A}_t ,\;
\operatorname{clip}\big( r_t^{k}(\theta),\, 1 - \epsilon_{\min},\, 1 + \epsilon_{\max} \big) \, \hat{A}_t \Big) \Bigg].
\tag{2}\]

<p>Here $r_t^{k}(\theta) = \pi_\theta^{k}(\cdot \mid \mathbf{c}<em>t) / \pi</em>{\theta_{\text{old}}}^{k}(\cdot \mid \mathbf{c}<em>t)$ is the per-branch importance ratio, and $(\epsilon</em>{\min}, \epsilon_{\max})$ are asymmetric clip bounds: we follow LAPO [8] in decoupling the lower and upper range instead of using the symmetric range of vanilla PPO [12], which lets the update grow faster on positively advantaged samples while remaining conservative on negative ones. The ratio is evaluated on the <em>stored</em> rollout variables under both the current and the behavior policy. We omit the value-function and entropy terms for brevity; the full objective is $\mathcal{L} = \mathcal{L}<em>{\text{policy}} + \lambda_v \mathcal{L}</em>{\text{value}} + \lambda_{\text{SFT}} \mathcal{L}_{\text{SFT}}$.</p>

<blockquote>
  <p><strong>Why share the advantage?</strong> $\hat{A}_t$ measures whether the <em>executed action</em> was better than expected. Feeding the same signal into the video branch is exactly the inductive bias we want: imagined futures that precede a high-return action are reinforced, futures that precede a failure are suppressed. The video branch stops being a passive reconstruction head and becomes an action-facing predictor in the sense of [6].</p>
</blockquote>

<h3 id="32-action-branch-flow-sde-with-a-closed-form-ratio">3.2 Action branch: Flow-SDE with a closed-form ratio</h3>

<p>We convert the deterministic flow-matching ODE of Sec. 2.1 into a stochastic differential equation that preserves the marginal $q_\tau(\mathbf{a}^{\tau})$ [5]:</p>

\[\mathrm{d}\mathbf{a}^{\tau} =
\Big[ \underbrace{\mathbf{v}_\theta(\mathbf{a}^{\tau}, \tau, \mathbf{c}_t)}_{\text{velocity}}
\;-\; \tfrac{1}{2}\sigma_\tau^{2} \underbrace{\nabla \log q_\tau(\mathbf{a}^{\tau})}_{\text{score}} \Big] \mathrm{d}\tau
\;+\; \sigma_\tau \, \mathrm{d}\mathbf{w}_\tau .
\tag{3}\]

<p>For the linear path of Sec. 2.1, the score admits the identity [5]</p>

\[\nabla \log q_\tau(\mathbf{a}^{\tau}) = -\,\frac{\mathbf{a}^{\tau}}{\tau} - \frac{1 - \tau}{\tau}\,\mathbf{v}_\theta(\mathbf{a}^{\tau}, \tau, \mathbf{c}_t)
= -\,\frac{\hat{\mathbf{a}}^{\tau}}{\tau},
\tag{4}\]

<p>using the one-step clean-action estimate of Sec. 2.1. Substituting Eq. (4) into Eq. (3) gives the compact drift used throughout this post:</p>

\[\boxed{\;
\mathrm{d}\mathbf{a}^{\tau} =
\Big[ \mathbf{v}_\theta(\mathbf{a}^{\tau}, \tau, \mathbf{c}_t) + \frac{\sigma_\tau^{2}}{2\tau}\,\hat{\mathbf{a}}^{\tau} \Big] \mathrm{d}\tau
+ \sigma_\tau \, \mathrm{d}\mathbf{w}_\tau
\;}
\tag{5}\]

<p>Discretizing with step $\delta = 1/K$ and writing $\mathbf{u}<em>\theta^{\tau} \triangleq \mathbf{v}</em>\theta + \tfrac{\sigma_\tau^2}{2\tau}\hat{\mathbf{a}}^{\tau}$ for the drift, the transition is an isotropic Gaussian [5],</p>

\[p_\theta\big(\mathbf{a}^{\tau + \delta} \mid \mathbf{a}^{\tau}\big) = \mathcal{N}(\boldsymbol{\mu}_\tau, \boldsymbol{\Sigma}_\tau), \qquad
\begin{cases}
\boldsymbol{\mu}_\tau = \mathbf{a}^{\tau} + \mathbf{u}_\theta^{\tau} \, \delta,\\[2pt]
\boldsymbol{\Sigma}_\tau = \sigma_\tau^{2} \, \delta \, \mathbf{I}.
\end{cases}
\tag{6}\]

<h3 id="33-video-branch-latent-tokens-as-gaussian-decision-variables">3.3 Video branch: latent tokens as Gaussian decision variables</h3>

<p>The video branch emits a tensor of continuous latents rather than a categorical token, so no exact likelihood is available. Following LaST-R1 [8], we place an isotropic Gaussian of fixed width $\sigma_z$ around the <em>deterministic</em> output of the current policy and treat the latent tensor as the branch’s decision variable:</p>

\[\pi_\theta^{z}(\mathbf{Z}_t \mid \mathbf{c}_t)
\;\propto\;
\exp\!\left( -\,\frac{1}{2\sigma_z^{2}} \sum_{i=1}^{N_z} \big\lVert \mathbf{z}_{t,i}^{\text{old}} - \mathbf{z}_{t,i}^{\theta} \big\rVert^{2} \right),
\tag{8}\]

<p>where $\mathbf{z}<em>{t,i}^{\text{old}}$ are the latents stored in the rollout buffer, $\mathbf{z}</em>{t,i}^{\theta}$ the latents the current policy would produce for the same context, and $N_z$ the number of tokens retained after a downsampling.</p>

<p>When the branch is rolled out deterministically, the stored latents <em>are</em> the behavior policy’s output, $\mathbf{z}<em>{t,i}^{\theta</em>{\text{old}}} = \mathbf{z}_{t,i}^{\text{old}}$, the second term vanishes, and Eq. (9) reduces to the single-term form used in [8]:</p>

\[r_t^{z}(\theta)
= \frac{\pi_\theta(\mathbf{Z}_t^{\text{old}} \mid \cdot)}{\pi_{\theta_{\text{old}}}(\mathbf{Z}_t^{\text{old}} \mid \cdot)}
= \exp\!\left( -\frac{1}{2\sigma_z^{2}} \sum_{i=1}^{N_z} \big\lVert \mathbf{z}_{t,i}^{\text{old}} - \mathbf{z}_{t,i}^{\theta} \big\rVert^{2} \right).
\tag{10}\]

<h3 id="34-randomly-choose-steps">3.4 Randomly choose steps</h3>

<p>Making the video branch stochastic at <em>every</em> one of the $K$ denoising steps would require storing $K$ latent tensors per environment step and backpropagating a ratio through the full chain — prohibitive for a 5B-parameter video DiT. We instead adopt a <strong>Random Strategy</strong>: we draw a single index $n \sim \mathcal{U}{0, \dots, K-1}$, where pre-step latent feature noise is injected and the transition is recorded， and the video-branch ratio is computed at step $n$ only.</p>

<h3 id="35-algorithm">3.5 Algorithm</h3>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Algorithm 1  Dual-branch PPO for WAM policies (one iteration)
──────────────────────────────────────────────────────────────────────────────
Require: WAM policy π_θ = (action expert, video DiT), SFT weights θ, value net V_φ,
         denoising steps K, step δ = 1/K, schedule scales α_a, α_z,
         video-token subsample size N_z, clip bounds (ε_min, ε_max)

 1:  θ_old ← θ
 2:  ══════════ ROLLOUT  (behaviour policy θ_old, no gradients) ═════════════
 3:  for each environment step t do
 4:      c_t ← encode(o_{≤t}, ℓ)                              # shared context
 5:      n ∼ Uniform{0, …, K−1}                               # ONE index, both branches
 6:
 7:      # ── action branch: K steps, noise only at k = n ────────────────────
 8:      a^{τ_0} ∼ N(0, I)
 9:      for k = 0 … K−1 do
10:          τ_k ← k/K ;   v_k ← v^a_{θ_old}(a^{τ_k}, τ_k, c_t)
11:          â_k ← a^{τ_k} + (1 − τ_k) v_k                    # clean-action estimate
12:          u_k ← v_k + ((σ^a_{τ_k})² / (2 τ_k)) â_k         # score-corrected drift
13:          if k = n then   a^{τ_k+δ} ∼ N(a^{τ_k} + δ u_k , (σ^a_{τ_k})² δ I)
14:          else            a^{τ_k+δ} ← a^{τ_k} + δ u_k      # deterministic ODE step
15:      end for
16:      store (a^{τ_n}, Δa ← a^{τ_n+δ} − a^{τ_n}, u_n)       # ONE transition only
17:
18:      # ── video branch: same K steps, noise at the SAME k = n ─────────────
19:      z^{τ_0} ∼ N(0, I)
20:      for k = 0 … K−1 do
21:          w_k ← v^z_{θ_old}(z^{τ_k}, τ_k, c_t)
22:          ẑ_k ← z^{τ_k} + (1 − τ_k) w_k
23:          m_k ← w_k + ((σ^z_{τ_k})² / (2 τ_k)) ẑ_k
24:          if k = n then   z^{τ_k+δ} ∼ N(z^{τ_k} + δ m_k , (σ^z_{τ_k})² δ I)
25:          else            z^{τ_k+δ} ← z^{τ_k} + δ m_k
26:      end for
27:      store (z^{τ_n}, Δz ← z^{τ_n+δ} − z^{τ_n}, m_n)       # subsampled to N_z tokens
28:
29:      execute the first h actions of a^{τ_K} = a¹;  observe r_t and o_{t+1}
30:  end for
31:
32:  ══════════ UPDATE ══════════════════════════════════════════════════════
33:  Â_t ← GAE(r_{1:T}, V_φ)                                   # one shared advantage
34:  for each PPO epoch do
35:      for each minibatch do
36:          # re-evaluate ONE denoising step per branch, at the STORED states
37:          u^θ_n ← drift of v^a_θ at (a^{τ_n}, τ_n, c_t)     #   1 forward pass
38:          m^θ_n ← drift of v^z_θ at (z^{τ_n}, τ_n, c_t)     #   1 forward pass
39:          log r^a ← Eq. (7) with (Δa, u^θ_n, u_n)
40:          log r^z ← Eq. (7) with (Δz, m^θ_n, m_n)          # over subsampled tokens
41:          update θ with Eq. (2) ;  update φ with the value loss
42:      end for
43:  end for
</code></pre></div></div>

<h3 id="36-notation">3.6 Notation</h3>

<table>
  <thead>
    <tr>
      <th>Symbol</th>
      <th>Meaning</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>$t$</td>
      <td>environment time step</td>
    </tr>
    <tr>
      <td>$\tau \in [0,1]$</td>
      <td>flow-matching time; $\tau = 1$ is data</td>
    </tr>
    <tr>
      <td>$K = 10$, $\delta = 1/K = 0.1$</td>
      <td>number of denoising steps, integration step size</td>
    </tr>
    <tr>
      <td>$\mathbf{c}_t$</td>
      <td>shared observation–language context</td>
    </tr>
    <tr>
      <td>$\mathbf{a}_t^{\tau}, \mathbf{a}_t$</td>
      <td>noisy action chunk at flow time $\tau$; clean action chunk</td>
    </tr>
    <tr>
      <td>$\mathbf{v}_\theta(\cdot)$</td>
      <td>predicted velocity field</td>
    </tr>
    <tr>
      <td>$\hat{\mathbf{a}}^{\tau}$</td>
      <td>one-step clean-action estimate, Sec. 2.1</td>
    </tr>
    <tr>
      <td>$\sigma_\tau = \alpha\sqrt{\tau/(1-\tau)}$</td>
      <td>SDE noise schedule of the <strong>action</strong> branch</td>
    </tr>
    <tr>
      <td>$\mathbf{Z}<em>t, \mathbf{z}</em>{t,i}$</td>
      <td>future video latents; the $i$-th latent token</td>
    </tr>
    <tr>
      <td>$N_z$</td>
      <td>number of latent tokens retained after subsampling</td>
    </tr>
    <tr>
      <td>$\sigma_z$</td>
      <td>fixed width of the Gaussian over <strong>video</strong> latents</td>
    </tr>
    <tr>
      <td>$r_t^{a}, r_t^{z}$</td>
      <td>importance ratio of the action / video branch</td>
    </tr>
    <tr>
      <td>$\hat{A}_t$</td>
      <td>(shared) GAE advantage</td>
    </tr>
    <tr>
      <td>$\epsilon_{\min}, \epsilon_{\max}$</td>
      <td>asymmetric PPO clip bounds</td>
    </tr>
  </tbody>
</table>

<p>Three notation traps are worth flagging, since all of them appear in the source draft: (i) $\sigma$ denotes <em>different</em> quantities in the two branches — a $\tau$-dependent SDE schedule in Eq. (5) and a fixed latent width in Eq. (10) — hence $\sigma_\tau$ versus $\sigma_z$; (ii) $A$ is used both for advantage and for action, hence advantage $\hat{A}<em>t$ (decorated) versus action $\mathbf{a}$ (lowercase bold); (iii) $\delta = 0.1$ is the *integration step size* fixed by $K = 10$, not a noise level — the noise scales are $\sigma</em>\tau$ and $\sigma_z$.</p>

<hr />

<h2 id="4-experiments">4. Experiments</h2>

<h3 id="41-setup">4.1 Setup</h3>

<p><strong>Base policy.</strong> Fast-WAM [7], whose action branch is a flow-matching action expert and whose video branch is a video DiT; we RL-fine-tune from the supervised checkpoint without any additional embodied pretraining.</p>

<p><strong>Benchmark.</strong> LIBERO [13], a lifelong manipulation benchmark with four task suites (Spatial, Object, Goal, Long). We report the <strong>LIBERO-Long</strong> suite, which has the longest horizons and is therefore the most sensitive to credit assignment over time. For reference, Fast-WAM reports $98.2 / 100.0 / 97.0 / 95.2$ (avg. $97.6\%$) on the four suites under its own full training recipe [7].</p>

<p><strong>Training.</strong> PPO with the dual-branch objective of Eq. (2); global batch size 2048; noise scale $\sigma = 0.1$; 1k RL training steps; evaluation every 10 steps. Advantages are computed with GAE [11] from binary task-success returns. Infrastructure is built on <strong>RLinf</strong> [14].</p>

<p><strong>Denoising budget.</strong> We set the number of flow-matching denoising steps to $K = 10$ (hence $\delta = 0.1$), whereas the LIBERO-Long numbers reported in [7] were obtained with $K = 20$. This is a deliberate trade-off: rollout dominates wall-clock time in on-policy RL, and halving $K$ roughly halves the cost of every action sampled on the robot side. The cost is discretization error, and it is visible in the starting point — <strong>our checkpoint evaluates to $91.1\%$ rather than the reported $95.2\%$</strong>. This $4.1$-point offset is a property of the <em>shared</em> starting checkpoint, so it applies identically to both curves in Fig. 1 and does not affect the comparison between them. We did not re-run the $K = 20$ configuration under our own protocol, so the residual part of the gap may also reflect the checkpoint itself, the evaluation protocol, or random seeds.</p>

<p><strong>Baseline.</strong> The same Fast-WAM checkpoint fine-tuned with Flow-SDE [5] applied to the action branch only — i.e. Eq. (2) restricted to $k \in {a}$, with the same $K = 10$ rollout. This isolates the contribution of the video branch.</p>

<h3 id="42-results">4.2 Results</h3>

<div align="center">
<img src="/wam_rl/assets/fig1_eval_success.png" alt="LIBERO-Long success rate versus RL training steps" width="760" />
</div>

<p><strong>Figure 1.</strong> LIBERO-Long success rate versus RL training steps (mean over the evaluation set; one evaluation every 10 steps). <strong>Red:</strong> Fast-WAM + dual-branch RL (ours). <strong>Blue:</strong> Fast-WAM + Flow-SDE on the action branch only (baseline). Both runs start from the same supervised checkpoint ($91.1\%$).</p>

<table>
  <thead>
    <tr>
      <th>Method</th>
      <th>$K$</th>
      <th>Init.</th>
      <th>Final</th>
      <th>Last-100 mean</th>
      <th>Peak</th>
      <th>$\Delta$ vs. baseline</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Fast-WAM (SFT checkpoint, ours)</td>
      <td>$10$</td>
      <td>$91.1\%$</td>
      <td>—</td>
      <td>—</td>
      <td>—</td>
      <td>—</td>
    </tr>
    <tr>
      <td>+ Flow-SDE (action branch only)</td>
      <td>$10$</td>
      <td>$91.1\%$</td>
      <td>$94.5\%$</td>
      <td>$94.4\%$</td>
      <td>$95.1\%$</td>
      <td>—</td>
    </tr>
    <tr>
      <td>+ <strong>Dual-branch RL (ours)</strong></td>
      <td>$10$</td>
      <td>$91.1\%$</td>
      <td>$\mathbf{96.0}\%$</td>
      <td>$\mathbf{96.0}\%$</td>
      <td>$\mathbf{96.5}\%$</td>
      <td>$\mathbf{+1.5}$</td>
    </tr>
  </tbody>
</table>

<p><strong>Table 1.</strong> LIBERO-Long success rate. “Final” is the last evaluation at step 1000; “last-100 mean” averages the final ten evaluations to remove evaluation noise. The last row is copied from [7] for reference only: it uses a different denoising budget and protocol, so it is not a like-for-like comparison with the three rows above it.</p>

<h3 id="43-analysis">4.3 Analysis</h3>

<p><strong>The margin opens early and persists.</strong> The two curves separate within the first $\sim 200$ steps and the gap never closes; the peak-to-peak advantage reaches $2.6$ points and settles at $+1.5$ points. This is consistent with the intended mechanism: the video branch starts receiving return signal immediately, so the shared representation is reshaped from the beginning of RL rather than only through the action head.</p>

<p><strong>A higher plateau, not just faster climbing.</strong> The baseline saturates around $94.5\%$ and its last-100 mean ($94.4\%$) is <em>below</em> its peak ($95.1\%$), i.e. it mildly over-optimizes and drifts back. Ours keeps improving and its last-100 mean matches its final value, suggesting that the latent-space trust region of Eq. (10) also acts as a stabilizer for the joint update.</p>

<p><strong>RL repays the discretization error, and then some.</strong> Recall that we deliberately pay $4.1$ points up front by halving the denoising budget ($95.2\% \to 91.1\%$). After 1k RL steps, ours reaches $96.0\%$ at $K = 10$ — <em>above</em> the $95.2\%$ that [7] reports at $K = 20$, using half the sampling compute per action. In other words, RL does not merely close the gap that coarse sampling opened; it more than covers it. A plausible mechanism is that optimizing the return under SDE sampling implicitly <em>straightens</em> the flow trajectories [10]: the policy learns to place coarse, few-step integration endpoints inside high-return regions, effectively absorbing the truncation error instead of suffering it. If true, this would make cheap-sampled RL a more attractive operating point than expensive-sampled SFT — but the comparison crosses protocols (checkpoint, evaluation setup, and $K$ all differ), so we treat it as a suggestive observation rather than a result. Confirming it requires running both $K = 10$ and $K = 20$ RL curves from the same checkpoint.</p>

<p><strong>What we have not yet established.</strong> We do not yet know whether the video branch improves because the <em>imagined futures</em> become more accurate, or because RL on the latent chain acts as a representation regularizer. Reporting the prediction quality of imagined futures (e.g. FVD / LPIPS at fixed rollout steps) alongside success rate would separate these hypotheses, and is the most informative next experiment.</p>

<hr />

<h2 id="5-discussion-and-limitations">5. Discussion and Limitations</h2>

<p><strong>Approximations we make.</strong> (i) The branch factorization of Eq. (1) ignores the interaction between branches through shared attention; the true joint ratio is not the product of the marginals. (ii) The video-branch likelihood of Eq. (8) is an isotropic Gaussian with a <em>fixed</em> width — a strong assumption for a high-dimensional latent whose true posterior is neither isotropic nor homoscedastic. (iii) The mixed rollout of Sec. 3.4 estimates a step-averaged objective from a single sample, which adds variance that we have not quantified.</p>

<p><strong>Sensitivity.</strong> The video branch is governed by a single hyperparameter $\sigma_z$ that interpolates between a saturated ratio (too large) and a hypersensitive one (too small). We report a single setting here; a sweep is needed before claiming the method is robust.</p>

<p><strong>Denoising budget.</strong> Every number in this post is measured at $K = 10$. Two consequences are worth separating. For the action branch, a shorter chain means the ratio is a product of fewer terms, which <em>reduces</em> the variance of the likelihood estimate. For the mixed rollout of Sec. 3.4, however, a single sampled index $n$ now stands for one tenth of the chain rather than one twentieth, so its estimate of the step-averaged objective is correspondingly noisier. We have quantified neither effect, and we have not repeated any experiment at $K = 20$; the trade-off between sampling cost and RL signal quality is therefore unmeasured.</p>

<p><strong>Scope.</strong> Results are limited to LIBERO-Long with a single base policy, a single denoising budget, and a single seed. Robustness under distribution shift — the setting in which a world model should pay off most — is untested.</p>

<p><strong>Next steps.</strong> I am running LIBERO-Plus [15], a robustness suite built on LIBERO that perturbs object layout, camera pose, language, lighting, background and sensor noise, where we expect the return-conditioned video branch to help more than on the in-distribution suites. We are also releasing training and inference code.</p>

<hr />

<h2 id="acknowledgments">Acknowledgments</h2>

<p>We build on <strong>RLinf</strong> [14] for large-scale RL infrastructure, and thank its authors for prior work.</p>

<hr />

<h2 id="references">References</h2>

<ol>
  <li>A. Brohan et al. <strong>RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control.</strong> arXiv:2307.15818, 2023.</li>
  <li>M. J. Kim, K. Pertsch, S. Karamcheti et al. <strong>OpenVLA: An Open-Source Vision-Language-Action Model.</strong> arXiv:2406.09246, 2024.</li>
  <li>K. Black, N. Brown, D. Driess et al. <strong>$\pi_0$: A Vision-Language-Action Flow Model for General Robot Control.</strong> arXiv:2410.24164, 2024.</li>
  <li>H. Li, Y. Zuo, J. Yu et al. <strong>SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning.</strong> arXiv:2509.09674, 2025.</li>
  <li>K. Chen, Z. Liu, T. Zhang et al. <strong>$\pi$RL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models.</strong> arXiv:2510.25889, 2025. <em>(Flow-Noise and Flow-SDE are the two algorithms introduced in this paper.)</em></li>
  <li>Q. Shen, S. Zhang, Y. Liao et al. <strong>World Action Models: A Survey — Dream Less, Act More.</strong> arXiv:2606.20781, 2026.</li>
  <li>T. Yuan, Z. Dong, Y. Liu, H. Zhao. <strong>Fast-WAM: Do World Action Models Need Test-time Future Imagination?</strong> arXiv:2603.16666, 2026.</li>
  <li>H. Chen, J. Liu, Z. Yan et al. <strong>LaST-R1: Reinforcing Action via Adaptive Physical Latent Reasoning for VLA Models.</strong> arXiv:2604.28192, 2026.</li>
  <li>Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, M. Le. <strong>Flow Matching for Generative Modeling.</strong> ICLR 2023, arXiv:2210.02747.</li>
  <li>X. Liu, C. Gong, Q. Liu. <strong>Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow.</strong> NeurIPS 2022, arXiv:2209.03003.</li>
  <li>J. Schulman, P. Moritz, S. Levine, M. I. Jordan, P. Abbeel. <strong>High-Dimensional Continuous Control Using Generalized Advantage Estimation.</strong> ICLR 2016, arXiv:1506.02438.</li>
  <li>J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov. <strong>Proximal Policy Optimization Algorithms.</strong> arXiv:1707.06347, 2017.</li>
  <li>B. Liu, Y. Zhu, C. Gao et al. <strong>LIBERO: Benchmarking Knowledge Transfer in Lifelong Robot Learning.</strong> NeurIPS 2023 (Datasets and Benchmarks), arXiv:2306.08268.</li>
  <li>C. Yu, Y. Wang, Z. Guo et al. <strong>RLinf: Flexible and Efficient Large-scale Reinforcement Learning via Macro-to-Micro Flow Transformation.</strong> arXiv:2509.15965, 2025.</li>
  <li><strong>LIBERO-plus.</strong> Robustness evaluation suite for vision-language-action policies built on LIBERO (community project; no canonical paper at the time of writing). <a href="https://github.com/sylvestf/LIBERO-plus">https://github.com/sylvestf/LIBERO-plus</a></li>
</ol>

<hr />

<h2 id="cite-this-post">Cite this post</h2>

<div class="language-bibtex highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">@misc</span><span class="p">{</span><span class="nl">dualbranchrl-wam-2026</span><span class="p">,</span>
  <span class="na">title</span>  <span class="p">=</span> <span class="s">{Dual-Branch Reinforcement Learning for World-Action Models}</span><span class="p">,</span>
  <span class="na">author</span> <span class="p">=</span> <span class="s">{[[fill: Trayfour Yu]]}</span><span class="p">,</span>
  <span class="na">year</span>   <span class="p">=</span> <span class="s">{2026}</span><span class="p">,</span>
  <span class="na">howpublished</span> <span class="p">=</span> <span class="s">{\url{[[fill: https://trayfouryu.github.io/wam_rl/]]}}</span><span class="p">,</span>
  <span class="na">note</span>   <span class="p">=</span> <span class="s">{Research blog post}</span>
<span class="p">}</span>
</code></pre></div></div>

<hr />

<!--
================================================================================
AUTHOR NOTES — delete this comment block before publishing.
================================================================================

1) FIGURE 1 NUMBERS (read off the vector data of eval_success_comparison.pdf)
   Ours (red):      init 0.9111 -> final 0.9604, last-100 mean 0.9595, peak 0.9647
   Baseline (blue): init 0.9111 -> final 0.9454, last-100 mean 0.9444, peak 0.9513
   Gap: +1.5 pts (final / last-100 mean), max 2.6 pts
   x-axis spans 0-1000 steps (the "1000" tick label is cropped); 101 evaluations
   => one eval every 10 steps. y-axis ticks are 0.90-0.97.
   Colour mapping was verified from the legend handles: red = "(ours)" row.
   PLEASE SANITY-CHECK these against your training logs before posting.

2) SIGMA = 0.1 — AMBIGUOUS IN THE DRAFT
   The draft says only "sigma 为 0.1". The post currently assumes BOTH
   alpha = 0.1 (action-branch SDE schedule, Eq. 8) and sigma_z = 0.1
   (video-latent Gaussian, Eq. 13). If 0.1 refers to only one of them, edit
   Sec. 4.1. Also note: if you use a CONSTANT sigma_tau instead of piRL's
   alpha*sqrt(tau/(1-tau)) schedule, sigma_tau^2/(2 tau) diverges at tau -> 0
   and you must clamp tau >= tau_min; the post currently assumes piRL's schedule.

3) BASE CHECKPOINT DISCREPANCY — RESOLVED, BUT PLEASE RE-CHECK THE STEP COUNT
   Sec. 4.1 now explains the 91.1% vs 95.2% gap as the denoising budget
   (K=10 here vs K=20 in [7]), and Sec. 4.3 turns it into a selling point
   (10-step RL reaches 96.0% > 20-step SFT's 95.2%).
   *** VERIFY BEFORE POSTING ***: when I fetched Fast-WAM's HTML (v2, Sec. 4.1)
   it says "During inference, we use 10 denoising steps with classifier-free
   guidance (CFG) scale set to 1.0" — i.e. 10, NOT 20. That statement may refer
   to a different experiment (real robot / RoboTwin) or to the open-source
   default, while LIBERO was evaluated at 20; or the 20 may come from the code
   config rather than the paper. Since [7] is a preprint that is still being
   revised, please double-check the LIBERO evaluation step count in the current
   version. If the paper's LIBERO number is actually at K=10, DELETE the
   denoising-budget explanation in Sec. 4.1, the last row of Table 1, the
   "RL repays the discretization error" paragraph, and the "Denoising budget"
   limitation, and instead attribute the gap to checkpoint / eval protocol /
   random seeds only.

4) VIDEO-BRANCH RATIO: TWO FORMS
   Eq. (10) (single term, = LaST-R1's form) is only valid if the stored latents
   were produced deterministically, so that z^{theta_old} = z^{old}. If you
   inject noise at step n in the mixed ODE-SDE rollout, use Eq. (9) (two terms)
   so that r = 1 at the start of each PPO epoch, and cache z^{theta_old}.
   Decide which one your implementation actually does and delete the other.

5) "Noise-Flow" — NAME CHECK
   No standalone paper named "Noise-Flow" was found. The method you refer to is
   almost certainly Flow-Noise, one of the two algorithms inside piRL [5].
   The post cites piRL for both Flow-Noise and Flow-SDE and says so explicitly
   in Ref. 5. If you meant a different paper, fix Ref. 5.

6) LIBERO-PLUS
   No canonical paper exists for LIBERO-Plus; it is cited as a community
   benchmark project (Ref. 15). Update if the authors publish a paper.

7) PLACEHOLDERS TO FILL: grep for "[[fill:" — there are two, both in the
   "Cite this post" BibTeX block (author name and page URL).
   Also consider adding: number of seeds, per-suite results for Spatial / Object
   / Goal, sigma_z sensitivity sweep, and video-prediction quality (FVD/LPIPS).

8) PUBLISHING (Jekyll / GitHub Pages)
   - Copy _posts/2026-09-02-dual-branch-rl-for-wam.md into <repo>/_posts/
   - Copy assets/fig1_eval_success.{png,svg} into <repo>/assets/
   - Make sure MathJax is not double-processed. With kramdown (the GitHub Pages
     default), add this to _config.yml:
         markdown: kramdown
         kramdown:
           math_engine: null
     Otherwise kramdown rewrites $$...$$ into <script type="math/tex"> tags,
     which MathJax v3 (loaded in this post) does NOT process, and every formula
     renders as raw LaTeX. If your theme already ships MathJax (e.g.
     minimal-mistakes with `mathjax: true`), delete the two <script> tags at the
     top of this file instead.
   - The image uses Liquid (), which is correct on
     Jekyll. For a plain Markdown renderer, use: assets/fig1_eval_success.png
   - assets/references.bib is provided for jekyll-scholar / pandoc users.
================================================================================
-->]]></content><author><name></name></author><category term="embodied-ai" /><category term="reinforcement-learning" /><category term="VLA" /><category term="world-action-model" /><category term="flow-matching" /><category term="PPO" /><category term="LIBERO" /><summary type="html"><![CDATA[]]></summary></entry></feed>