Multi-Turn Agentic Reinforcement Learning with Tunix and Gemma 4 on Google TPUs

NOV. 19, 2026
Wei Wei Developer Advocate

In our previous posts - Easy FunctionGemma finetuning with Tunix on Google TPUs and Autonomous LLM post-training with Tunix on TPUs - we showed how Google Tunix streamlines Supervised Fine-Tuning (SFT) and single-turn Group Relative Policy Optimization (GRPO) on Cloud TPUs. Real-world AI agents, however, rarely finish their job in a single turn. They have to reason over partial observations, invoke tools, observe environment feedback, and adapt their strategy across multiple steps. So in this post, we will walk through how to train a multi-turn, tool-using LLM agent using Tunix's agentic reinforcement learning pipeline on Cloud TPU v6e-8. You can find the complete code in this GitHub repository.

The Game Setup

We will revisit the Plane Strike board game from an earlier TensorFlow/JAX reinforcement learning blog, replacing the custom MLP policy with Gemma 4 E2B in thinking mode on a single Cloud TPU v6e-8 host. Plane Strike is played on a 6x6 grid (A1 to F6) where one 8-cell plane is hidden in one of 4 orientations. Each turn, the agent sees the result of its previous strike and the raw visible board (. untried, X hit, O miss), thinks through candidate placements, and calls the strike(cell="...") tool to strike one cell. The game ends when all 8 plane cells are hit, or after 23 strikes.

Although google/gemma-4-E2B-it has decent spatial reasoning and tool calling capabilities, out of the box, it wins only 9.4% of games on held-out boards (averaging 22.44 strikes). After 100 steps of agentic GRPO with Tunix, its win rate reaches 95% while cutting the average game length to ~16 strikes.

Building the Agentic RL Pipeline in Tunix

We set up the multi-turn RL pipeline in four modular components:

  1. Task Environment (BaseTaskEnv in env.py)
  2. Tool and Chat Template Parser (BaseTool and Gemma4ChatTemplateParser in strike_tool.py)
  3. Conversation Agent (ConversationAgentBase in agent.py)
  4. Trajectory Collection and Agentic GRPO Learner (TrajectoryCollectEngine and GRPOLearner in trajectory_utils.py and train_planestrike.py)

Let's look at the key steps for each piece.

Step 1: Defining the Environment (env.py)

In env.py, we subclass Tunix's BaseTaskEnv to implement PlaneStrikeEnv. Each environment instance receives a task dictionary with a seed that fixes the hidden plane position so all 8 rollouts in a GRPO group play the exact same board:

class PlaneStrikeEnv(BaseTaskEnv):
  """One Plane Strike game. `entry["seed"]` fixes the hidden plane position."""

  def __init__(
      self,
      entry: dict[str, Any],
      group_id: int | None = None,
      pair_index: int | None = None,
      max_steps: int = MAX_STEPS,
      **kwargs,
  ):
    entry = {k: _unwrap(v) for k, v in entry.items()}
    self.seed = int(entry.get("seed", 0))
    self.split = str(entry.get("split", "train"))
    super().__init__(task=dict(entry), max_steps=max_steps, **kwargs)
    self.extra_kwargs["group_id"] = group_id
    self.extra_kwargs["pair_index"] = pair_index
    self._reset_board()
Python

On each turn, _step_impl evaluates the struck cell index against the hidden board, updates the visible grid, computes the step reward (summed into the trajectory-level reward over the episode), and returns EnvStepResult(observation, reward, done, info).

For standard multi-tool agents, tunix.rl.agentic also provides ready-to-use ToolEnvironment, ToolAgent and ToolManager abstractions out of the box so we don't have to write custom message/observation plumbing unless we need custom turn control.

Step 2: Native Gemma 4 Tool Calling (strike_tool.py)

In Gemma 4, tool schemas are declared in the system prompt inside <|tool>declaration:...<tool|>, and a model turn that calls a tool ends with:

<|channel>thought
...model reasoning...
<channel|><|tool_call>call:strike{cell:<|"|>C5<|"|>}<tool_call|><|tool_response>
Plain text

<|tool_response> is one of Gemma 4's native stop tokens: the model pauses there and waits for response:strike{value:<|"|>...<|"|>}<tool_response|>, then continues generating its next thought and tool call within the same assistant turn.

In strike_tool.py, we define StrikeTool(BaseTool) and subclass Tunix's Gemma4ChatTemplateParser to append the tool declaration to the system prompt and render "tool" messages as in-turn responses:

class Gemma4ToolChatTemplateParser(parser_lib.Gemma4ChatTemplateParser):
  """Gemma4ChatTemplateParser plus native tool declarations and in-turn tool
  responses."""

  def __init__(self, tokenizer, tools: list[base_tool.BaseTool], enable_thinking: bool = True):
    super().__init__(tokenizer, enable_thinking=enable_thinking)
    self.tool_declarations = render_tool_declarations(tokenizer, tools)
    self.tool_response_id = tokenizer.convert_tokens_to_ids(TOOL_RESPONSE_START)

  def _parse_system(self, content: str) -> str:
    return self.tokens.system_token + content.strip() + self.tool_declarations + self.tokens.eot_token

  def _parse_tool(self, content: str) -> str:
    return f"response:{TOOL_NAME}{{value:{_STR}{content}{_STR}}}<tool_response|>"

  def parse(self, messages, add_generation_prompt: bool = False, is_first_msg: bool = False) -> str:
    if messages and messages[-1]["role"] == "tool":
      add_generation_prompt = False  # The model continues its own turn.
    return super().parse(messages, add_generation_prompt=add_generation_prompt,
                         is_first_msg=is_first_msg)
Python

Step 3: Building the Conversation Agent (agent.py)

In agent.py, PlaneStrikeAgent(ConversationAgentBase) translates between environment observations and model messages.

When the model finishes a turn, tool_strike ignores everything inside <|channel>thought...<channel|> and parses the final answer after <channel|>; only a final answer with a single valid strike call counts as a valid board coordinate (otherwise INVALID_ACTION is returned). If the model stopped at <|tool_response>, the next board observation is appended with role="tool" so the model continues its turn:

class PlaneStrikeAgent(base_agent.ConversationAgentBase):

  def _observation_to_messages(
      self, observation: Any, reward: float, done: bool, info: dict[str, Any]
  ) -> None:
    if not done:
      role = "tool" if self._awaiting_tool_response else "user"
      self._messages.append({"role": role, "content": str(observation)})

  def update_from_model(self, response: str, **kwargs) -> agent_types.Action:
    action = tool_strike(response, self.tool)
    answer = final_answer(response)
    self.tool_calls_per_turn.append(
        0 if answer is None else len(strike_tool.parse_tool_calls(answer)))
    self._awaiting_tool_response = response.rstrip().endswith(strike_tool.TOOL_RESPONSE_START)
    self._messages.append({"role": "assistant", "content": response})
    self._trajectory.steps.append(self.cur_step)
    self.cur_step.action = action
    self.cur_step.model_response = response
    return agent_types.Action(action=action)
Python

Step 4: Wiring Up Agentic GRPO (train_planestrike.py)

In train_planestrike.py, we configure Tunix's asynchronous vllm rollout engine, RLEngine, and GRPOLearner:

rollout_config = base_rollout.RolloutConfig(
    max_prompt_length=args.max_prompt_length,
    max_tokens_to_generate=args.max_response_length,
    kv_cache_size=args.max_prompt_length + args.max_response_length + 256,
    temperature=args.temperature,
    top_p=1.0,
    top_k=0,
    return_logprobs=True,
    eos_tokens=[tokenizer.eos_token_id, tokenizer.convert_tokens_to_ids("<turn|>"),
                tokenizer.convert_tokens_to_ids(strike_tool.TOOL_RESPONSE_START)],
    rollout_vllm_model_version=MODEL_VERSION,
    rollout_vllm_hbm_utilization=args.vllm_hbm_utilization,
    rollout_vllm_tpu_backend_type="jax",
    rollout_vllm_server_mode=True,
    rollout_vllm_kwargs={
        "enable_prefix_caching": True,
        "dtype": "bfloat16",
        "hf_overrides": {
            "final_logit_softcapping": 30.0,
            "text_config": {"final_logit_softcapping": 30.0},
            "architectures": ["Gemma4ForCausalLM"],
        },
    },
    rollout_vllm_sampling_kwargs={"skip_special_tokens": False},
)
Python

Multi-turn agentic RL is often bottlenecked by variable-length rollout generation, so we set rollout_vllm_server_mode to true to enable continuous asynchronous multi-turn rollouts via vllm-tpu and enable_prefix_caching to true to true to cache the growing multi-turn conversation history. Since we are using a v6e-8 single host, the actor model, the reference model, and the rollout workers share the 8 chips.

We then pass our actor, tokenizer, ClusterConfig, GRPOConfig (loss_algo="gspo-token", advantage_estimator="rloo", exact_token_continuity=True to preserve exact token IDs across multi-turn rollouts without retokenization drift), agent, and environment into RLEngine and PlaneStrikeGRPOLearner:

rl_engine = rl_engine_lib.RLEngine(
    actor=actor, reference=reference, tokenizer=tokenizer, cluster_config=cluster_config)

learner = PlaneStrikeGRPOLearner(
    rl_engine=rl_engine,
    agent_class=agent_lib.PlaneStrikeAgent,
    agent_kwargs={},
    env_class=env_lib.PlaneStrikeEnv,
    env_kwargs={"max_steps": args.max_steps_per_game},
    algo_config=grpo_config,
    chat_parser=chat_parser,
    metric_fns=[metric_fn],
)
learner.train(train_dataset, eval_dataset=eval_dataset)
Python

All of these enable us to launch the 100-step training job with:

python train_planestrike.py \
  --run_dir runs/grpo_6x6_tool_thinking_100 \
  --learning_rate 2e-6 \
  --lr_end 2e-7
Shell

Training Results

Over 100 steps (8 boards x 8 games per step), the agent learns both reliable tool calling and spatial deduction:

sample_train_curves
  • Tool calling compliance converges first: game/train/tool_call_rate quickly reaches ~100%, while game/train/invalid drops to ~0 and game/train/repeats falls from ~0.55 to ~0.15. Because all strikes in a game currently share a single trajectory-level reward, further improvements can be achieved with finer per-turn credit assignment.
  • Win rate (the ratio of the agent successfully hitting all 8 hidden cells within 23 steps) climbs from ~0.20 on initial training rollouts (9.4% on held-out evaluation boards) to ~0.95.
  • Average game length drops from ~22 down to ~16 strikes per game.

What's Next?

With Tunix's tunix.rl.agentic abstractions, vllm-tpu, and GRPOLearner, multi-turn, tool-using agentic RL on Cloud TPUs is easy and efficient. Check out the planestrike-gemma4-rl GitHub repository and read the Tunix documentation to train your own agentic model with Tunix and Google TPUs.