In our previous posts - Easy FunctionGemma finetuning with Tunix on Google TPUs and Autonomous LLM post-training with Tunix on TPUs - we showed how Google Tunix streamlines Supervised Fine-Tuning (SFT) and single-turn Group Relative Policy Optimization (GRPO) on Cloud TPUs. Real-world AI agents, however, rarely finish their job in a single turn. They have to reason over partial observations, invoke tools, observe environment feedback, and adapt their strategy across multiple steps. So in this post, we will walk through how to train a multi-turn, tool-using LLM agent using Tunix's agentic reinforcement learning pipeline on Cloud TPU v6e-8. You can find the complete code in this GitHub repository.
We will revisit the Plane Strike board game from an earlier TensorFlow/JAX reinforcement learning blog, replacing the custom MLP policy with Gemma 4 E2B in thinking mode on a single Cloud TPU v6e-8 host. Plane Strike is played on a 6x6 grid (A1 to F6) where one 8-cell plane is hidden in one of 4 orientations. Each turn, the agent sees the result of its previous strike and the raw visible board (. untried, X hit, O miss), thinks through candidate placements, and calls the strike(cell="...") tool to strike one cell. The game ends when all 8 plane cells are hit, or after 23 strikes.
Although google/gemma-4-E2B-it has decent spatial reasoning and tool calling capabilities, out of the box, it wins only 9.4% of games on held-out boards (averaging 22.44 strikes). After 100 steps of agentic GRPO with Tunix, its win rate reaches 95% while cutting the average game length to ~16 strikes.
We set up the multi-turn RL pipeline in four modular components:
BaseTaskEnv in env.py)BaseTool and Gemma4ChatTemplateParser in strike_tool.py)ConversationAgentBase in agent.py)TrajectoryCollectEngine and GRPOLearner in trajectory_utils.py and train_planestrike.py)Let's look at the key steps for each piece.
env.py)In env.py, we subclass Tunix's BaseTaskEnv to implement PlaneStrikeEnv. Each environment instance receives a task dictionary with a seed that fixes the hidden plane position so all 8 rollouts in a GRPO group play the exact same board:
class PlaneStrikeEnv(BaseTaskEnv):
"""One Plane Strike game. `entry["seed"]` fixes the hidden plane position."""
def __init__(
self,
entry: dict[str, Any],
group_id: int | None = None,
pair_index: int | None = None,
max_steps: int = MAX_STEPS,
**kwargs,
):
entry = {k: _unwrap(v) for k, v in entry.items()}
self.seed = int(entry.get("seed", 0))
self.split = str(entry.get("split", "train"))
super().__init__(task=dict(entry), max_steps=max_steps, **kwargs)
self.extra_kwargs["group_id"] = group_id
self.extra_kwargs["pair_index"] = pair_index
self._reset_board()
On each turn, _step_impl evaluates the struck cell index against the hidden board, updates the visible grid, computes the step reward (summed into the trajectory-level reward over the episode), and returns EnvStepResult(observation, reward, done, info).
For standard multi-tool agents, tunix.rl.agentic also provides ready-to-use ToolEnvironment, ToolAgent and ToolManager abstractions out of the box so we don't have to write custom message/observation plumbing unless we need custom turn control.
strike_tool.py)In Gemma 4, tool schemas are declared in the system prompt inside <|tool>declaration:...<tool|>, and a model turn that calls a tool ends with:
<|channel>thought
...model reasoning...
<channel|><|tool_call>call:strike{cell:<|"|>C5<|"|>}<tool_call|><|tool_response>
<|tool_response> is one of Gemma 4's native stop tokens: the model pauses there and waits for response:strike{value:<|"|>...<|"|>}<tool_response|>, then continues generating its next thought and tool call within the same assistant turn.
In strike_tool.py, we define StrikeTool(BaseTool) and subclass Tunix's Gemma4ChatTemplateParser to append the tool declaration to the system prompt and render "tool" messages as in-turn responses:
class Gemma4ToolChatTemplateParser(parser_lib.Gemma4ChatTemplateParser):
"""Gemma4ChatTemplateParser plus native tool declarations and in-turn tool
responses."""
def __init__(self, tokenizer, tools: list[base_tool.BaseTool], enable_thinking: bool = True):
super().__init__(tokenizer, enable_thinking=enable_thinking)
self.tool_declarations = render_tool_declarations(tokenizer, tools)
self.tool_response_id = tokenizer.convert_tokens_to_ids(TOOL_RESPONSE_START)
def _parse_system(self, content: str) -> str:
return self.tokens.system_token + content.strip() + self.tool_declarations + self.tokens.eot_token
def _parse_tool(self, content: str) -> str:
return f"response:{TOOL_NAME}{{value:{_STR}{content}{_STR}}}<tool_response|>"
def parse(self, messages, add_generation_prompt: bool = False, is_first_msg: bool = False) -> str:
if messages and messages[-1]["role"] == "tool":
add_generation_prompt = False # The model continues its own turn.
return super().parse(messages, add_generation_prompt=add_generation_prompt,
is_first_msg=is_first_msg)
agent.py)In agent.py, PlaneStrikeAgent(ConversationAgentBase) translates between environment observations and model messages.
When the model finishes a turn, tool_strike ignores everything inside <|channel>thought...<channel|> and parses the final answer after <channel|>; only a final answer with a single valid strike call counts as a valid board coordinate (otherwise INVALID_ACTION is returned). If the model stopped at <|tool_response>, the next board observation is appended with role="tool" so the model continues its turn:
class PlaneStrikeAgent(base_agent.ConversationAgentBase):
def _observation_to_messages(
self, observation: Any, reward: float, done: bool, info: dict[str, Any]
) -> None:
if not done:
role = "tool" if self._awaiting_tool_response else "user"
self._messages.append({"role": role, "content": str(observation)})
def update_from_model(self, response: str, **kwargs) -> agent_types.Action:
action = tool_strike(response, self.tool)
answer = final_answer(response)
self.tool_calls_per_turn.append(
0 if answer is None else len(strike_tool.parse_tool_calls(answer)))
self._awaiting_tool_response = response.rstrip().endswith(strike_tool.TOOL_RESPONSE_START)
self._messages.append({"role": "assistant", "content": response})
self._trajectory.steps.append(self.cur_step)
self.cur_step.action = action
self.cur_step.model_response = response
return agent_types.Action(action=action)
train_planestrike.py)In train_planestrike.py, we configure Tunix's asynchronous vllm rollout engine, RLEngine, and GRPOLearner:
rollout_config = base_rollout.RolloutConfig(
max_prompt_length=args.max_prompt_length,
max_tokens_to_generate=args.max_response_length,
kv_cache_size=args.max_prompt_length + args.max_response_length + 256,
temperature=args.temperature,
top_p=1.0,
top_k=0,
return_logprobs=True,
eos_tokens=[tokenizer.eos_token_id, tokenizer.convert_tokens_to_ids("<turn|>"),
tokenizer.convert_tokens_to_ids(strike_tool.TOOL_RESPONSE_START)],
rollout_vllm_model_version=MODEL_VERSION,
rollout_vllm_hbm_utilization=args.vllm_hbm_utilization,
rollout_vllm_tpu_backend_type="jax",
rollout_vllm_server_mode=True,
rollout_vllm_kwargs={
"enable_prefix_caching": True,
"dtype": "bfloat16",
"hf_overrides": {
"final_logit_softcapping": 30.0,
"text_config": {"final_logit_softcapping": 30.0},
"architectures": ["Gemma4ForCausalLM"],
},
},
rollout_vllm_sampling_kwargs={"skip_special_tokens": False},
)
Multi-turn agentic RL is often bottlenecked by variable-length rollout generation, so we set rollout_vllm_server_mode to true to enable continuous asynchronous multi-turn rollouts via vllm-tpu and enable_prefix_caching to true to true to cache the growing multi-turn conversation history. Since we are using a v6e-8 single host, the actor model, the reference model, and the rollout workers share the 8 chips.
We then pass our actor, tokenizer, ClusterConfig, GRPOConfig (loss_algo="gspo-token", advantage_estimator="rloo", exact_token_continuity=True to preserve exact token IDs across multi-turn rollouts without retokenization drift), agent, and environment into RLEngine and PlaneStrikeGRPOLearner:
rl_engine = rl_engine_lib.RLEngine(
actor=actor, reference=reference, tokenizer=tokenizer, cluster_config=cluster_config)
learner = PlaneStrikeGRPOLearner(
rl_engine=rl_engine,
agent_class=agent_lib.PlaneStrikeAgent,
agent_kwargs={},
env_class=env_lib.PlaneStrikeEnv,
env_kwargs={"max_steps": args.max_steps_per_game},
algo_config=grpo_config,
chat_parser=chat_parser,
metric_fns=[metric_fn],
)
learner.train(train_dataset, eval_dataset=eval_dataset)
All of these enable us to launch the 100-step training job with:
python train_planestrike.py \
--run_dir runs/grpo_6x6_tool_thinking_100 \
--learning_rate 2e-6 \
--lr_end 2e-7
Over 100 steps (8 boards x 8 games per step), the agent learns both reliable tool calling and spatial deduction:
game/train/tool_call_rate quickly reaches ~100%, while game/train/invalid drops to ~0 and game/train/repeats falls from ~0.55 to ~0.15. Because all strikes in a game currently share a single trajectory-level reward, further improvements can be achieved with finer per-turn credit assignment.With Tunix's tunix.rl.agentic abstractions, vllm-tpu, and GRPOLearner, multi-turn, tool-using agentic RL on Cloud TPUs is easy and efficient. Check out the planestrike-gemma4-rl GitHub repository and read the Tunix documentation to train your own agentic model with Tunix and Google TPUs.