AgentFly is an extensible framework for building LLM agents with reinforcement learning. It supports multi-turn training by adapting traditional RL methods with token-level masking. It features a decorator-based interface for defining tools and reward functions, enabling seamless extension and ease of use. To support high-throughput training, it implemented asynchronous execution of tool calls and reward computations, and design a centralized resource management system for scalable environment coordination. A suite of prebuilt tools and environments are provided.
4/2026 SWE and SimuScene: Added SWE-Bench style (R2E-Gym) tasks and SimuScene training.
3/2026 Resource System: Added a resource system for better environment and container management, and integrated SWE like tasks training.
12/2025 Method-Based Tool: Support using @tool for a class method
12/2025 Verl Update: Updated verl to 0.6.x version.
08/2025 Multi-Modal (Vision) Agent Training Support: Thanks to the powerful template system, AgentFly now supports training vision-language agents! 🎉 Train agents that can see and understand visual content, including GUI automation and image-based QA. See our predefined training examples for ready-to-use scripts.
08/2025 Chat Template System: A flexible framework for creating conversation templates with multi-model support, vision capabilities, and tool integration. It now lives in its own library, chat-bricks. Learn more →
AgentFly uses uv and requires Python 3.12 on Linux.
Option 1: Install from PyPI. The package is named agentfly-rl (the agentfly package on PyPI is an unrelated project); it is imported as agentfly.
uv venv --python 3.12 && source .venv/bin/activate
uv pip install agentfly-rl # agents, tools, rewards, and rolloutsOption 2: One-line installation from source, recommended for RL training. It installs uv if missing and builds a project .venv from the committed uv.lock, so the training stack matches the versions we test with.
bash install.shOption 3: Customized installation
Please refer to installation.md for customized installation, including the [verl] training extra.
| Task | Model | Report | Status |
|---|---|---|---|
| SearchR1 | Qwen2.5 | report | ✅ |
| WebShop | Qwen2.5 | report | ✅ |
| ScienceWorld | Qwen3-4B-Instruct | report | ✅ |
| SWE | Qwen3-32B | report | ✅ (on going) |
| SimuScene | SFT DeepSeek-R1-Distill-Qwen | report | ✅ |
| ALFWorld | Qwen2.5-3B-Instruct | report link — to add | ✅ |
# Really small example to build an agent and run
import asyncio
from agentfly.agents import HFAgent
from agentfly.tools import calculator
async def main():
messages = [{"role": "user", "content": "What is the result of 1 + 1?"}]
agent = HFAgent(
model_name_or_path="Qwen/Qwen2.5-3B-Instruct",
tools=[calculator],
template="qwen2.5",
# gpu_memory_utilization: share of GPU memory vLLM reserves (default 0.92)
backend_config={"backend": "async_vllm", "gpu_memory_utilization": 0.5},
)
result = await agent.run(
messages=messages,
max_turns=3,
num_chains=1,
)
for traj in result.trajectories:
print(agent.extract_final_response(traj.segments[-1].messages), traj.reward)
if __name__ == "__main__": # required: vLLM starts worker processes that re-import this file
asyncio.run(main())template names a chat template from chat-bricks
(e.g. qwen2.5, qwen3, llama-3.2; AgentFly adds a few more); list them all with
python -c "import agentfly.agents; from chat_bricks.registry import TEMPLATES; print(sorted(TEMPLATES))".
See Build an Agent for reading, batching and saving results.
To support algorithms like GRPO, Reinforce++, we design multi-chain inference, enabling agents to solve one task with multiple paths at the same time. We build RL computation and update LLMs in multi-turn manner by applying token masks. The training is based on verl.
Define tools and rewards, which can be used directly by agents.
@tool(name=...)
def customized_tool(...):
...
@reward(name=...)
def customized_reward(...):
...
agent = HFAgent(
model_name_or_path=model_name,
tools=[customized_tool],
reward_fn=customized_reward,
backend_config={"backend": "async_vllm"},
)Decoupled agent and training module. Simply customize your own agent, which can directly be applied to training.
Suppose you are in a compute node (with 8 GPUs). We have prepared training scripts for different tasks and tools in examples/train_scripts/. The scripts can download prepared datasets and run training.
Run RL training of code_interpreter:
bash examples/train_scripts/train_example.shTo customize your own training, you need to prepare: 1. Datasets. 2. Define or use existing tools. 3. Define or use existing rewards. 3. Define your own agents or use an existing type of agent.
Data should be a json file, which contain a list of dicts with the following keys:
[
{
"question": ...
"optional_field1": ...
"optional_field2": ...
...
}
]During training, question will be used to format the input messages, while other fields can be used in reward function. An example message that are put into the agent looks like this:
{
"messages": [
{"role": "user", "content": [{"type": "text", "text": question}]}
]
"optional_field1": ...
"optional_field2": ...
...
}You can use any existing tool, which is in documentation, or define a tool by decorating it with @tool. The output should eighther be a string, or a dictionary containing observation as a key.
@tool(name="customized_tool")
def customized_tool(arg1, arg2):
# tool logic hereDefine your reward function or use an existing one. The reward function can accept final_response and trajectory as arguments, which are the agent's final response and the whole trajectory. Other fields will also be given if you defined them in the dataset. To use them, simply add these fields as arguments of the reward function.
@reward(name="customized_reward")
def customized_reward(final_response, trajectory, optional_field1, optional_field2):
# calculate reward
...For tools and rewards that share an environment (a game, a sandbox, a browser), see Build a Custom Environment.
You can use existing agents, or customize an agent. A custom agent inherits BaseAgent, which handles tool calling and rollouts, and overrides parse, which turns raw model outputs into assistant messages with their tool calls. Refer to the Agents API reference for more details.
from agentfly.agents import BaseAgent
class CustomizedAgent(BaseAgent):
def parse(self, responses, context=None, **kwargs):
# One assistant message per response; put requested tool calls in "tool_calls".
return [
{"role": "assistant", "content": [{"type": "text", "text": r}], "tool_calls": []}
for r in responses
]-
The following shows an example of WebShop agent.
-
What does the training look like. During training, the resource system will dynamically allocate environments.
-
Monitoring training on WANDB. Items include number of turns for each step, numer of tool calls, allocated environments.
demo.mp4
If you used our code or find it helpful, please cite:
@misc{wang2025agentfly,
title={AgentFly: Extensible and Scalable Reinforcement Learning for LM Agents},
author={Renxi Wang and Rifo Ahmad Genadi and Bilal El Bouardi and Yongxin Wang and Fajri Koto and Zhengzhong Liu and Timothy Baldwin and Haonan Li},
year={2025},
eprint={2507.14897},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2507.14897},
}