Skip to content

Phase 1 Roadmap #43

Open
Open

Description

@Dar-rius

Hi everyone,

Our first release 0.1.1 is out!

For this release, we focused on designing a simple, modular RL research framework architecture with minimal abstractions, while providing the core features needed to prototype new algorithms:

  • A modular training orchestrator including rollout collection, visualization (W&B, TensorBoard), observation normalization, customizable weight update functions, model checkpointing, and profiling.
  • Data structure classes to easily configure training runs and RL algorithms.
  • Abstraction classes for designing agents and environments, providing a safer interface without restricting customization.
  • An extensible tensor buffer capable of storing multiple variables.
  • PPO as the first algorithm included in the framework, with a modular design allowing researchers to modify individual parts of the algorithm according to their needs.

In the next steps, we will make the framework more modular for an immediate mode, make it more robotics-oriented by integrating Mujoco, improve the debug mode, write benchmarks and examples, and create documentation.

Feel free to contribute to the roadmap or report an issue to us.

This roadmap is a living document and will naturally evolve based on feedback and the needs of the research community.

Current Focus

Features we are actively working on right now.

  • Add an periodic determinist periodic eval with multi-seed.
  • Implement Sotf Actor Critic (SAC), check the paper
  • Add an cnn architecture to the ActorCriticAgent factory Atari #100 .
  • Add system of hyperparam sweeps for hyperparams research easier.
  • Create some benchmarks that allow for comparing zeroRL with other implementations of RL libraries
  • Add more examples based on research use cases.
  • Create a documentation with mkdocs.

Planned Features

Next in the pipeline once the current focus is stable.

Core

  • Make the debug mode more powerful to check the overall status of the training.
  • Add a system to resume optimizer, scheduler, step counters and random state
  • Allow enginers/researchers to experiments some multiagent systems and implement MAPPO.
  • Integrate natively Mujoco Mujoco Integration #64
  • Adapt the architecture to immediate mode to help researchers set up their own research architecture Make zeroRL more modular. #65 .
  • Save the NormMeanStd state for simulators such as Mujoco and Isaac sim Make zeroRL more modular. #65 .
  • Add necessary buffer and agent architectures to support algorithms that do not use an actor-critic architecture.
  • Automatically resize the buffer
  • Add tools to help engineers and researchers understand what is happening inside their agents, buffers, training loop and other.

Documentation

Implement basic RL algorithms

Performance and stability

  • Add tests to verify that RL algorithm implementations are correct, robust and performant.

Compatibility and portability

  • Add support for Metal-compatible PyTorch backend, check the Metal Support #59
  • Support for most environments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

FeaturesNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions