Researchers at Meta FAIR and the National University of Singapore have developed a new reinforcement learning framework for self-improving AI systems.
This framework, called Self-Play in Corpus Environment (SPICE), pits two AI agents against each other, creating their own challenges and gradually improving without human supervision.
While currently a proof of concept, this self-play mechanism could provide a foundation for future AI systems that can dynamically adapt to their environments, making them more robust against the unpredictability of real-world applications.
Challenge of self-improvement in AI
The goal of self-improving AI is to create systems that can enhance their capabilities by interacting with their environment.
A common approach is reinforcement learning with verifiable rewards (RLVR), where models are rewarded for providing correct answers to problems. It is often limited by its reliance on human-curated problem sets and domain-specific reward engineering, making it difficult to scale.
Self-game, where a model improves by competing against itself, is another promising paradigm. But existing self-play methods for language models are often limited by two important factors.
FActual errors are mixed into the questions and answers generated, creating a feedback loop of hallucination.
When problem generators and solvers have information isomorphism (i.e., share the same knowledge base) they fail to generate truly new challenges and fall into repetitive patterns.
As the researchers write in their paper, “These systematic empirical failures indicate that self-improvement requires interaction with an external source providing diverse, verifiable feedback, rather than closed-loop pure introspection.”
How does Spice work?
SPICE is a self-play framework where a single model serves two different roles.
A "contender" Creates a curriculum of challenging problems from a large collection of documents.
A "logical" Attempts are then made to solve these problems without access to the source documents.
This setup breaks the information isomorphism that limits other self-play methods, because the Reasoner does not have access to the documents and knowledge that the Challenger uses to generate problems.
Incorporating tasks into a large and diverse collection of documents can prevent hallucinations by incorporating questions and answers into real-world content. This is important because for AI systems to reliably self-improve, they need external grounding sources. Therefore, LLM agents must learn from interactions with humans and the real world, not just from their outputs, to avoid compounding errors.
The adversarial dynamic between the two roles creates an automatic course.
The Challenger is rewarded for creating problems that are diverse and at the limit of the Reasoner’s ability (not too easy and not impossible).
The reasoner is rewarded for giving the correct answer. This symbiotic interaction pushes both agents to continually discover and overcome new challenges.
Because the system uses raw documents rather than pre-defined question-answer pairs, it can generate diverse task formats, such as multiple-choice and free-form questions.
This flexibility allows SPICE to be applied to any domain, breaking the barrier that has limited previous methods to narrow areas such as mathematics and code. It also reduces reliance on expensive human-curated datasets for specialized domains such as legal or medical analysis.
spice in action
The researchers evaluated SPICE on several base models, including the Qwen3-4B-base and the OctoThinker-3B-hybrid-base.
They compared its performance against baselines, such as a base model with no training, a reasoner model trained with a fixed "strong challenger" (Qwen3-32b-instruct), and pure self-play methods such as R-Zero and Absolute Zero. The assessment covered a wide range of mathematical and general reasoning benchmarks.
Across all models, SPICE consistently outperformed the baseline, providing significant improvements on both mathematical and general reasoning tasks.
The results show that reasoning abilities evolved widely through corpus-grounded self-play transfer across different models, thanks to the diverse external knowledge corpus they used.
A key finding is that adversarial dynamics create an effective automatic course. As training progresses, the Challenger learns to create increasingly difficult problems.
In one experiment, Reasoner’s pass rate on a certain set of problems increased from 55% to 85% over time, indicating its improved capabilities.
Meanwhile, later versions of Challenger were able to generate questions that reduced the pass rate of the early-stage reasoner from 55% to 35%, confirming that both roles were successfully developed.
The researchers conclude that this approach “presents a paradigm shift in self-improvement reasoning methods from closed-loop self-play that often freezes due to hallucinatory drift, to open-ended improvement through interaction with vast, verifiable knowledge embedded in web document corpora.”
Currently, the corpus used for SPICE represents the human experience captured in text. The ultimate goal is for self-improving systems to generate questions based on interactions with reality, including the physical world, the Internet, and human interaction across multiple modalities such as video, audio, and sensor data.