For months I’ve been researching whether I can build a harness with open-weight models (8B, 16B, 32B parameters, some even running locally) that matches the quality of Claude Code. Not for sport. Because if it works, you get full sovereignty over your data and pay a fraction of what frontier APIs charge.
The main pain point is obvious: these models have small context windows. A medium-sized codebase doesn’t even come close to fitting. I considered several solutions. RAG, grep, summaries. But the only one that really made sense was RLM: Recursive Language Models, a paradigm from an MIT paper.
How RLM works: the context doesn’t go into the prompt; it lives as a variable in a code environment.
The idea: instead of stuffing all the context into the prompt and hoping the model doesn’t get lost, the context lives as a variable in a programming environment. The model writes code to inspect it, filter it, and decompose it. And it can spawn sub-models on specific fragments.
It’s not RAG. It’s not a bigger context window. It’s the model deciding programmatically what it needs to see.
The official paper says RLM is not ideal for small models. But you know how it goes. I had to test it myself.
From this research I published a LinkedIn post, read what others were saying (good analyses from Brainqub3 and Neural Breakdown on the same paper), and designed a formal benchmark. Here are the results.
Spoiler: they surprised me.
First, the quick context
Today’s coding agents already solve this in practice. Claude Code, Codex, Cursor all give the model a terminal with grep, ripgrep, find, cat. The model searches for what it needs, reads on demand, and it works. But they all depend on frontier models and expensive APIs, which is exactly what I’m trying to avoid.
So the real question is twofold: can RLM do something a terminal can’t? And if it can, can it do it with small models?
To answer that, I built a benchmark with three arms:
Direct dump (vanilla). You put as much code as fits in the prompt, truncating the rest. No tools, no navigation. The model only sees what fits.
Terminal agent (Pi). A minimal coding agent: just enough system prompt, just enough tools. It has bash and read, nothing more. I chose it precisely for that reason: if the most spartan agent already competes, the argument is stronger.
RLM. The technique from the MIT paper. The context is loaded as a variable in a REPL. The model navigates it with code and can invoke sub-models on fragments of the corpus. The full context never enters the root model’s window.
The three paradigms I compared. Same model, same questions, different way of accessing the code.
What happened before I got clean results
This wasn’t a straight path. The rlms pip library had a confusing API where sub-models fell back to gpt-3.5-turbo if you didn’t configure parameters that weren’t documented. Qwen3.6-27B’s reasoning tokens (<think>...</think>) broke the REPL parser. OpenRouter rate-limited me mid-run. After a lot of trial and error, with my companion for the day (Claude Code, using Opus 4.6 to think through architecture and Opus 4.8 for design and testing), I switched to fast-rlm and restarted from scratch more than once.
One of many failed attempts. Qwen’s thinking tokens broke the REPL parser.
I mention this because it matters. If someone shows you RLM results without mentioning that setting it up is significantly more complex than giving a model a terminal, they’re only telling you half the story. Either that, or I’m the only one who found it this hard.
What I discovered: task complexity picks the winner, not the tool
This was the most valuable finding of the whole experiment. I designed tasks at three levels of computational complexity:
Constant. Needle in a haystack. “Where is X defined?”, “What is the default for Y?” A single data point.
Linear. You have to process most of the input. “List all CLI configuration options.”
Quadratic. Relationships between dispersed entities. “Which files in src/ don’t have tests?” Cross every item against all the others.
I used the same model across all three arms (kimi-k2.6 via OpenRouter) to compare paradigms, not models. The codebase was repomix: 1037 files, 1.13 million tokens. Ground truth was verified mechanically with set-based F1, not “by eye.”
First test: can a cheap model with RLM match an expensive one?
Before the head-to-head between the three paradigms, I ran a simpler test. I pitted a cheap model (Qwen) orchestrating with RLM alone against an expensive model (kimi) with all the context dumped in directly. Ten factual questions and two synthesis questions.
Result: the cheap model with RLM got all ten factual questions right and reached 80% of the expensive model’s synthesis quality, for one-ninth of the cost. And the weirdest part: when I put the expensive model as the RLM orchestrator, it did worse. 3× more expensive and 5× slower than the cheap model doing the same thing.
This confirmed what the MIT paper says: RLM lets small models work with contexts they normally couldn’t handle. Promising. But the real test was missing: what happens when you add a terminal agent to the comparison?
The head-to-head: the three paradigms
Same model across the three arms (kimi-k2.6), tasks at all three complexity levels, F1 verified.
| Complexity | vanilla | Pi (terminal) | RLM |
|---|---|---|---|
| constant (needle) | 1.00 | 1.00 | 1.00 |
| linear (enumerate) | 0.97 | 0.97 | 0.97 |
| quadratic (cross-ref) | 0.00 | 0.72 | 0.70 |
Total cost (6 tasks): Pi $0.16 · RLM $0.51 · vanilla $1.09.
What it means, tier by tier
Constant: total tie, Pi wins by a landslide on cost. All three score 100%. But Pi cost ~$0.004 per task. RLM $0.04. Vanilla $0.18 (because it resends 212K tokens on every question). For finding a single data point, grep in one turn is unbeatable. RLM is overkill.
Linear: tie across the board. All three scored ~0.97. To be honest: our linear tasks (listing the 49 CLI flags, config options) turned out to be concentrated in 1-2 files that fit in the window. They didn’t force reading the whole repo. That’s why they didn’t discriminate. A true linear test would require information scattered across hundreds of files. Pending.
Quadratic: the surprise that broke my hypothesis. I went in expecting only RLM to handle massive cross-referencing. I was wrong.
Vanilla went to 0.00 (it can’t cross-reference what it can’t see). But Pi scored 0.72 and RLM 0.70. Practically tied. On the task “which files in src/ don’t have tests?”, Pi got a perfect F1: 1.0. RLM got 0.99. Vanilla: 0.0.
Why did Pi win a “quadratic” task? Because with bash it wrote a script. It listed src/, listed tests/, and computed the set difference. It didn’t grep the answer. It computed it.
So what is RLM actually for?
The real differentiator is running code over the corpus, not the specific paradigm. Vanilla sinks to 0 in the quadratic tier because it only receives passive text. The other two win because both can compute over the corpus: RLM via its REPL, Pi via its terminal. They are the same essential capability (programmatic navigation over data) served two ways. A coding agent’s bash is already 80% of what RLM offers.
In my code tests, RLM did not outperform the terminal agent at any tier. And it cost more. It tied on constant and linear, tied (slightly below) on quadratic, and was 3× more expensive than Pi overall.
But the RLM niche exists. Quadratic and semantic tasks over 6-11 million token corpora that neither fit in the window nor reduce to shell operations. “Which of these 900 functions are semantically equivalent despite different names?” No keyword. Not scriptable. Requires understanding meaning. For that, RLM’s recursive sub-models process what grep can’t read and the window can’t hold.
What changed in my head
I started this experiment hoping RLM would solve a concrete problem: making my small models, with limited context windows, able to understand large codebases without relying on expensive APIs.
The results left me with two conclusions that seem contradictory but aren’t.
First: RLM did help small models. Qwen alone, with truncated context, got 8 out of 10 factual questions right and scored 4.0 on synthesis. The same Qwen with RLM got all 10 right and rose to 6.0, for less money. For my original goal, that’s a real and promising data point.
Second: when I confronted the three paradigms with a capable model (kimi-k2.6), the terminal agent matched RLM across the board and cost 3× less. For a model that already has enough capability, the terminal covers what RLM offers.
What I didn’t test, and it’s the open question, is giving a terminal to that same small model. I don’t know whether Qwen’s improvement came specifically from RLM, or whether any form of context navigation (including a terminal) would have given it the same boost. Pending.
The process taught me something I didn’t expect: what matters isn’t the paradigm, but whether the agent can execute code over the data. And it showed me there’s a real territory (massive semantic reasoning over huge corpora, deep legal analysis, enterprise document synthesis) where RLM unlocks things you can’t do any other way today.
I didn’t find exactly what I was looking for. I found something more useful: a map of when to use each tool, and a new question worth answering.
Weeks of work condensed into a table. Sometimes what you find is better than what you were looking for.
Limitations (so you read this honestly)
I don’t want anyone to take these numbers as definitive. There are things I know I didn’t cover:
The scale was 1.13M tokens. It barely exceeds current windows. I didn’t reach the 6-11M regime where RLM should really take off. The cost was prohibitive for this study.
The linear tier didn’t discriminate because the answers were concentrated in a few files. A real linear test needs information scattered across the whole repo.
The quadratic tasks were scriptable (cross-referencing filenames). Semantic quadratics (RLM’s niche) remain pending.
I used a single model (kimi-k2.6) and small n. The numbers are directional, not definitive.
Reproducibility
The full harness, raw data from the published runs, and frozen ground truth are in aleksandarlabs/rlm-benchmark.
To reproduce with the same corpus (repomix):
git clone https://github.com/aleksandarlabs/rlm-benchmark.git
cd rlm-benchmark
git clone https://github.com/yamadashy/repomix.git corpus
cp examples/questions-repomix-exp1.json questions/questions.json
cp examples/questions-repomix-exp2.json questions/questions-complexity.json
python benchmark/corpus.py
python benchmark/run_bench.py --condition rlm-all-small
python benchmark/compare.py
python benchmark/run_complexity.py --limit-per-tier 1
python benchmark/run_complexity.pyRequired keys (in .env): GROQ_API_KEY, OPENROUTER_API_KEY. The Pi arm requires the pi CLI installed.
This study came from a practical problem: making small models useful for programming over codebases that don’t fit in their context window. RLM looked like the answer. It turned out to be part of the answer, but not the one I expected. And the question I’m left with is better than the one I started with.
AleksandarLabs · July 2026