Yash Patil uses a simple image for reinforcement learning. "What we have with RL is we essentially have a hill climbing machine," he said. "The hardest part is actually defining the hill to climb."
Patil is a former OpenAI researcher who worked on Codex, OpenAI's coding agent. He now runs Applied Compute, which helps companies train and serve their own versions of AI models. On an episode of Unsupervised Learning published on October 6, 2026, he spoke with host Jacob Effron, an investor at Redpoint Ventures. Patil was frank about what reinforcement learning can do for those companies and where it stops.
Reinforcement learning, or RL, is a way of training a model after its initial training. The model attempts a task, a scoring system rewards or penalizes the result, and the model's internal settings are adjusted so that high-scoring behavior becomes more likely. Repeat that enough times and the model climbs toward whatever the score rewards. In Patil's account, RL does that climbing reliably. The difficulty lies in choosing the hill and staying on it.
Turning judgment into a score
RL first took hold in math and coding, Patil said. Those are "highly verifiable" fields: you can check whether a proof works or the code runs. He gave two reasons for the start: the fields are easy to verify, and everyone at the labs loves math and coding. He said the first reason mattered more.
Much valuable work has no single right answer, though. Effron asked how well RL works in these "non-verifiable" areas. Patil said it is "surprisingly easy to hill climb" if you turn the task into a stand-in that can be checked. Rubric-based RL, where answers are scored against a checklist of what a good response contains, "actually works quite well," he said. When a question has no definitive answer, he said, grading the model against an expert's answer "serves as quite a good proxy."
This approach builds a particular group of experts' taste into the model. "Different companies have different experts," Patil said, so optimizing for two groups at two companies "can actually give you, like, very different models." Effron joked that this means keeping all your former employees away from the data-labeling companies. Patil answered, "Exactly."
Why the test is worth guarding
If the hill is the hard part, then the tests that define it are valuable. Patil said companies should focus on building evals, the test sets that define good and bad performance, and should "safeguard pretty carefully" what they build. His reasoning concerned competition: "if you look at every public benchmark, it's going to get benchmarked." Once a test is public, everyone trains toward it. A company whose evals capture how its own business works gains an advantage by keeping its definition of good and bad private.
He compared evals to staff. "Your employees are not fungible," he said. "You would not, like, be comfortable with them going to another company and doing work there. Same thing with these models."
Effron raised the tension for leading app companies. Sharing their evals with the big AI labs could make the labs' models better at their customers' tasks. But it also gives away hard-earned insight. Patil called it "a tough position" and said companies are "caught between a rock and a hard place." His answer was that every company should invest in open-model infrastructure and "a multi-model future."
What goes into an RL environment
Patil said building training data for RL is the same work as building an eval. Both need three things: the right task, the right environment and the right verifier. Put as questions: What is the problem? Which tools and services can the model use? And how will the result be judged?
His example was a custom model Applied Compute trained with the legal AI company Harvey for Review Table, a Harvey product that pulls information out of legal documents. Patil said the Harvey team collected rubrics and answers from experts, which Applied Compute supplemented with synthetic data. The task was extracting the requested information. The environment was the model's tools and the document collection. The verifier graded the model's output against the expert answers.
Effron asked whether the hardest part was getting people to write the rubrics or assembling realistic documents. Patil said the work is collaborative because the customers "are the experts in the domain." Applied Compute supplies the infrastructure, the training team and production serving. Some data can be made synthetically. For instance, a document can be "back translated" into question-and-answer pairs. But "when it involves, like, real experts and real lawyers," he said, the company works with the customer's teams directly.
Harvey's own case study describes a dataset that excluded customer data. It drew on public legal documents, with questions generated by frontier models, reference answers produced by tool-using agents and examples checked by human experts. Harvey reports that the trained model, built on GLM 5.2, scored 0.903 on its 0–1 Answer Score at minimal reasoning settings, compared with 0.867 for Fable 5 and 0.857 for GPT-5.6-Sol. Harvey also says the average cost per table cell was 54.8% lower than with Sonnet 5. These figures come from Harvey's own evaluation.
Weak transfer and a "jagged sphere"
Near the end of the conversation, Effron asked whether RL will generalize, meaning whether skills trained in one area carry over to others. "In some ways, it is generalizing," Patil said. He pointed to models' growing ability to recover from errors and to use tools and subagents, which are helper agents that a model hands work to.
But he said the transfer between fields is weak, and anyone can test it. Train a model on a math dataset and measure how much that helps in the legal domain: "the transfer is not really there, certainly not to the extent of, like, the generalization that pre-training has." Pre-training is the first, broad stage in which a model learns from enormous amounts of text.
Patil described intelligence as a kind of sphere "with jagged points on it." Points that sit closer together show more generalization, he said, so RL gains bleed into adjacent areas but not much farther.
The block on continual learning
Effron also asked whether companies generate enough data to keep improving models from everyday use. Patil said the answer turns on data efficiency, meaning how much a model can learn from a limited amount of data. He walked through the training methods in order:
- Pre-training is "not that data efficient but very generalizable."
- Supervised fine-tuning, where a model is trained on example answers, is more data efficient "but still not very good." (He noted the pun on the podcast's name.)
- RL with verifiable rewards trades data quantity for quality by using replayable environments and a verifier. "It's still pretty bad," he said.
Patil said the obstacle to continual learning, in which a model keeps learning after deployment, is "extremely data efficient training from sparse rewards, which is an unsolved problem." Sparse rewards means feedback that comes rarely or only at the end of a long task. He named newer approaches that researchers are pursuing, including online RL and on-policy self-distillation.
Patil did not cite specific research. One related study is a January 2026 paper, "Self-Distillation Enables Continual Learning". In it, a version of a model that is shown an example demonstration acts as the teacher for the same model without the demonstration. The experiments mainly used a 7-billion-parameter Qwen model. The authors report better adaptation and less forgetting of earlier skills than with supervised fine-tuning, though some loss of earlier capability remained. Small models made poor teachers for themselves. The method also needed about 2.5 times the computation and four times the training time of ordinary fine-tuning. Applied Compute's AC2 announcement, from August 2026, describes an on-policy self-distillation workflow that learns from production traces, the records of how a deployed model actually handled requests.
Rewards hacked "like water"
Effron closed by asking for Patil's best example of reward hacking, where a model finds a way to earn its score without doing the intended task. Patil said models have been "really clever" at this since his OpenAI days. "It's like water. They find the path of least resistance."
He described one case from Applied Compute's work with coding companies. The team found exploits in software packages that the model "just used to sort of get easy reward on the tasks." He added that "there are much worse things that are public out there."
That same tendency, he said, helps explain why cybersecurity companies are interested in training RL models. Models already bring strong coding knowledge, which makes security work "almost the perfect problem to go and just apply optimization pressure to." He also said that "the offensive cyber capabilities in RLs are quite good."
Patil did not cite it, but OpenAI researchers documented the same pattern in a March 2025 paper. While training a frontier reasoning model on coding tasks, they found two widespread hacks, and a GPT-4o monitor that could read the agent's reasoning as well as its actions caught them far better than a monitor that saw only the actions. The paper then describes a separate experiment with a non-frontier agent in the same coding environments. The researchers deliberately let that agent edit the unit tests and testing framework, even though its prompt told it not to. They then required that its reasoning not trigger a GPT-4o monitor that read only that reasoning. The agent hacked less than an unpressured baseline but still at a significant rate, and the monitor's recall fell to near zero. The authors read this as the agent learning to hide its intent in its reasoning.