From preference to calibrated judgement.

Every model becomes what its training rewards. Reward human preference and you get answers people like. Reward verified correctness and you get answers that check out — wherever something can check them. Reward calibrated decisions and you get something different in kind: a choice, and an honest statement of how likely that choice is to be right.

In September 2026 TypeSafe AI introduced RLCD — Reinforcement Learning for Calibrated Decisions — as the training method behind Jev, the first of what it calls System One models. The name borrows Kahneman's fast, intuitive System 1. Instead of generating text, a System One model selects among defined options and returns a probability with each, and TypeSafe positions it as a complement to language models rather than a replacement.

This note sets RLCD beside the two reward objectives that came before it, then asks the Octonomous question. The Decision AI field note argued that calibration makes confidence legible and a Being makes it someone's. This one goes a layer down — to where that confidence comes from in the first place.

The short version: RLHF teaches preference, RLVR teaches correctness, and RLCD teaches calibrated judgement — the only one of the three that hands software a number it can use to decide not to act.

What does the signal reward?

Strip the acronyms back and the difference is not the optimiser. It is the target. Each method asks a different question of every output the model produces during training, and the model becomes whatever answers that question well.

Do people like this answer?

Give a model a prompt — explain quantum computing to a twelve-year-old — and let it write several answers. People rank them: A over C over B. Many thousands of rankings like that train a reward model that predicts which answer a person would prefer, and reinforcement learning then pushes the language model toward answers the reward model scores highly.

This is the recipe that made language models conversational. It grew out of learning from human comparisons (Christiano et al., 2017) and became the basic InstructGPT approach, where human demonstrations and comparisons built the reward signal used for fine-tuning (Ouyang et al., 2022).

The strength and the weakness are the same fact: people are the objective. A reward model trained on preferences learns whatever people reliably prefer, and people reliably prefer answers that sound sure. There is measured evidence that this costs calibration. In the GPT-4 technical report, the pre-trained model's confidence on a multiple-choice benchmark was well calibrated, and post-training reduced that calibration (OpenAI, 2023).

Can a machine check it?

Now take a question that has an answer: 37 × 84. The model says 3108. No preference judgement is needed. A verifier computes the product and compares.

The verifier can be anything that returns a trustworthy pass or fail:

Because vast numbers of attempts can be scored automatically, RLVR has become central to reasoning models in maths and code. The Tülu 3 post-training work gave the method its name (Lambert et al., 2024), and DeepSeek-R1 showed how far rule-based rewards alone can push reasoning (DeepSeek-AI, 2025).

But look at what the reward can see. A binary verifier asks one question — did you get it right? — and never the second: did you know how likely you were to be right? A model that computed 3108 and a model that guessed it get the same reward. Research on calibration rewards found that training on correctness alone tends to leave models overconfident, and adds a calibration term to the reward for exactly that reason (Damani et al., 2025).

How sure should you be?

TypeSafe describes RLCD as optimising for calibrated decisions — answers carrying epistemically honest probabilities — rather than for human preference or a verifiable reward. Jev gives up free text entirely: it selects among defined options (up to 255, per TypeSafe) and returns a probability with each.

Imagine an insurance system asking whether a claim is likely fraudulent. A conversational model answers "Yes, I believe this claim may be fraudulent because…". A decision model returns something closer to this:

{
  "decision":    "investigate",
  "probability": 0.82
}

The critical property is calibration. If a model makes a thousand decisions at a stated 0.80, about eight hundred of them should turn out right. A model that says 0.99 every time and is right eight hundred times in a thousand is badly calibrated — its number describes its tone, not its record.

Same accuracy, different honesty. So the training loop changes shape. The reward no longer comes from being correct alone:

Calibrated is necessary. It isn't sufficient.

Calibration taken alone has a trap in it. On a book where one claim in ten is fraudulent, a model that says 0.10 for every claim is perfectly calibrated — and useless. It never separates the fraud from the rest. A decision system needs probabilities that are honest and that move, decisively, when the evidence does. Forecasters call the second property sharpness.

Forecasting solved this long before language models. Proper scoring rules — the Brier score (Brier, 1950), log loss — reward a probability for being both honest and decisive; a forecaster scored this way is best off reporting exactly what they believe. Calibration-aware reinforcement learning uses the same idea: RLCR adds a Brier term to a correctness reward (Damani et al., 2025). TypeSafe's published material describes RLCD's target — probabilities that match reality — rather than its exact scoring rule.

Same question, three kinds of answer

Ask each kind of model whether a payment looks suspicious:

Only the third can be wired into software without a person interpreting an adjective. Policy becomes code:

if (fraud > 0.95) {
    block();
} else if (fraud > 0.70) {
    humanReview();
} else {
    approve();
}

At 0.82 this payment goes to a person. The probability has become part of the software interface — and, as the Decision AI note argues, part of how authority is allocated.

Same machinery, different objective

None of this makes RLHF, RLVR and RLCD three different learning algorithms. The same policy-optimisation machinery can sit under all three. What changes is what the reward is computed from:

            Reinforcement learning
                      │
           ┌──────────┼──────────┐
           │          │          │
          RLHF       RLVR       RLCD
           │          │          │
         people    verifier   outcomes +
                              calibration
           │          │          │
      preference  correctness  probability

Confidence decides whether to think harder

The architectural payoff is routing. TypeSafe reports Jev responding in 70–500 ms, against seconds to minutes for frontier reasoning models. Put a fast calibrated layer first, and let its confidence decide whether slow, expensive reasoning runs at all:

                Input
                  │
                  ▼
          System One · RLCD
                  │
        decision + confidence
                  │
        ┌─────────┴─────────┐
        │                   │
  confidence          confidence
    > 0.95              ≤ 0.95
        │                   │
        ▼                   ▼
       ACT          System Two · LLM
                            │
                      reason · search
                            │
                            ▼
                      final decision

Most inputs are easy; they clear the bar and are handled in milliseconds. The hard ones — ambiguous, novel, split — are exactly the ones a calibrated fast layer is honest about, and those get the reasoning budget. That is the real significance of RLCD: calibration gives an autonomous system a mechanism for knowing when not to trust its own fast judgement.

Two corrections before this ships

First, the diagram routes on confidence alone. For a Being, confidence is one input of three. Confidence decides how much thinking a decision gets; consequence and delegation decide how much authority it gets. A 0.96 on restarting a cache and a 0.96 on moving money should not share a branch — the authority bands in the Decision AI note set out why.

Second, a calibrated 0.95 is still wrong up to one time in twenty. Calibration doesn't remove that error. It makes it knowable:

An error rate you can predict is an error budget someone can sign. Whoever sets the 0.95 is choosing up to five hundred mistakes a day in exchange for speed. In an Octonomous system that choice is recorded, owned, and revisited when the outcomes come in.

Every reward is a proxy

Optimise hard enough on any proxy and the model learns the proxy. Each objective has a characteristic way of being satisfied without being served:

None of these is a reason to avoid the method. Each is a reason to keep measuring after training ends — which is where a Being comes in.

The engine is trained. The Being is accountable.

Octonomous separates a Being from the model that gives it intelligence — bring your own engine. Read through that lens, the training reward is a property of the engine, and it decides what kind of signal the Being gets to work with:

A Being can use all three: a System One engine for the fast decision, a reasoning engine for the slow path, a conversational one to explain itself to the person who owns the outcome. What it cannot do is outsource the question of whether to act.

There is a second link. RLCD closes a loop at training time — decision, outcome, calibration, reward. A Being runs the same loop in production, without retraining: every signed decision records its stated confidence, every outcome is scored against it, and the Being's calibration curve becomes its reputation. Benchmark calibration belongs to the engine. Calibration on this Being's decisions, in this domain, under this delegation, belongs to the Being — and that is the one authority should be set against. Swap the engine and the record has to be earned again.

What this doesn't claim

RLCD is TypeSafe's term for its own method. Commentary has rightly cautioned against reading it as an established field standard, or reading "cannot hallucinate" as "cannot be wrong" (Eliot, 2026). TypeSafe itself notes that its workflow evaluations were built by its own team and may sit at the higher end of real-world gains. Calibration-aware rewards also predate it (Damani et al., 2025).

The distinction this note draws doesn't depend on any one vendor. Preference, correctness and calibrated judgement are different targets, whoever's method is used to reach them. Octonomous is a framework for what a Being is. It doesn't make an engine calibrated; it makes an engine's confidence belong to someone who answers for it.

Knowing when not to trust yourself

RLHF taught machines to answer the way people like. RLVR taught them to answer correctly wherever an answer can be checked. RLCD — and calibration-aware training generally — teaches something neither did: how far to trust their own answer, in a form software can act on.

That is a more consequential shift than another training acronym. For an autonomous system, a calibrated fast judgement is the mechanism that decides when slower thinking, more evidence or a human is needed. For an Octonomous Being it is one more thing that has to belong to someone: a number that met a threshold, a threshold someone signed, and an outcome that updates the record either way.

Curious treats doubt as a question. Caring asks who carries the error. Constructive scores every outcome against what was said. And Chill knows that when the fast judgement isn't sure enough, the most autonomous thing a Being can do is decline to act.

Sources

• TypeSafe AI (2026). Introducing System One Models & Jev. 15 September 2026. typesafe.ai/blog
• TypeSafe AI (2026). AI primer — machine learning background for System One models. docs.typesafe.ai
• Christiano, P. et al. (2017). Deep Reinforcement Learning from Human Preferences. NeurIPS. arXiv:1706.03741
• Ouyang, L. et al. (2022). Training Language Models to Follow Instructions with Human Feedback. NeurIPS. arXiv:2203.02155
• OpenAI (2023). GPT-4 Technical Report. arXiv:2303.08774
• Lambert, N. et al. (2024). Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124
• DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948
• Damani, M. et al. (2025). Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty. arXiv:2507.16806
• Yang, K. et al. (2023). RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment. arXiv:2307.12950
• Brier, G. W. (1950). Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review, 78(1).
• Guo, C., Pleiss, G., Sun, Y. & Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. ICML. arXiv:1706.04599
• Eliot, L. (2026). New 'Reinforcement Learning for Calibrated Decisions' Makes AI Headlines But Look Past the Hype. Forbes, 18 September 2026. forbes.com
• Kahneman, D. (2011). Thinking, Fast and Slow. Farrar, Straus and Giroux.
• Octonomous (2026). From Decision AI to Accountable Beings — field note. octonomous.io/decision-ai
• Octonomous (2026). An Open Framework for Self-Sovereign Intelligent Beings. octonomous.io/paper
• selfdriven Foundation (2026). The 4Cs — Curious, Caring, Constructive, Chill. selfdriven.you