Constitutional AI vs RLHF: Which Alignment Method Should You Use in 2027?

By Kory White · Updated Oct 2026 · 7 min read

Direct answer

RLHF (reinforcement learning from human feedback) trains a model against a reward signal learned from human preference labels. Constitutional AI replaces most of those human labels with a short list of written principles and feedback from an AI model, a step the research calls RLAIF (reinforcement learning from AI feedback). The difference is mainly where the preference data comes from: people in RLHF, a model guided by a constitution in Constitutional AI.

Use human feedback when you need to capture what real users want and can afford to collect and review the labels. Use AI feedback when preference labels are the bottleneck, or when you want the rules the model follows written down and inspectable. In practice this is rarely an either/or choice. Published work treats the two as comparable on the tasks it tested, and current pipelines commonly mix supervised fine-tuning, human and AI preference data, and newer preference-optimization methods such as DPO (Direct Preference Optimization) that skip the separate reward model.

This page relies on the original papers and official documentation, linked in each section. It does not claim what any company uses in production today unless that company has published it.

What RLHF is

RLHF is the method behind InstructGPT (Ouyang et al., submitted 4 March 2022). The authors fine-tuned GPT-3 first with supervised learning on labeler-written demonstrations, then applied reinforcement learning from human feedback using rankings of model outputs. Their starting point was that making language models bigger does not inherently make them better at following a user's intent.

Hugging Face's RLHF explainer (December 9, 2022) lays the process out in three steps.

Pretrain a language model. Start from a base model that already handles a wide range of instructions.
Train a reward model. Collect human judgments on model outputs. Annotators rank outputs rather than score them, because raw scores are inconsistent between people, and the rankings are turned into a scalar reward.
Fine-tune with reinforcement learning. Optimize a copy of the model with a policy-gradient algorithm (PPO) against the reward model, with a KL divergence penalty that keeps it from drifting too far from the original.

The InstructGPT paper reports that human evaluators preferred outputs from the 1.3B-parameter InstructGPT model over the 175B-parameter GPT-3, even though it had 100x fewer parameters. It also reports improvements in truthfulness, less toxic output and minimal regressions on public NLP datasets, while noting that InstructGPT still makes simple mistakes.

What Constitutional AI and RLAIF are

Constitutional AI: Harmlessness from AI Feedback (Bai et al., Anthropic, submitted 15 December 2022) describes training a harmless assistant through self-improvement, without human labels identifying harmful outputs. The only human oversight is a list of rules or principles, the constitution. Anthropic's research summary describes two phases.

Supervised phase. The model samples responses, critiques and revises them against the principles, and the original model is fine-tuned on the revised responses.
Reinforcement learning phase. The model samples two responses per prompt, an AI model judges which is better, a preference model is trained on those AI preferences, and it becomes the reward signal for reinforcement learning. The authors call this RL from AI Feedback (RLAIF).

The paper reports that the resulting assistant is harmless but non-evasive: it engages with harmful queries by explaining its objections. It also reports that chain-of-thought style reasoning can improve human-judged performance and the transparency of the model's decisions, and that the approach allows more precise control of behavior with far fewer human labels.

The constitution itself is public. Anthropic's 2023 constitution post (May 9, 2023) said the principles drew on sources such as the UN Declaration of Human Rights, Apple's terms of service and DeepMind's Sparrow rules, and that the constitution was neither finalized nor likely the best it can be. A note on that page says a new version was published on January 21, 2026, and the current constitution says it plays a crucial role in Anthropic's training process and that Anthropic generally prefers cultivating good values and judgment to imposing strict rules.

RLAIF is a general idea, not only Anthropic's recipe. Google researchers tested it separately, as covered below.

How they differ in data and cost

Source of preference labelsRLHF: human annotators rank model outputs. Constitutional AI: an AI model judges outputs against written principles.
Human effortRLHF: labeling at scale. Constitutional AI: writing and revising the principles, plus whatever human data the rest of the pipeline uses.
InspectabilityRLHF: the values are implied by the labelers' judgments. Constitutional AI: the principles are written down and can be read.
Training stagesBoth: supervised fine-tuning, then a reward or preference model, then reinforcement learning.

On cost, the sources agree on where the pressure is. Hugging Face lists expensive human data, annotator disagreement, limited public datasets and the compute of scoring every output with a reward model among the challenges of RLHF. The RLAIF paper from Google (Lee et al., submitted 1 September 2023) opens from the same point: gathering high-quality preference labels is a key bottleneck and expensive.

The RLAIF paper trains the reward model on preferences generated by an off-the-shelf language model instead of humans. Across summarization, helpful dialogue and harmless dialogue, it reports that RLAIF achieves comparable performance to RLHF, and that RLAIF can outperform a supervised fine-tuned baseline even when the AI labeler is the same size as the policy or the same checkpoint. Its Direct-RLAIF variant skips reward model training and takes rewards directly from a language model during reinforcement learning, and the authors report it outperforms canonical RLAIF.

None of these sources publish a like-for-like dollar comparison, so this page does not give one. What they support is narrower: AI feedback can reduce the number of human preference labels you need, at the price of depending on the AI labeler's judgment and on how well the principles were written.

Where DPO and other preference-optimization methods fit

Direct Preference Optimization (Rafailov et al., submitted 29 May 2023) starts from a criticism of RLHF: it is a complex and often unstable procedure that fits a reward model and then fine-tunes the language model against it with reinforcement learning. DPO reparameterizes the reward model so the optimal policy can be extracted in closed form, solving the same problem with only a simple classification loss.

The authors describe DPO as stable, performant and computationally lightweight, with no need to sample from the model during fine-tuning or to do significant hyperparameter tuning. They report that it aligns models with human preferences as well as or better than existing methods, exceeds PPO-based RLHF in controlling the sentiment of generations, and matches or improves response quality in summarization and single-turn dialogue.

DPO changes the optimizer, not the source of the labels. Hugging Face's TRL documentation says DPO needs a preference dataset of paired completions to the same prompt, one chosen and one rejected, and optimizes the model against a reference model without an explicit reward model. Those pairs can come from humans or from an AI judge, so DPO combines with either approach above. TRL also documents several other preference-optimization loss types, such as IPO, BCO, SPPO, APO and SimPO-style length normalization, and lets you combine them.

When each method fits

These are judgment calls drawn from the trade-offs above, not rules from the papers.

Choose human feedback (RLHF) when the target is hard to write down, such as tone, helpfulness for a specific audience or domain taste, and you can pay for labelers and handle their disagreement.
Choose AI feedback (Constitutional AI or RLAIF) when labels are your bottleneck, when the behavior can be stated as principles, or when you need the guiding rules to be readable and auditable. Check the AI labeler's judgments against human review before trusting them.
Choose DPO or a related method when you already have preference pairs and want a simpler training loop than a separate reward model plus reinforcement learning.
Combine them when you need broad coverage. The sources describe overlapping stages, so mixing is natural: supervised fine-tuning first, preference data from humans, AI or both, then a preference-optimization or reinforcement learning step.

What most pipelines do in practice

The three methods are not rival products. Each of the papers builds on the same shape of pipeline: a supervised stage, a source of preference comparisons, and an optimization step. The differences are who supplies the comparisons (people or a model) and how the model is optimized against them (reinforcement learning or a direct loss). That is why Constitutional AI's reinforcement stage is called RLAIF, why the RLAIF paper benchmarks directly against RLHF, and why the DPO paper frames itself as a way of solving the standard RLHF problem.

This page does not state what any particular lab runs today. Where a company has published its method, as Anthropic has for Constitutional AI and as the InstructGPT authors did for RLHF, the link is above. Check the vendor's current model documentation for anything newer.

Questions people ask

What is the main difference between Constitutional AI and RLHF?

The source of the preference labels. RLHF uses human rankings of model outputs to train a reward model. Constitutional AI has a model critique and compare outputs against a written constitution, then uses those AI preferences as the reward signal (RLAIF). Source: Bai et al. and Ouyang et al..

Is RLAIF as good as RLHF?

On the tasks Google tested (summarization, helpful dialogue and harmless dialogue), the RLAIF paper reports comparable performance to RLHF. That is one set of tasks and models, so test it on your own use case. Source: Lee et al..

Does DPO replace RLHF?

DPO solves the same preference-alignment problem with a simple classification loss and no separate reward model, and its authors report it matches or beats PPO-based RLHF on the tasks they tested. It still needs preference data. Source: Rafailov et al..

Do I need human labels at all for Constitutional AI?

The paper trains for harmlessness without human labels identifying harmful outputs, but humans still write the principles, and a full pipeline may use human data for other goals such as helpfulness. Source: Anthropic.

Which method should I use in 2027?

Pick by your bottleneck: human feedback for hard-to-specify preferences, AI feedback when labels are scarce or you want written principles, and a direct method such as DPO when you want a simpler training loop. Many pipelines combine them.

Sources

Ouyang et al.: Training language models to follow instructions with human feedback (InstructGPT) · RLHF applied to GPT-3, March 2022
Bai et al.: Constitutional AI: Harmlessness from AI Feedback · Constitutional AI and RLAIF, December 2022
Anthropic: Constitutional AI: Harmlessness from AI Feedback · Anthropic's summary of the two phases
Anthropic: Claude's Constitution (2023) · The published principles and how they are used in training
Anthropic: Claude's Constitution (current) · The current published constitution
Lee et al. (Google): RLAIF vs. RLHF · AI feedback compared with human feedback, September 2023
Rafailov et al.: Direct Preference Optimization · DPO, May 2023
Hugging Face: Illustrating RLHF · Three-step RLHF explainer and its costs
Hugging Face TRL: DPO Trainer · DPO data format and preference-optimization loss types

Statements were read from the linked papers and pages on October 18, 2026. Results come from the authors' own experiments on the tasks and models they chose and will not transfer automatically to your data. The field changes quickly, so check current documentation before you pick a training recipe.