How Do You Add LLM Evals to Your CI/CD Pipeline?

LLM evals in your CI/CD pipeline are the difference between shipping a better prompt and shipping a broken feature. Without automated evaluation, every change to a prompt template, model version, retrieval configuration, or system instruction is a gamble — you find out quality dropped when users complain, not when the PR merges.

Most teams building LLM applications evaluate manually: a developer eyeballs a few outputs, decides they look reasonable, and ships. This fails at scale for the same reason manual testing fails in software engineering — it does not catch regressions, it does not run on every change, and it produces no signal you can track over time. By the time you notice a quality drop, it has already reached production users.

This guide covers how to build an LLM evaluation CI/CD pipeline that catches quality regressions before they reach production: the data you need, the frameworks available, how to set quality gates on non-deterministic outputs, and how to control evaluation cost. It pairs with our deep-dive on evaluating AI agents in production (runtime monitoring after deployment) and our guide on getting reliable structured output from LLMs, which covers the schema-enforcement layer that makes many eval metrics measurable in the first place.

What Are LLM Evals and Why Do They Belong in CI/CD?

An LLM eval is a programmatic check that measures whether a model output meets a quality bar for a specific task. It is the LLM-world equivalent of a unit or integration test: you provide an input, optionally a reference output or rubric, and a scoring function that returns a pass/fail or a numeric score. Evals run on a dataset of representative inputs — your golden dataset — and produce an aggregate quality signal that can be compared across commits.

Evals belong in CI/CD for the same reason tests do: to catch regressions before users see them. In LLM applications, regressions happen silently. A prompt change that improves performance on one class of inputs often degrades another. A model version bump from your provider changes output distribution without notice. A retrieval index update changes what context gets injected, shifting answer quality in ways that are invisible to a reviewer eyeballing a handful of samples.

• →Prompt regressions: changing a system prompt or few-shot examples can increase hallucination rate, reduce instruction following, or alter tone in ways that are not obvious from manual inspection of 5 to 10 samples but become visible across a 100-example golden dataset.
• →Model version drift: LLM providers update base models and fine-tunes with little advance notice. An eval suite run as a PR check catches quality shifts from a provider update before they ship to users.
• →RAG pipeline changes: modifying chunk size, embedding model, reranker, or retrieval top-k changes what context the LLM sees, which changes answer accuracy and faithfulness — metrics your eval suite can measure automatically.
• →System integration changes: adding a new tool, modifying a function schema, or changing the format of injected context can break instruction following in structured output pipelines without producing any obvious error.

The Four Levels of LLM Evaluation Maturity

Teams building LLM applications fall into one of four maturity levels. Understanding where your team is now helps you prioritize what to build next rather than jumping to advanced tooling before the fundamentals are in place.

• →Level 0 — Manual review: outputs are evaluated by eyeballing samples before each release. No automation, no dataset, no regression detection. This is where most teams start and where most remain until they are bitten by a production regression that would have been caught by any automated check.
• →Level 1 — Deterministic checks: unit-test-style assertions on model outputs. Check that a JSON output parses, that a required field is populated, that output length is within bounds, that a structured extraction schema is satisfied. Fast to run, zero LLM cost, catches structural regressions. Every team should have this layer before shipping to production.
• →Level 2 — LLM-as-judge automated scoring: a second LLM evaluates the output against a rubric. Metrics like answer relevancy, faithfulness, hallucination rate, and coherence are measured automatically on a golden dataset. This is the layer that catches semantic regressions — the ones level 1 misses. Targets roughly 70 percent eval coverage on your golden dataset with CI/CD integration.
• →Level 3 — Continuous production eval: a sample of live production traffic is evaluated automatically, scored by an LLM judge, and monitored for distribution shifts over time. Failures alert the on-call team. This is the state of the art for teams where LLM quality is a direct revenue-affecting metric.

Building a Golden Dataset for LLM Regression Testing

A golden dataset is a versioned collection of inputs — and optionally reference outputs — that represents the real distribution of your LLM application's traffic. It is the foundation of a CI/CD eval pipeline. Without a representative dataset, your evals measure performance on synthetic examples that may not reflect the failures users actually hit. The quality of your dataset determines the quality of your evaluation signal more than any tool or metric choice.

• →Source inputs from production: seed your golden dataset by sampling real user inputs from production logs. Anonymize or redact PII before storing. Production inputs expose edge cases and phrasing patterns that synthetic data never covers — they represent what your users actually ask, not what you think they ask.
• →Include known failure cases: every time a user reports a quality issue, add the input to the dataset with the expected correct output. A dataset that includes your historical failure modes is more valuable than one that only covers the happy path, because it catches the same failure from recurring in a future deploy.
• →Stratify by input class: ensure the dataset includes examples from each major input category your application handles — question answering, summarization, extraction, comparison, multi-step instructions. A dataset skewed toward one type misses regressions in the others.
• →Version the dataset alongside the code: store the golden dataset in your repository or an artifact store with a content hash. Pin the version used for each eval run so you can reproduce CI results and distinguish whether a score change came from a code change or a dataset change.
• →Start with 50 to 200 high-quality examples: 100 carefully labeled examples with ground-truth annotations outperform 5,000 synthetic examples at regression detection. Build the dataset manually, validate it against human judgments, and expand it incrementally as production inputs arrive.

LLM-as-Judge: How to Automate Quality Scoring at Scale

LLM-as-judge evaluation uses a second language model — often a stronger or different model from the one powering your application — to score whether an output meets a quality rubric. It is the only scalable approach to measuring semantic quality automatically: deterministic metrics can tell you whether output is well-formed, but only a language model can assess whether an answer is accurate, faithful, or contextually appropriate.

• →Define the rubric before writing the judge prompt: specify what 'correct' means for your task before building the evaluator. For a RAG-based Q&A system, this means — is the answer grounded in the provided context? Does it answer the question asked? Is it factually consistent? Vague rubrics produce inconsistent and unreliable judge scores.
• →Calibrate against human labels: run your judge on 25 to 50 examples where you have verified human quality labels and measure agreement rate. A judge that agrees with human raters less than 80 percent of the time needs prompt revision before it can be trusted as a CI/CD gate.
• →Use structured judge output: prompt the judge to return a score (1 to 5, or 0/1) plus a brief reasoning string in JSON. Structured output makes scores parseable by your CI pipeline and the reasoning auditable when a PR is blocked by a failing eval.
• →Use a different model family for the judge than for the application: if your application uses Claude, use GPT-4o or Gemini as the judge, and vice versa. Same-model judging introduces systematic bias — the judge model tends to favor outputs that match its own generation patterns.
• →G-Eval for task-specific dimensions: G-Eval prompts the evaluator to assess quality on user-defined dimensions using chain-of-thought reasoning, reducing positional and verbosity bias. Use it for quality dimensions not covered by standard off-the-shelf metrics — for example, evaluating whether a sales email matches the target persona's seniority level.

Choosing Your Eval Framework: DeepEval, Promptfoo, or Braintrust

Three frameworks dominate enterprise LLM eval CI/CD pipelines in 2026. They are not interchangeable — the right choice depends on where in the development workflow your primary evaluation needs are, and whether you are optimizing for engineering-owned testing, prompt security checks, or full-lifecycle quality management.

• →DeepEval: a pytest-compatible framework with 50+ built-in metrics including hallucination detection, answer relevancy, contextual recall, faithfulness, and G-Eval. Integrates with GitHub Actions, GitLab CI, and Jenkins via the deepeval test run command. Best choice when evaluation is a code-level engineering activity and your team already runs pytest in CI. The Confident AI platform adds team dashboards, dataset management, and regression tracking on top of DeepEval's metric layer.
• →Promptfoo: an open-source CLI optimized for prompt testing, comparison, and red teaming. Declarative YAML configuration means non-engineers can contribute eval definitions. Ships 50+ red team plugins covering prompt injection, PII leakage, and jailbreaks — the strongest open-source option for LLM security testing in CI/CD. Best run as a parallel pipeline alongside DeepEval: Promptfoo handles 'can it be broken' and DeepEval handles 'does it meet quality thresholds.'
• →Braintrust: the only platform that connects all eval lifecycle stages — dataset versioning, human annotation, automated LLM scoring, production monitoring, and CI-based quality gate enforcement — in a single system. Best for teams where LLM quality directly affects customer trust or revenue and every release needs documented quality evidence before it ships. Supports self-hosted deployment for enterprise data governance.
• →LangSmith: strong fit for LangGraph-based pipelines where you want tracing and evaluation in the same tool. Covers eval dataset management, human feedback collection, and automated scoring with tight LangChain ecosystem integration. Less flexible than DeepEval for custom metrics outside the LangChain model of prompt runs and chain traces.

Setting Quality Gates Without Breaking CI on Non-Determinism

The hardest engineering problem in LLM CI/CD is setting quality gates on outputs that are inherently non-deterministic. A fixed threshold gate that blocks the build when answer relevancy drops below 0.75 will produce false positives — the same code, the same dataset, the same model, a different run, a different score — because LLM outputs vary across invocations. The teams that build stable CI/CD eval pipelines treat scores as probabilistic signals, not binary assertions.

• →Gate on aggregate score, not individual samples: block a deployment when the average score across the full golden dataset drops below a threshold, not when any single sample scores below threshold. Individual sample scores fluctuate; aggregate scores across 100+ examples are far more stable and reliable.
• →Use a delta threshold relative to baseline: instead of an absolute score cutoff, block the deployment when the aggregate score drops more than a defined percentage below the last passing baseline. This approach catches genuine regressions while tolerating natural score variance. Store the baseline score from the last green main-branch commit as a versioned artifact.
• →Sample consistency check for critical metrics: for high-stakes metrics like hallucination rate, run the eval twice on the same dataset within a single CI run and average the two passes before comparing to threshold. This adds cost but dramatically reduces false positives from high-variance judge calls on edge-case inputs.
• →Separate blocking evals from informational evals: not all metrics need to block deployment. Faithfulness and hallucination rate are blocking metrics for a RAG application. Verbosity, formality, or tone-matching scores are informational — track them as trends in a dashboard but do not let them block the build.
• →Set the evaluator model to temperature 0: in CI/CD contexts, reproducibility matters more than diversity. Setting the judge model to temperature 0 maximizes score consistency across runs, which makes thresholding and delta comparisons reliable.

Controlling Evaluation Cost in CI/CD Pipelines

A CI/CD eval pipeline that runs on every commit against a 500-example dataset with an LLM judge accumulates significant API cost quickly. Evaluation cost is a first-class engineering concern — ignoring it leads to either a prohibitively expensive pipeline or pressure to skip evals when quarterly budgets tighten. Our guide to agentic AI cost governance at enterprise scale covers the attribution and budget enforcement patterns that apply directly to eval infrastructure.

• →Tier evals by trigger: run the full golden dataset eval only on merges to main; run a fast subset of 20 to 50 examples on every pull request. The fast subset should include your highest-regression-risk examples — known past failures and a stratified sample across input classes.
• →Use a smaller model for simple metrics: deterministic checks cost nothing. LLM-as-judge evals for simple metrics like answer relevancy on short outputs run reliably on smaller, cheaper models. Reserve the strongest judge model for hallucination detection and complex multi-step reasoning evaluation.
• →Cache eval results for unchanged components: if a code change does not touch the retrieval pipeline, skip re-running faithfulness metrics that depend only on retrieval quality. Cache eval results at the component level and invalidate only when the component's code, configuration, or prompt changes.
• →Run evals only on diff-affected pipelines: for large LLM applications with multiple distinct pipelines, use file-level change detection in CI to run only the evals for the pipelines whose code, prompts, or retrieval configuration changed. A full eval sweep on every pipeline for every commit is rarely justified.
• →Self-hosted evaluator models for high-volume pipelines: for teams with strict data governance requirements or high eval volume, self-hosting a judge model eliminates per-call API costs and keeps evaluation data off third-party infrastructure. At CI/CD scale with dozens of runs per day, the throughput and cost advantages compound quickly.

Frequently Asked Questions

What is LLM evaluation in CI/CD?

LLM evaluation in CI/CD is the practice of automatically running quality checks against LLM application outputs on every code change — before the change reaches production. Each eval run scores a batch of representative inputs from a golden dataset using deterministic checks and LLM-as-judge metrics. A CI/CD quality gate blocks the deployment if aggregate scores fall below a defined threshold, catching prompt regressions and model drift before users are affected.

When should LLM evals block a deployment?

Block a deployment when your highest-risk quality metrics — typically faithfulness for RAG systems, hallucination rate for factual applications, and answer relevancy for Q&A — drop by more than a defined percentage relative to the established baseline on the main branch. Not all evals should be blocking: use blocking gates only for metrics that directly affect user trust, safety, or revenue impact, and track secondary metrics like tone or verbosity as informational trends.

What is LLM-as-judge evaluation?

LLM-as-judge is a technique where a second language model — separate from the one powering your application — scores the quality of application outputs against a defined rubric. The judge receives the input, the application's output, and optionally a reference answer or retrieved context, then returns a numeric score and brief reasoning. It is the only scalable approach to measuring semantic quality automatically, because deterministic functions cannot assess whether an answer is accurate, faithful, or contextually appropriate.

How do you handle non-determinism in LLM eval quality gates?

Gate on aggregate scores across the full golden dataset rather than individual sample pass/fail. Use a delta threshold that compares the current run's aggregate score to the last green baseline rather than a fixed absolute threshold. Set the judge model to temperature 0 for consistent scoring. For high-stakes metrics, run the eval twice within the same CI run and average the results before comparing to threshold. Together, these practices make the gate stable across natural output variance while still detecting genuine regressions.

What metrics should an LLM eval pipeline track?

Core blocking metrics for most production applications: answer relevancy (does the output answer the question asked?), faithfulness (is the output grounded in provided context, for RAG systems?), and hallucination rate (does the output assert facts unsupported by context or known ground truth?). Track latency per pipeline component and cost per eval run as operational metrics. Add task-specific dimensions using G-Eval — for example, a structured extraction pipeline should also track schema compliance and field-level accuracy.

How Belsoft Helps Teams Build LLM Quality Gates

Shipping an LLM application without automated quality gates is the engineering equivalent of shipping software without tests — it works until it does not, and you find out in production. At Belsoft, the AI & Automation engineering team builds LLM evaluation infrastructure as part of every AI product engagement: golden dataset design, LLM-as-judge calibration against human labels, CI/CD framework integration, and production monitoring pipelines. We scope the eval layer to the actual risk profile of your application — not the maximum theoretical coverage.

If your team is shipping LLM features without confidence that each deployment is at least as good as the last, book a call with Belsoft and we will walk through what a practical eval pipeline looks like for your specific application and team structure.

“An LLM application without evals in CI/CD is not a product — it's a demo you're charging for.”