What is the OpenAI Evals framework and why is evaluation critical for production applications?

Evals (evaluations) are automated tests that measure an LLM application's quality, accuracy, and reliability. OpenAI provides both an Evals API (for running evaluations programmatically) and the OpenAI Evals Framework (open-source, run locally). Without systematic evals, you cannot confidently iterate on prompts, upgrade models, or know if your application is regressing.

from openai import OpenAI
client = OpenAI()

# Using the Evals API:
# 1. Create a dataset for evaluation:
dataset = client.evals.datasets.create(
    name="code-review-eval",
    items=[
        {"input": "Review: def add(a,b): return a-b", "expected": "Bug: subtraction used instead of addition"},
        {"input": "Review: import os; os.system(user_input)", "expected": "Security: OS injection vulnerability"},
    ]
)

# 2. Create and run an eval:
eval_run = client.evals.runs.create(
    name="code-reviewer-v2-test",
    model="gpt-5.5",
    data_source={"type": "dataset", "id": dataset.id},
    grader_config={
        "type": "llm_as_judge",
        "model": "gpt-5.4-mini",  # fast/cheap grader
        "criteria": "Does the review correctly identify all bugs and security issues?"
    }
)
print(f"Score: {eval_run.result.score}")

# Open-source evals (run locally):
# pip install evals
# oaieval gpt-5.5 my-code-review-eval
# Compare two models:
# oaieval gpt-5.5 my-eval
# oaieval gpt-5.4 my-eval

When to evaluate:

• Before deploying any prompt change to production
• Before upgrading to a new model version
• After any significant change to the application logic
• Periodically in production to detect model drift

Good evals catch regressions before users do and give you confidence to upgrade models without fear of breaking existing behaviour.

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.