SDK reference¶

The Python SDK runs experiments from your own code. Install it with the otel extra at your server's version (see Install) and import from evalkit.sdk. SDK runs walks through it.

Running experiments¶

run_experiment ¶

run_experiment(
    *,
    dataset: str,
    agent: str,
    task: Task,
    workspace: str | None = None,
    evaluators: Sequence[PythonEvaluator] = (),
    repetitions: int = 1,
    concurrency: int | None = None,
    case_ids: Sequence[str] | None = None,
    split: str | None = None,
    model: str | None = None,
    resume: int | None = None,
    dry_run: bool | int = False,
    trajectory_wait_seconds: int | None = None,
    judge: JudgeSettings | None = None,
) -> ExperimentResult

Run an experiment: call task once per case and repetition, store the runs on the server and score them.

Scoring uses the judge, the remote evaluators of the agent's workspace and the Python evaluators you pass. Reads evalkit.yaml (cases, fixtures, credentials) from the current directory or a parent, and the login from evalkit login.

Parameters:

dataset (str) –

The dataset to run. A bare name means your personal workspace; if it is also a directory under cases in evalkit.yaml, it is synced first. name@workspace runs a shared dataset, every review state included.
agent (str) –

The code-driven agent on the server that the runs belong to, created once with evalkit agents create. A bare name means your personal workspace.
task (Task) –

Your function, called once per case and repetition, sync or async. Its parameters bind by name from input (the case input), files (list[Fixture]), model and metadata (session_id, experiment_id, case_id, repetition, tags). A task with one parameter of another name receives the input. Return a str (the answer), a dict (the structured output) or a RunResult. A task that raises records a failed run.
workspace (str | None, default: None ) –

The workspace of dataset when it is a bare name.
evaluators (Sequence[PythonEvaluator], default: () ) –

Python evaluators made with evaluator. A case selects one by name under evaluators.custom, like an HTTP evaluator.
repetitions (int, default: 1 ) –

How many times each case runs.
concurrency (int | None, default: None ) –

How many tasks run at once. Defaults to the agent's max_concurrent_runs.
case_ids (Sequence[str] | None, default: None ) –

Run only these cases.
split (str | None, default: None ) –

Run only the cases of this split (a case tag).
model (str | None, default: None ) –

Passed to the task's model parameter and recorded on the experiment.
resume (int | None, default: None ) –

An experiment id to resume: runs only the cases it has not finished.
dry_run (bool | int, default: False ) –

True runs one case without creating an experiment and prints the answer and what the server made of the session; an int runs that many.
trajectory_wait_seconds (int | None, default: None ) –

How long to wait for a run's spans to settle into a trajectory. Defaults to the agent's setting.
judge (JudgeSettings | None, default: None ) –

The judge for LLM-graded evaluators. Defaults to the server's default judge. Its API key comes from credentials.judges in evalkit.yaml, keyed by base_url.

Returns:

ExperimentResult –

The experiment id, its URL and one summary per run.

Raises:

RuntimeError –

When called from a running event loop; await arun_experiment there instead.

arun_experiment `async` ¶

arun_experiment(
    *,
    dataset: str,
    agent: str,
    task: Task,
    workspace: str | None = None,
    evaluators: Sequence[PythonEvaluator] = (),
    repetitions: int = 1,
    concurrency: int | None = None,
    case_ids: Sequence[str] | None = None,
    split: str | None = None,
    model: str | None = None,
    resume: int | None = None,
    dry_run: bool | int = False,
    trajectory_wait_seconds: int | None = None,
    judge: JudgeSettings | None = None,
    api_transport: AsyncBaseTransport | None = None,
    custom_eval_transport: AsyncBaseTransport | None = None,
    progress: Progress = print,
    poll_interval: float = 5.0,
    tracer_provider: Any = None,
) -> ExperimentResult

The async form of run_experiment, for code already in an event loop (a notebook).

Same arguments and result as run_experiment. The arguments after judge exist for tests.

evaluate_experiment ¶

evaluate_experiment(
    *,
    experiment_id: int,
    evaluators: Sequence[PythonEvaluator] = (),
    refetch_trajectories: bool = False,
) -> ExperimentResult

Re-score an experiment that used Python evaluators.

evalkit evaluate refuses such experiments, because the evaluators live in your script; call this from the script instead. Exported as evaluate_experiment.

Parameters:

experiment_id (int) –

The experiment to re-score.
evaluators (Sequence[PythonEvaluator], default: () ) –

The Python evaluators the experiment ran with. Each must have been part of the experiment.
refetch_trajectories (bool, default: False ) –

Read the trajectories of runs whose trajectory was partial or unavailable again before scoring.

Returns:

ExperimentResult –

The experiment id, its URL and one summary per run.

Raises:

RuntimeError –

When called from a running event loop; await aevaluate_experiment there instead.

aevaluate_experiment `async` ¶

aevaluate_experiment(
    *,
    experiment_id: int,
    evaluators: Sequence[PythonEvaluator] = (),
    refetch_trajectories: bool = False,
    api_transport: AsyncBaseTransport | None = None,
    custom_eval_transport: AsyncBaseTransport | None = None,
    progress: Progress = print,
) -> ExperimentResult

The async form of evaluate_experiment, for code already in an event loop.

Same arguments and result as evaluate_experiment. The arguments after refetch_trajectories exist for tests.

Tracing¶

register ¶

register(
    agent: str, only: SpanFilter | None = None
) -> None

Send this process's OpenTelemetry spans to the evalkit server.

Adds an exporter to the global tracer provider, or creates one when none is set. Call it after your own tracing setup, so it joins your provider. Spans go to the server you logged in to (or EVALKIT_SERVER_URL), with your token or EVALKIT_OTEL_TOKEN (an agent token). Needs the otel extra.

Parameters:

agent (str) –

The agent the spans belong to, e.g. myagent or myagent@dev.
only (SpanFilter | None, default: None ) –

Export only the spans this function returns True for.

Evaluators¶

evaluator ¶

evaluator(fn: Callable[..., Any]) -> PythonEvaluator

evaluator(
    *,
    name: str | None = None,
    kind: Kind = "evaluator",
    requires_trajectory: bool = False,
) -> Callable[[Callable[..., Any]], PythonEvaluator]

evaluator(
    fn: Callable[..., Any] | None = None,
    /,
    *,
    name: str | None = None,
    kind: Kind = "evaluator",
    requires_trajectory: bool = False,
) -> (
    PythonEvaluator
    | Callable[[Callable[..., Any]], PythonEvaluator]
)

Turn a function into a Python evaluator for run_experiment.

Use it bare (@evaluator) or with options (@evaluator(name=...)). The function receives the same EvalCase and EvalRun the Custom Evaluator API sends: the case id, input, expected block and this evaluator's config from the case YAML (case.config); the answer, structured output and ATIF trajectory as a dict. It returns a float, a ScoreItem, or a list of them.

Parameters:

fn (Callable[..., Any] | None, default: None ) –

The function, when used bare.
name (str | None, default: None ) –

The name cases use under evaluators.custom. Defaults to the function's name.
kind (Kind, default: 'evaluator' ) –

evaluator for a score, gauge for a measurement.
requires_trajectory (bool, default: False ) –

Skip runs that have no trajectory.

Returns:

PythonEvaluator | Callable[[Callable[..., Any]], PythonEvaluator] –

The evaluator, or a decorator that makes one.

Types¶

RunResult ¶

What a task can return instead of a plain answer.

Attributes:

answer (str | None) –

The agent's answer text.
output (dict[str, Any] | None) –

The agent's structured output.
model (str | None) –

The model the agent used, recorded on the run.
error (str | None) –

An error message; the run is recorded as failed.
session_id (str | None) –

The agent's own session id, when its spans do not continue the SDK's trace. The SDK reads the trajectory from that session.
trajectory (dict[str, Any] | None) –

An ATIF document to store as the trajectory, instead of building one from spans.

Fixture ¶

A case's fixture file, downloaded before the task is called.

Attributes:

name (str) –

The fixture key the case names.
path (Path) –

Where the file sits on this machine.
context (str | None) –

The file's description from the fixture registry, or None.

Judge ¶

The LLM judge an experiment uses; exported from evalkit.sdk as Judge.

Its API key comes from credentials.judges in evalkit.yaml, keyed by base_url.

Attributes:

base_url (str) –

An OpenAI-compatible endpoint that supports structured output.
model (str) –

The model name at that endpoint.
reasoning_effort (str) –

Passed to the model as reasoning_effort.
max_tokens (int) –

The most tokens one judge reply may use.
concurrency (int) –

How many judge calls run at once.

ExperimentResult ¶

The result of run_experiment or evaluate_experiment.

Attributes:

experiment_id (int | None) –

The experiment, or None after a dry run.
url (str | None) –

The experiment's page in the web UI, or None after a dry run.
runs (list[RunSummary]) –

One entry per case and repetition.

RunSummary ¶

One run of an experiment.

Attributes:

case_id (str) –

The case.
repetition (int) –

1 for the first repetition, 2 for the second, and so on.
status (str) –

completed, failed, cancelled or timeout.
trajectory_status (str | None) –

valid, partial, invalid or unavailable, or None when no trajectory was read.
session_id (str | None) –

The session the run's spans belong to.

SDK reference¶

Running experiments¶

run_experiment ¶

arun_experiment async ¶

evaluate_experiment ¶

aevaluate_experiment async ¶

Tracing¶

register ¶

Evaluators¶

evaluator ¶

Types¶

RunResult ¶

Fixture ¶

Judge ¶

ExperimentResult ¶

RunSummary ¶

arun_experiment `async` ¶

aevaluate_experiment `async` ¶