Datasets

Curate golden datasets straight from your real traffic, then re-run evaluation and security over them as a regression suite. Trace-backed examples capture the whole agent run, not just an input/output pair, so full agentic evaluation works offline, forever.

Build a dataset from traces

Open any run in the Traces drawer and click Add to Dataset. On a root span this captures the entire run; pick an existing dataset or create a new one inline. You can also add the currently open example from the Prompts playground.

Trajectory capture

When you add a trace-backed example, Fluiq pins the run's whole trajectory into a retention-independent store: every span, including LLM calls, agent/task steps, tool calls, MCP calls, the multi-agent DAG, and media references. Media is offloaded to object storage and re-linked on read. The pinned snapshot means a dataset run evaluates the exact trajectory even after the original trace has passed its retention window.

Why the whole trajectory?

A single input/output pair can only be scored as one turn. Pinning the full run lets Fluiq re-run agentic evaluation (tool-selection correctness, trajectory quality, and multi-agent coordination) and security, exactly as it would on a live trace. Expand any example in the Datasets dashboard to inspect the captured steps, agents, tools, and MCP calls.

Connect Agents

Rather than adding runs one at a time, click Connect Agents on a dataset and pick a traced agent. Fluiq imports every run of that agent to date (deduplicated, full trajectory pinned) and keeps the dataset in sync; future runs of a linked agent are appended automatically.

Batch evaluation & security

From a dataset you can launch three kinds of batch run over every example: agentic evaluation (tool selection, trajectory, coordination against the pinned trajectory), security (risk level and detections), and a metrics run that grades each example's recorded answer against its expected_output with the metrics you pick (hallucination, faithfulness, relevance, toxicity, coherence, completeness). Results roll up into a per-run report with per-metric averages and per-example scores.

Compare runs. Every report has a Compare vs… selector: pick any earlier run of the same kind and Fluiq diffs them — per-metric score deltas (computed only over examples present in both runs) and every example classified as regressed, improved, or unchanged. That's your regression gate before shipping a prompt or model change; the same check runs headless in CI via python -m fluiq.ci (see Evaluation).

Programmatic access

Datasets are also reachable over the API: list datasets, add examples, and fetch an example's pinned trajectory.

Python

import os, requests

BASE = "https://api.fluiq.ai/api/v1"
H = {"Authorization": f"Bearer {os.environ['FLUIQ_API_KEY']}"}

# Create a dataset and add an example captured from a trace
ds = requests.post(f"{BASE}/datasets", headers=H,
                   json={"name": "checkout-regressions"}).json()

requests.post(f"{BASE}/datasets/{ds['dataset_id']}/examples", headers=H, json={
    "input": "Refund my last order",
    "expected_output": "Opened refund #4821",
    # link the run so Fluiq pins its full trajectory for agentic eval
    "metadata": {"source_trace_id": "0f9c...e21"},
})

Pass a source_trace_id in an example's metadata (use the run's root trace id) and Fluiq snapshots that run's whole trajectory into the dataset automatically.

import os, requests BASE = "https://api.fluiq.ai/api/v1" H = {"Authorization": f"Bearer {os.environ['FLUIQ_API_KEY']}"} # Create a dataset and add an example captured from a trace ds = requests.post(f"{BASE}/datasets", headers=H, json={"name": "checkout-regressions"}).json() requests.post(f"{BASE}/datasets/{ds['dataset_id']}/examples", headers=H, json={ "input": "Refund my last order", "expected_output": "Opened refund #4821", # link the run so Fluiq pins its full trajectory for agentic eval "metadata": {"source_trace_id": "0f9c...e21"}, })