AI-generated test cases require a Premium subscription synced directly from Stripe. Other Premium capabilities include roadmap work and are not newly gated. AI calls still require configuration, dollar budgets and consent; provider charges apply.
The essentials stay open. The workflow gets smarter.
Experiments are part of the core product. Premium is about going deeper and reducing repetitive work, not putting every AI judge behind a paywall.
Product direction only. Premium capabilities below include planned work, not a promise of availability today.
Capability
Free
Premium direction
Experiments
Manual runs and basic baseline/candidate comparisons
Multiple configurations, parameter sweeps, and repeated trials
Evaluations
Automatic agent runs, basic AI judging, deterministic checks and human feedback
Advanced evidence-based judges and judge calibration
Failure analysis
Inspect failed traces manually
Group recurring failures and suggest causes with trace evidence
Test creation
Write test questions, run your agent and judge captured answers automatically
AI drafts test questions and references for approval
Prompt improvement
Edit and version prompts
Generate candidates, benchmark variants, and recommend improvements
Production quality
Basic dashboards and captured quality signals
Continuous sampled evaluation, alerts, and spending controls
Release confidence
Inspect comparisons and regressions
CI regression gates and release approval workflows
Collaboration
Private workspace and individual review
Shared projects, assigned reviews, roles, and audit history
Cost insights
Inspect recorded token usage and cost
Identify costly loops and recommend cheaper configurations to test
On smaller screens, scroll the comparison sideways. Existing deployment controls still determine what can run today; this page does not change access or billing.
Premium roadmap
Find what failed. Generate a test. Prove the fix.
Production alerts, spending controls, bounded sweeps and team review are implemented. Other improvement workflows below remain planned. Labels distinguish working capabilities from roadmap direction.
01Implemented · bounded sweeps
Explore the configuration space
Advanced experiments
Queue pinned prompt/model/HTTP-agent variants and repeated trials. Compare quality, latency and known cost with exploratory paired confidence intervals; incomplete runs block conclusions. Retrieval variants require separately configured agent targets.
02Planned
Find the recurring pattern
AI failure analyst
Group recurring failures, suggest likely causes, and link findings to trace evidence. Treat explanations as hypotheses to verify, not automatic diagnoses.
03Premium · Stripe subscription
Generate. Review. Run. Evaluate.
AI-generated test cases
AI-written questions and suggested references are available with a synced Premium subscription. Entering your own questions, automatic HTTP agent runs and basic semantic AI judging are available on Free. Automatic failure-based test generation remains planned.
04Planned
Make the rubric clearer
Evaluation copilot
Get help drafting evaluation criteria and investigating disagreements between AI judges and human reviewers. Keep final rubric changes under human control.
05Planned
Suggest. Benchmark. Review.
Prompt optimization
Generate candidate prompts and compare quality, cost, and latency through experiments. Recommend a change only after checking regressions; never auto-promote it.
06Planned
Prove the change before release
CI regression gates
Run agreed checks in pull requests, report baseline/candidate differences, and block releases when quality thresholds regress. Require approval for important changes.
07Implemented · configure worker
Keep watching production
Continuous quality monitoring
Sample incoming traces, run background checks, and monitor quality failure rates. Deliver issue transitions by email or signed webhook, with atomic daily/monthly AI dollar budgets and a reviewed built-in model catalog.
08Implemented · in-app invitations
Improve together
Team review workflows
Share private team workspaces with Owner/Admin/Editor/Reviewer roles, assigned trace reviews, authenticated review attribution and action history. Existing verified accounts accept invitations in the app. No SSO or enterprise SLA is implied.
09Planned
Spend where it helps
Cost optimization insights
Identify expensive calls and excessive tool loops. Suggest cheaper configurations to benchmark against your quality baseline before switching models.
Experiments at the center
A recommendation is not a release.
Keep datasets and evaluation criteria fixed, compare per-example results, and review cost and latency. Automated suggestions should lead to an experiment and a human decision, not an unreviewed deployment.
More than a score. A reason to make the next change.
Go beyond a basic manual judge with calibration and evidence-based evaluation. Existing evaluators depend on deployment configuration; the proposed plans do not gate access today.
01
Implemented · structured captured turns
Conversation quality
Judge resolution, coherence and observed user frustration across captured conversations. Authorized HTTP-agent experiments replay sequential turns and retain candidate evidence; flattened or one-shot outputs do not masquerade as multi-turn runs.
02
Implemented · evidence-based, not certification
Safety sample checks
Evaluate observed toxicity and prompt-injection resistance with captured evidence, plus deterministic exact-sensitive-literal leakage against explicit ground truth. Missing or partial evidence stays unavailable; these checks do not certify safety or compliance.
03
Implemented evaluation workflow
Agent Quality
Score task completion, error recovery, required tools, exact tool arguments, tool-call budgets, and repeated calls against pinned experiment evidence. AI judges require configuration and consent.
04
Implemented with human review
Pairwise preference
Judge baseline and candidate outputs in both presentation orders, retain ties and disagreements, and review pinned evidence. Approval requires human decisions and passing candidate quality scores; it never deploys a change.
05
Align AI and human judgment
Judge calibration
Compare judge results with human scores, inspect disagreements, and tune thresholds with evidence. Basic AI judging is available on Free when configured.
06
Evidence, not guesswork
Groundedness
Check whether response claims are supported by captured documents. Surface unsupported statements instead of trusting a convincing answer.
07
On-topic responses
Answer relevance
Use a model-based judge to assess whether the response addresses the user's actual question, beyond exact wording or keyword matches.
08
Check the claim, not just the link
Citation support
Look beyond valid citation IDs. Evaluate whether the referenced documents actually support the claims they accompany.
09
Better context, better answers
Retrieval relevance
Judge whether retrieved documents are useful for the question. Separate helpful context from results that only look similar.
10
Understand the whole run
Agent evidence review
Review tool use, failures, missing evidence, and uncertainty before a final quality judgment. Evaluate the path, not just the final response.
Keep quality in the loop
Bring AI judges to incoming traces.
Background evaluation rules can apply your criteria to new runs. Keep dataset versions, judge criteria, and model choice explicit so results stay interpretable.
AI judgments are signals, not guarantees. Results depend on the judge model and available evidence. Review important decisions with a human.
Premium upgrade
Upgrade to Premium
Subscribe without leaving Tracework. Payment details are collected securely by Stripe in the embedded checkout below.
Premium is granted only after the server retrieves an active or trialing subscription directly from Stripe. Manual questions, automatic agent runs, basic AI judging and captured-evidence workflows remain available on Free with configuration and consent.
A saved agent stays available for as long as this subscription is paid. There is no other time limit. Scratch tests still expire. Cancelling the host fee keeps the saved agent and stops new runs until hosting is paid again.
A little more clarity
Good questions. Straight answers.
What the plans mean, how experiments and evaluation work, and what is available today.
What is the difference between Free and Premium?+
Free lets you enter test questions, run a configured agent, capture its answers and latency, and evaluate their meaning with a basic AI judge. References are optional; judging criteria and consent are required. Provider charges and operator budgets apply. Premium adds AI-written test questions for workspaces with a synced active Stripe subscription. Other Premium descriptions remain product direction.
How are experiments different from evaluations?+
An evaluation scores an answer or run. An experiment executes a specific prompt, model, or agent configuration against a fixed dataset so you can test a change. Compare baseline and candidate runs with the same dataset version, evaluator, thresholds, and criteria, then inspect quality, regressions, latency, and recorded usage.
Which Premium workflows are available today?+
Tracing, versioned datasets/prompts, Agent Quality, pairwise review, conversation/safety sample checks, bounded sweeps and team review are implemented. Background checks, alerts and hard AI dollar controls require operator configuration. Task-based AI case drafts are available to Premium workspaces. Automatic failure-based tests, failure grouping, evaluation copilot, prompt optimization, integrated PR gates and AI cost recommendations remain planned. Confidence is exploratory and safety checks are not certification.
Can I purchase Premium now?+
Yes, when Stripe is configured for this deployment. Sign in, complete the embedded Stripe checkout, and Tracework will retrieve and sync the resulting subscription before enabling Premium.
Are model-provider costs included?+
No included AI allowance is promised. AI evaluation requires operator-enabled model access and explicit approval to send the selected data and accept provider usage charges. Creating an account does not enable paid model calls.
Do AI judges need access to my agent's data?+
Yes. Depending on the evaluator, selected inputs, outputs, reference answers, retrieved documents, or tool evidence are sent to the configured model provider. Review the data before consenting. Deterministic checks do not require a model call.
Does Premium introduce evaluation limits?+
No plan-specific quotas are enforced. Operator and workspace AI dollar safety budgets, request/provider limits and storage constraints still apply. Models missing from Tracework's built-in price catalog remain blocked. These controls are not subscription allowances.