LifeHubber. Resources Use AI on your terms.
LifeHubber.Resource
Opening…

AI Resources

Tracely

GitHub stars: 948 GitHub forks: 85 Declared license: MIT: MIT Last pushed November 9, 2026: Pushed 7d ago
Stats from GitHub

Tracely is a self-hostable toolkit for tracing, evaluating and regression-testing AI agents. It connects recorded runs, failure analysis, test cases and checks on proposed code changes in one workspace.

Its central workflow starts with something an agent actually did: inspect a failing run, define what should have happened, and use that case when checking a later candidate version.

What it is

Agent runs, evaluations and regression cases

A trace records the steps of an agent run, including model and tool activity. Tracely displays those runs, applies structural checks or model-based evaluators, groups failures into issues, and lets a developer promote a failing trace into a test case.

Why it stands out

The failure supplies the test input

A promoted case keeps the original request and recorded tool and model calls alongside a contract describing the expected behaviour. The case documentation requires that the original run fail that contract. A developer can then check a candidate against the same expectations.

Availability

Public source, SDK and hosting guides

The project publishes application source, a Python SDK and CLI, and self-hosting instructions; a hosted service is also offered. It declares an MIT licence; see the main repository for current terms. The repository includes a populated Docker demo for exploring the workflow.

Why it matters

What makes it useful

Suppose an agent ignores a failed lookup and continues with an answer. Its recorded calls and errors give a developer a specific incident to examine. Tracely's case workflow can preserve that input, the tool responses and expectations such as a required tool call or error-handling rule. Replay history and the original-versus-candidate view help follow the attempted fix rather than losing the incident among newer traces.

Notable points

What stands out

Recorded replay runs the agent's code with saved tool and model outputs. It can exercise orchestration and error handling under those recorded conditions. Changing a prompt still serves the old recorded model answer, so that replay cannot establish whether the new prompt produces a better answer. The replay guide describes live runs and recapturing a case when the fix needs new calls or new responses.

Before using

What to review

The local Docker setup is a development environment. The deployment runbook identifies its open development-authentication mode and published demo ingest key, and documents the production settings that replace them.

Traces can contain requests, messages, tool arguments and results. The self-hosting guide says model-based judges use a workspace's OpenRouter key; structural checks can run without it. Review what the chosen instrumentation records and which evaluators will send content to a model provider.

Replay only substitutes recorded outputs at supported call interfaces. The guide lists provider and streaming gaps that can remain live, and offers explicit call_tool and call_llm wrappers as the portable integration path. An API or framework being traced does not by itself establish that all its calls are intercepted during replay.

Check the original listing's documentation for current requirements, limitations, licensing and any relevant privacy or security considerations.

Reader fit

Who may find it relevant

Developers maintaining agent services and CI pipelines are the clearest audience, especially teams trying to keep a known failure from returning after a change. The source and demo also suit builders learning how trace inspection, test expectations and release checks fit together. Self-hosting involves operating several services as well as instrumenting the agent.

Editorial note

Why LifeHubber lists it

LifeHubber lists Tracely because it gives a reported agent failure a continuing role in development. The original request, a concrete expectation and a later candidate result can stay connected as the agent changes. For a team investigating a missed tool call or mishandled error, that connection helps explain why a test exists and what a proposed fix was checked against, beyond an overall quality score.

Source links

Source materials

Reader note

About our listings

LifeHubber’s listings help you explore what’s available in AI and are not endorsements. We don’t independently verify every detail or guarantee that each listing is accurate, safe, actively maintained, or right for your needs.

Advertisements

Advertisements

For project maintainers

Listed here? You can use the badge.

If you maintain a project with a current LifeHubber listing, you may add the optional “Listed on LifeHubber AI Resources” badge to its README, docs, or website. No introduction or permission request is needed.

MORE ON LIFEHUBBER

Use AI on your own terms.

Explore more AI projects, stories, guides, tools and community insights across LifeHubber.

See what’s moving