Choose theme
AI Resources
ClawMark
ClawMark is a benchmark for agents that work across files, email, calendars, and other tools while their task environment changes over several simulated working days.
An agent receives new instructions at each stage and must work with updated records and raw evidence, including images, documents, audio, and video. Use this as a first read, not a recommendation. Open the original project before trusting details like terms, limits, privacy, cost, setup, or safety.
What it is
A benchmark for coworker agents
The repository provides 100 tasks across 13 professional domains, tool environments, an agent runner, and scoring code for evaluating a model and its tool setup. It is not a standalone assistant app.
Why it stands out
The environment changes between stages
New emails, changed records, and updated files can arrive between stages. Information an agent used earlier may need to be checked again before it continues.
Availability
Repository with tasks and scoring framework
The public repository includes task assets and checkers, Docker setup, and run instructions. The technical paper explains the evaluation; a separate guide covers service credentials.
Why it matters
What makes it useful
If you are testing whether an agent keeps working from an outdated plan, ClawMark supplies tasks where outside records change independently of the agent. The next stage can require it to refresh a calendar, email, or database record before acting on earlier information.
What to know
Where it fits
Use it in an evaluation workflow where you choose a model, configure its tools, and run selected tasks. Its weighted score describes the checks passed in that setup; it does not establish that the agent can do the corresponding real-world job.
Notable points
What stands out
Python checkers inspect the resulting files and service state, without an LLM judge. That makes the work left behind part of the evaluation: a convincing final reply alone does not show that the required records were updated. Weighted partial credit can also differ from completing every requirement of a task. After a run, start with a failed check in result.json, follow the tool calls in messages.jsonl, then compare them with the saved workspace/ files. These outputs support investigating the attempt; they do not by themselves establish why it failed.
Before using
What to review
The full run needs model credentials and Docker services. The documented dry run checks task and framework plumbing without Docker; it is not the full evaluation.
The credentials guide uses live Notion and Google services. Notion has separate admin and agent integrations, and the evaluation workspace is cleared and reseeded during setup. Read that guide before connecting a workspace.
For score comparisons, check the run definition: avg@3 averages three independent runs per task, then averages across tasks. A single run is a different measurement.
Reader fit
Who may find it relevant
Researchers and builders comparing agent setups on tasks with changing records and multiple tools.
Readers who want to inspect task assets and the checks applied to completed work.
For trying an everyday assistant directly, this benchmark requires an evaluation setup first.
Editorial note
Why LifeHubber lists it
We list ClawMark for builders evaluating agents for recurring work that changes while it is underway. Its reporting example moves from spotting a falling completion rate to building a weekly report and investigating a mismatch flagged by a colleague. That puts carrying new evidence into an ongoing job inside the evaluation: an agent must connect the follow-up work, rather than merely answer each instruction in isolation.
Source links
Source materials
Reader note
Before relying on this entry
LifeHubber lists entries to help readers inspect AI projects, not to endorse them or prove they are safe, suitable, accurate, maintained, or right for a specific use. We do not verify every entry in depth. Before relying on anything listed, review the original materials, terms, privacy practices, limits, and risks that matter for your situation.
More in Datasets
Keep browsing this category
Explore more datasets.
ParseBench
run-llama/ParseBench
A document parsing benchmark for AI-agent workflows, focused on whether parsed PDFs preserve structure and meaning for downstream evaluation.
UltraData-SFT-2605
openbmb/UltraData-SFT-2605
An OpenBMB supervised fine-tuning dataset with 15,036,178 thinking and non-thinking samples across math, code, knowledge, Chinese, instruction-following, and multilingual configurations, used in MiniCPM5-1B-SFT post-training.
Monitorability Evals
openai/monitorability-evals
An OpenAI evaluation-data release for studying monitorability, with public eval splits, prompt templates, dataset mappings, and metric code from the Monitoring Monitorability paper.
For project maintainers
Listed here? You can use the badge.
If you maintain a project with a current LifeHubber listing, you may add the optional “Listed on LifeHubber AI Resources” badge to its README, docs, or website. No introduction or permission request is needed.
