AFTER: A Benchmark for Skill Evolution in Agentic Frameworks
AFTER (Agentic Frameworks for Skill Evolution and Transfer) is a newly released benchmark dataset designed to evaluate how well autonomous agents can evolve, specialize, and reuse skill instructions across a variety of software‑engineering and data‑centric workflows. Authored by DavydenkoGr and documented in arXiv:2606.23127, the dataset comprises 382 tasks spanning six roles—software engineering (swe), data engineering (ds), data engineering (de), generative AI (genai), infrastructure (infra), and project management (pm)—and 22 reusable skill surfaces such as API design, Terraform, refactoring, and RAG. Each task follows a strict visibility contract, exposing only an instruction, generated input data, and an output directory to the agent, while reference solutions, verifiers, and provenance files remain hidden on the oracle side.
The benchmark emphasizes three research questions: (1) can a framework improve a skill after task experience, (2) does that improvement transfer across tasks, roles, and execution contexts, and (3) does the method respect task boundaries. Tasks are categorized by difficulty (165 easy, 126 medium, 73 hard, 18 extra‑hard) and include both single‑skill and multi‑skill scenarios (64 multi‑skill tasks). The repository provides a clear layout with skill definitions, task manifests, and oracle‑side generators and verifiers, facilitating both direct task evaluation and longitudinal skill‑evolution studies. AFTER is intended for research on agentic procedural memory, prompt and tool instruction evolution, and cross‑domain transfer, rather than serving as a traditional model leaderboard.
Project Ideas
- Implement a new skill‑evolution algorithm and benchmark its performance against existing methods using the full AFTER suite.
- Fine‑tune a large language model to generate updated skill bodies after each task episode and measure cross‑role transfer on the benchmark.
- Create a visualization dashboard that maps skill coverage, task difficulty, and transfer success across the six roles in AFTER.
- Develop a lightweight role‑specific subset of AFTER (e.g., only the data‑engineering tasks) for rapid prototyping of skill‑evolution pipelines.
- Build an automated evaluation harness that runs the AFTER evaluation protocol and reports detailed metrics for skill improvement and transfer.