EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments
EdgeBench introduces a comprehensive suite of 134 realistic, long‑horizon tasks covering scientific reasoning, formal verification, knowledge retrieval, system optimization, and game playing. Using the SForge harness, agents are evaluated on day‑scale challenges that require tens to hundreds of hours of human effort, revealing clear scaling laws across model size, compute, and environment interaction. The benchmark provides both a public 51‑task subset and a full private suite, establishing a new standard for measuring long‑term, real‑world learning in AI systems.
Project Ideas
- Create a curriculum learning schedule that gradually increases task difficulty based on the observed scaling laws.
- Develop a meta‑learning framework that leverages experience from solved tasks to accelerate learning on new, unseen tasks.
- Implement a hierarchical planning architecture that decomposes day‑scale challenges into reusable sub‑goals across categories.
- Integrate a memory‑augmented module to retain and retrieve knowledge from prior tasks, reducing redundant learning effort.
- Design an adaptive compute allocation strategy that dynamically scales resources according to task complexity and progress.