Is Your Agent Skill Actually Good? Microsoft's Dual-Paper Deep Dive into Skill Evaluation and Self-Evolving Optimization

WonderLab Posted on May 31 Is Your Agent Skill Actually Good? Microsoft's Dual-Paper Deep Dive into Skill Evaluation and Self-Evolving Optimization # ai # llm # agents # skills The Question Nobody Wants to Ask: Does Your Skill Actually Help? You spent an afternoon crafting a carefully structured Skill for your agent. Clear steps, thorough edge-case notes, well-formatted output requirements. You tested it manually a few times, the outputs looked great. You shipped it. Three weeks later, you notice that some task success rates have gone down compared to before the Skill existed. This is not a hypothetical. In May 2026, Microsoft Research published two concurrent papers — SkillLens ("From Raw Experience to Skill Consumption") and SkillOpt ("Executive Strategy for Self-Evolving Agent Skills") — that measured this failure mode at scale. Their finding: negative transfer happens in 25% of cases , and you cannot reliably identify the bad skills just by reading the text. One paper answers "why skills sometimes backfire." The other answers "how to make skills systematically better." Together they sketch a new paradigm for agent capability improvement. Part One: SkillLens — Mapping the Full Skill Lifecycle A Skill Is Not a Point — It's a Pipeline Most practitioners think of a Skill as "a block of text instructions for an agent." SkillLens decomposes this into a three-stage lifecycle : Stage 1: Experience Generation Target model M runs training tasks, producing an experience pool of trajectories (both successes and failures) ↓ Stage 2: Skill Extraction Extractor model E distills the experience pool into a structured skill document — procedural knowledge under a fixed budget ↓ Stage 3: Skill Consumption The same target model M, equipped with the extracted skill, is evaluated on held-out test tasks Enter fullscreen mode Exit fullscreen mode Notice there are two distinct roles in this chain: the Extractor (distills knowledge from trajectories) and the Target (consumes knowledg