Introduction
Software engineers have a knack for automating repetitive tasks—even the intellectual ones. As an AI researcher on the Copilot Applied Science team, I built a system called eval-agents to automate the analysis of coding agent trajectories. These trajectories are detailed JSON logs of how agents solve evaluation tasks from benchmarks like TerminalBench2 or SWEBench-Pro. By following a structured approach, you can build similar agent-driven tools that amplify your productivity and make it easy for your team to contribute. This guide walks you through the entire process, from identifying the right task to sharing your creation.
What You Need
- GitHub Copilot – installed and configured in your preferred IDE (VS Code, JetBrains, etc.)
- GitHub account – for version control and sharing agents
- Familiarity with coding agents – basic understanding of what agents are and how they produce trajectories
- Access to evaluation datasets – such as TerminalBench2 or SWEBench-Pro (or any similar benchmark you work with)
- Python (or your language of choice) – to write agent code; examples here use Python
- GitHub CLI (optional but helpful) – for managing repositories and workflows
Step-by-Step Guide
Step 1: Identify a Repetitive Cognitive Task
Start by pinpointing a mental chore you perform repeatedly. In my case, analyzing hundreds of thousands of lines of trajectory JSON files to evaluate agent performance was the toil. Look for patterns where you ask the same questions of data or code each time—questions like “Which tasks did the agent fail?” or “Are there common mistakes?” This is your automation opportunity.
Step 2: Use Copilot to Surface Patterns
Before automating, let GitHub Copilot help you understand the data. Open a few trajectory files and prompt Copilot with questions:
- “Summarize the actions taken in this trajectory.”
- “Highlight any error messages.”
- “Compare this trajectory to the expected solution.”
Copilot will generate code snippets (e.g., in Python) to parse and analyze the JSON. Use these to reduce the data you need to read manually—from thousands of lines to a few hundred. Document the patterns you discover; they’ll become the core logic for your agent.
Step 3: Define Clear Goals for Your Agent
With patterns in hand, set objectives for your agent. My guiding principle was that engineering and science teams work better together, so I aimed for three goals:
- Easy to share and use – agents should be accessible via GitHub.
- Easy to author new agents – lower the barrier for team contributions.
- Coding agents as primary vehicle – focus on code, not configuration.
Write these goals down—they’ll shape design and implementation.
Step 4: Design for Collaboration and Reuse
Now architect your agent. Use modular components: a data parser, analysis functions, and an output formatter. Make sure your code:
- Accepts input trajectories as files or from a directory.
- Outputs results in readable formats (e.g., markdown tables, CSV).
- Includes documentation (README) and examples.
Leverage GitHub Copilot while designing—ask it to generate boilerplate or suggest patterns for modularity. This step is where your earlier Copilot experiments pay off.
Step 5: Implement Your Agent with Copilot Assistance
Start coding. Use Copilot as your pair programmer:
- Begin with a comment describing a function, e.g., “
def analyze_trajectory(filepath):”. - Let Copilot suggest the implementation based on the patterns from Step 2.
- Iterate: test on a small set of trajectories, tweak prompts, and regenerate code.
For example, to parse JSON and extract task outcomes, write a comment like:
# Load JSON, list all tasks that have status 'failed'
Copilot will fill in the logic. Accept or modify suggestions to fit your exact needs.
Step 6: Test and Iterate
Run your agent against multiple benchmark runs. Check:
- Does it correctly identify failures?
- Is the output easy to understand?
- How long does it take?
Use Copilot to help debug—ask it to explain unexpected outputs or add error handling. You may find the agent over- or under-generalizes; adjust your prompts and logic accordingly.
Step 7: Share and Enable Your Team
Push your code to a public or internal GitHub repository. Add clear instructions: how to install dependencies, run the agent, and interpret results. Encourage team members to fork and extend the agent for their own analyses. The real power emerges when others contribute—suddenly your tool handles new benchmarks or reports new metrics.
Tips for Success
- Start small – automate one repetitive question first. You can always add features later.
- Embrace Copilot’s iterative nature – refine prompts as you learn what works. Copilot gets better with context.
- Document as you go – use inline comments and a README. This helps others (and your future self) understand the agent’s logic.
- Test with real data – agents behavior can vary across datasets. Validate on diverse examples.
- Encourage contributions – make it easy for teammates to create new agents by providing templates and examples. This transforms your personal automation into team superpower.
- Monitor and maintain – just like any software, your agent may need updates as benchmarks or Copilot evolves. Schedule occasional reviews.
By following these steps, you too can automate your intellectual toil and build tools that unlock faster, more collaborative research and development. Happy building!