Annotate

Maker

📌

Why is this important? Every company will be an AI agent company. But building agents is tough, and knowing if they’re actually working well is even harder. Can your agent give accurate answers? Is it handling tasks like generating code or analyzing contracts the way you’d expect? Most businesses don’t have a clear way to figure this out—they only realize something’s wrong when customers complain or workflows start breaking. Without the right tools, it’s a guessing game. Foundry takes out the guesswork, helping you build agents that work as they should and keep getting better, so you can trust them to run key parts of your business. Our solution Foundry is a platform that helps businesses build, evaluate, and improve AI agents: For New Agents: We help companies design modular agents tailored to their workflows, capable of automating tasks like analyzing contracts, generating code, or resolving customer queries across languages. For Existing Agents: Foundry provides tools to systematically evaluate and improve agents, measuring dimensions like accuracy, relevance, task completion rates, and code execution reliability. Using a SOTA factuality checker, internal knowledge bases, historical data, and evaluation datasets, we identify where agents fall short. We then improve their performance through auto-prompting, fine-tuning, or steering—automating over 90% of what businesses would otherwise handle manually. Moving forward We’re building more than just tools for evaluating and improving AI agents—our vision is to create the operating system for AI agents. Imagine a marketplace where businesses can discover, compare, and instantly deploy the best AI agents for any task, complete with performance leaderboards and seamless drag-and-drop integration into workflows. Foundry will become the App Store for AI agents, enabling companies to choose top-performing agents to automate everything from customer service to sales to complex operations. Long-term, Foundry will act as the orchestration layer for all AI agents. Businesses will be able to manage fleets of agents across functions, ensuring they collaborate effectively, improve autonomously, and work as a cohesive ecosystem. Team Manil and Pranav worked on the Gen AI team at Scale AI, where they helped top AI labs build and improve their models. Manil was an operator that led large-scale data projects. Pranav, an ML Researcher, developed new methods of human supervision, built production ML tools to optimize Scale’s data pipelines, and led the agentic tool use research for the SEAL Leaderboard, the gold standard for benchmarking AI agents. After working closely with AI labs, we saw the massive potential of Gen AI agents to transform businesses—and the lack of tools to design, evaluate, and improve them effectively.

Report

9mo ago