Lead Product Manager, Agent Runtime & Evals
Free account needed to apply or save. It takes a minute, and we'll bring you right back to this job.
Skills
About the role
This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Lead Product Manager, Agent Runtime & Evals based in United States.
We are looking for an experienced product leader to shape the foundations of an innovative AI platform designed to build, deploy, and manage intelligent agents at scale. In this role, you will own the agent runtime and evaluation infrastructure, enabling non-technical professionals to turn business needs into reliable, governed automations. You will define the product vision and roadmap across agent lifecycles, model governance, testing, deployment, and monitoring. Working closely with engineering, design, and product teams, you will solve complex platform challenges and establish scalable standards for AI-powered workflows. This is a hands-on leadership opportunity in an early-stage platform environment where experimentation, technical curiosity, and strong product judgment are essential. Your work will help transform traditional business operations into intelligent, agent-driven processes.
Accountabilities:
-
Own the vision, product strategy, and roadmap for the agent runtime platform, enabling non-technical users to create, test, publish, deploy, monitor, improve, and retire AI agents and automations without requiring traditional engineering support.
-
Lead the development of agent lifecycle management capabilities, including version control, deployment processes, runtime management, observability, and continuous improvement.
-
Define and evolve evaluation and model governance frameworks, including golden datasets, data labeling, automated evaluation, regression testing, and quality gates to ensure automations meet required standards before production.
-
Establish evaluation mechanisms for AI agents that assess other agents, ensuring quality, reliability, and measurable performance across large-scale automation environments.
-
Shape agent memory, learning, and self-improvement capabilities, ensuring that agents can retain relevant context and improve through experience while delivering measurable user value.
-
Lead the development of a model-agnostic platform, defining model routing, cost attribution, and pricing strategies that allow teams to select appropriate models for different use cases.
-
Define integration strategies that enable agents to securely access internal systems, business data, and third-party applications through reusable connectors, credentials, permissions, and access controls.
-
Establish agent identity and execution models, including user-based permissions and independent service identities, while ensuring appropriate governance and auditability.
-
Develop oversight and monitoring experiences that allow business teams to manage large fleets of agents through dashboards, execution histories, traces, metrics, logs, audit trails, and emergency controls.
-
Partner with workflow orchestration product leadership to define agent registries, structured contracts, and integration standards that allow agents to operate reliably within governed workflows.
-
Collaborate with engineering and design teams to create intuitive configuration tools that enable business specialists to build, test, and optimize automations independently.
-
Monitor emerging AI technologies, managed agent platforms, orchestration frameworks, and automation solutions to inform strategic build-versus-buy decisions and partnership opportunities.
-
Provide product leadership across adjacent teams, guide complex technical and strategic decisions, mentor product colleagues, and establish high standards for product execution and customer experience.
-
7+ years of product management experience in B2B or enterprise SaaS environments, including at least 3 years working in AI-focused product roles.
-
Proven experience owning complex products end to