SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Salary: USD 171,900 - 300,800 / annual
ServiceNow is seeking a Senior Staff Software Engineer to lead the Build Agent evaluation framework team. Build Agent is ServiceNow's AI coding assistant, purpose-built for the platform's metadata-driven substrate and operating natively across ServiceNow's scoped applications, tables, and metadata types.
You will lead a team of 8 engineers responsible for the evaluation framework, model support, and telemetry that underpins Build Agent's success. The framework has already driven measurable wins: recovering Build Agent correctness, cutting latency and inference cost ahead of major releases, and catching high-severity defects before production. The team is currently the bottleneck on its own scale—this role exists to convert a high-performing initiative into a durable, scalable function.
Key responsibilities include:
**Evaluation Infrastructure**: Design and maintain golden prompt sets, Pass@1 and functional scoring methodologies, failure categorization (plumbing vs. metadata-creation failures), and coverage across ServiceNow metadata types and UI workflows.
**Model Support & Benchmarking**: Conduct structured evaluation of candidate foundation models against the production default, with failure-consistency analysis to separate scaffold/tuning issues from genuine capability gaps. Lead model benchmarking efforts and make data-backed recommendations on model support decisions.
**Telemetry**: Track token usage, cache efficiency, and inference cost alongside correctness as first-class release signals.
**Cross-Team Arbitration**: Prioritize eval coverage, define quality standards across surfaces owned by different contributing teams, and drive eval findings to committed, owned fixes rather than open-ended reports.
**Strategy & Roadmap**: Own evaluation strategy, prompt set design, scoring methodology, failure-mode taxonomy, and telemetry instrumentation.
You will solve ambiguous, cross-cutting problems where eval signal, model capability, and platform architecture intersect, and where "correct" isn't obvious until you've defined the bar.
**Requirements**
- Direct experience building or leading evaluation systems for LLM-based products—scoring methodology, benchmark design, and failure analysis (not just prompt engineering)
- Understanding of coding agent architecture: tool use, agentic loops, context management, and trade-offs between generic and domain-specific scaffolding
- 6+ years of experience with technologies relevant to ServiceNow, including advanced coding skills and fluency in one or more of Java, C++, Ruby, Shell, or JavaScript
- Experience critically evaluating foundation models—distinguishing real capability differences from tuning or scaffold artifacts
- Ability to execute against ambiguous priorities, weighing context, risk, and desired outcomes, particularly where eval data is incomplete or contested
