SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Monte Carlo is building the agent trust platform that unifies data and agent observability to monitor, troubleshoot, and improve production AI systems. As enterprises deploy thousands of agents across business-critical use cases, Monte Carlo provides the reliability infrastructure to support this AI transformation.
You'll work end-to-end on agent observability problems—from ambiguous problem statements through research, prototyping, and production deployment. This is not a spec-driven role; you'll get the problem, design the approach, prototype to validate, and ship integrated solutions alongside engineering and data science teams.
Key responsibilities include:
- Taking open problems end-to-end: research approaches, prototype, prove what works, build it, and integrate into the platform
- Designing and shipping agent-powered features like root-cause analysis, incident triage, and monitor generation
- Building evaluation infrastructure: golden datasets, regression suites, offline/online scoring, and defining quality standards
- Owning retrieval and context pipelines over customer metadata, lineage, and query history
- Instrumenting agent behavior in production: traces, failure taxonomies, cost and latency budgets
- Partnering with data science on detection quality and experiment design
- Setting technical standards for how the company builds with LLMs—patterns, guardrails, and reusable internal tooling
You must have production experience building autonomous agents (not RAG wrappers or API tool integrations), running evals and monitoring agents post-launch, and owning eval frameworks in production. Strong Python skills with ML/data science background required. You work from problems, not specs, and use AI tools daily as part of your development workflow. You ship quickly and know which problems deserve depth versus which need good-enough answers fast.
Nice-to-haves: statistics and hypothesis testing, MCP server experience, familiarity with data/cloud tools (Snowflake, Databricks, dbt, Airflow).
Monte Carlo is Series D-funded ($236M raised) with backing from Accel, Redpoint, Notable Capital, ICONIQ Growth, and Salesforce Ventures. Customers include HubSpot, Fox, Nasdaq, Toast, and Mercado Libre. The company is remote-first by design and recognized as a Best Workplace.
About Monte Carlo Data
AI / Data / Infrastructure; Data & Analytics — data observability platform for data quality and reliability.
