SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
PointFive is an AI Efficiency OS that helps engineering and FinOps teams optimize cloud and AI spending. The company recently closed a $60M Series B led by Accel and was founded by the team behind IntSights (acquired by Rapid7).
As an LLM Researcher, you will design the next generation of evaluation methodologies for API-based coding agents. This role combines systems research, benchmarking, and applied machine learning to answer fundamental questions about how coding agents can become more accurate, efficient, and cost-effective without requiring access to model weights.
Key responsibilities include:
- Design and develop small language models (SLMs) for endpoint devices like laptops
- Design and develop a model router using endpoint state
- Create rigorous evaluation methodologies for API-based coding agents, with and without access to external tools
- Build statistically sound benchmarks measuring cost, latency, accuracy, reliability, and developer productivity
- Develop novel techniques for optimizing agent context, retrieval, memory, and tool interactions while preserving correctness
- Design and analyze large-scale experimental campaigns using production APIs across multiple foundation models
- Build reusable research infrastructure, datasets, simulators, and evaluation frameworks for agentic systems
- Publish technical reports and contribute research findings that influence product direction and the broader AI community
- Collaborate with engineering teams to translate research into production-ready optimization systems
- Stay current with advances in LLMs, agent architectures, retrieval systems, and AI evaluation methodologies
The tech stack includes Go, Python, ONNX Runtime, PyTorch, Transformers, Claude Code, Codex, Cursor, OpenAI API, Anthropic API, Gemini API, Kubernetes, Docker, AWS, GitHub Actions, and PostgreSQL.
REQUIREMENTS:
Must-have:
- PhD or equivalent research experience in Computer Science, Machine Learning, Mathematics, Statistics, or a related quantitative field
- Strong background in experimental design, statistical analysis, and empirical evaluation
- Excellent programming skills in Python and/or Go, with experience building research infrastructure
- Demonstrated ability to conduct independent research and communicate findings through publications, technical reports, or open-source projects
- Deep understanding of modern LLMs, retrieval systems, or AI agents
Nice-to-have:
- Experience working with API-based foundation models such as Claude, GPT, Gemini, or similar systems
- Publications in machine learning, systems, information retrieval, software engineering, or AI evaluation
- Experience designing benchmarks, datasets, or evaluation frameworks
- Familiarity with developer tools, coding assistants, or software engineering workflows
- Experience with distributed experimentation and large-scale data analysis
