SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Vals AI is seeking an exceptional researcher to design and build the next generation of AI benchmarks for cybersecurity. You will create high-impact, challenging evaluations that push the boundaries of what we can measure in foundation models, directly influencing how the world evaluates AI systems.
The role involves leading the design and development of novel benchmarks that assess real-world capabilities and risks of large language models. Vals AI's benchmarks shape how foundation models are developed and generative AI applications are built. The company works with every major foundation model lab, leading financial institutions, and application-layer companies at the frontier. Recent work has been featured by the Wall Street Journal, New York Times, Washington Post, and Bloomberg.
Key responsibilities include:
- Design and develop novel, high-impact benchmarks assessing challenging real-world cybersecurity capabilities
- Conduct research to ensure benchmarks are valid, reliable, and meaningful
- Collaborate with foundation model labs and enterprise security teams to understand evolving evaluation needs
- Analyze model performance across benchmarks and articulate results
- Publish research findings and contribute to the broader security and evaluation research communities
- Work closely with the infrastructure team to implement benchmark designs at scale
- Stay current with latest developments in LLM cyber capabilities and evaluation methodologies
About the company: Vals AI was founded by a team with prior experience at NVIDIA, Meta, Microsoft, Palantir, and HRT. The core methodology comes from NLP evaluation research conducted at Stanford. The founding team has over 300 citations in published work and includes Stanford PhDs, ex-Jane Street quants, and the first designer at Snorkel. The company raised a $5M seed and recently announced a $40M Series A at a $400M valuation, led by Andreessen Horowitz with participation from 8VC, Pear VC, Bloomberg, Hudson River Trading, and NextLadder Ventures. Tech stack: Python, Django, React, AWS with CDK.
The company values learning velocity, ownership, intensity, and a solution-oriented mindset. You'll work in a small, talent-dense team with autonomy and high speed of execution.
REQUIREMENTS:
- Cybersecurity experience: Hands-on work in security (security engineer, penetration tester, red teamer, or vulnerability researcher). May have found and reported vulnerabilities through bug bounty programs or CVEs.
- Systems depth: Strong grounding in operating systems, cloud infrastructure, and networking.
- Interest in benchmarking: Care about measuring capability rigorously. Ideally have built cyber evals, CTF challenges, or security benchmarks before.
- AI-forward: Already use LLMs in security work and want to work with them daily.
- Communication: Ability to clearly communicate complex research ideas to both technical and non-technical audiences.
- Collaboration: Experience working in research teams and integrating feedback.
- Location: In-person team in San Francisco; company will support relocation or transportation.
NICE TO HAVES:
- Prior work as security engineer at security company or on in-house security team, and/or experience on AI/ML team at security company
- Strong ties to security community: CTF teams, conference talks, open-source security tools, research collaborations, published security research, papers, or well-read technical write-ups
