SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Testlio is a fully managed crowdsourced testing platform powered by proprietary intelligence technology (LeoCore™), integrating expert on-demand testers into release processes. The platform covers 600,000+ devices, 800+ payment methods, 150+ countries, and 100+ languages.
You will serve as the visionary AI Solutions Architect, owning the definition, packaging, technical design, and delivery methodology for Testlio's AI testing offerings. This role combines deep practitioner expertise with strong business acumen. You will design testing approaches for complex AI-powered applications, including agentic systems, and advance Testlio's proprietary framework that combines automated and human-in-the-loop evaluation.
Key responsibilities include defining and maintaining the AI service catalog (offerings, tiers, packaging, delivery methodology), tracking and analyzing metrics for stakeholder leadership, and activating vision through white papers, articles, speeches, and webinars. You will collaborate with sales and engagement teams on pre-deal discovery, scoping, and value proposition articulation. You will decompose complex AI architectures into structured testing approaches, turning "unknown unknowns" into measurable testing criteria.
Your unique advantage: Testlio can deploy thousands of vetted experts across 150 countries and 100+ languages behind evaluation rubrics—a capability no other vendor offers. You will design evaluation systems combining model graders and human experts, shape industry standards for validating agentic AI and LLMs, and interface directly with data scientists, ML engineers, and executives at market-leading global brands.
The role includes hands-on delivery collaboration, supporting complex engagements and guiding delivery teams on AI testing opportunities. You will help build the pipeline and bring your expertise to high-impact business decisions. Testlio is a female-founded, profitable, cash-strong company with exceptional clients (many 7+ years tenure, spending $500K+/year), a 4.7 G2 rating, and a fully remote, globally distributed culture.
