SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Coursera and Udemy have merged to create a comprehensive skills platform serving 300+ million learners and 12,000+ enterprise customers. You will join the AI Platform and LLMOps team in Pune, working on the operational backbone that enables reliable, production-grade AI systems.
Your primary focus will be building and maintaining infrastructure and tooling for agentic AI systems. You'll own prompt and pipeline versioning, evaluation frameworks, monitoring, and cost governance for LLM-powered applications. The role bridges internal AI enablement (helping non-technical teams build and deploy AI tools) and customer-facing solutions (custom AI implementations for enterprise clients).
Key responsibilities include:
- Building operational tooling for LLM systems: prompt/pipeline versioning, evaluation harnesses, and CI/CD workflows designed for non-deterministic AI outputs
- Implementing comprehensive monitoring and observability for production LLM systems, tracking latency, token usage, cost per request, output quality, and drift
- Designing and running automated evaluation suites to catch regressions, hallucinations, and quality degradation before customer impact
- Managing RAG pipelines and vector store infrastructure for fresh, accurate, performant retrieval
- Implementing safety and compliance guardrails including content filtering, PII redaction, and access controls aligned with enterprise data privacy and residency requirements
- Owning cost governance for LLM usage through caching strategies, model routing, and usage reporting
- Collaborating with Product Managers and Senior Engineers to translate operational requirements into reliable systems
- Participating in sprint planning, technical grooming, and retrospectives
You'll work in a fast-paced innovation environment with cross-functional teams of Software Engineers, AI Specialists, and Product Managers, turning AI prototypes into production-ready systems.
REQUIREMENTS:
Education: Bachelor's or Master's degree in Computer Science, Computer Engineering, or related field
Basic Qualifications:
- 4+ years of software engineering experience with solid Backend Engineering and DevOps fundamentals
- Hands-on experience operating or supporting LLM-powered systems in production (APIs, RAG pipelines, frontier/open-weight/fine-tuned models)
- Proficiency in a backend language (Python, Java, TypeScript, or Go) and familiarity with Docker and container orchestration (Kubernetes)
- Experience implementing APIs, working with SQL and NoSQL databases, and writing automated tests
- Working knowledge of at least one cloud platform (AWS, GCP, or Azure) across managed and self-hosted services
- Familiarity with infrastructure-as-code (e.g., Terraform) and CI/CD tooling (e.g., GitHub Actions, Jenkins)
- Comfort debugging production issues involving non-deterministic systems with attention to detail and bias towards reliability
- Strong belief in engineering quality and building tooling that creates leverage for others
Preferred Qualifications:
- Experience with LLMOps-specific tooling (LangFuse, LangSmith, Weights & Biases-style evaluation frameworks, vector databases like Pinecone, Weaviate, pgvector)
- Working knowledge of RAG architectures, MCP implementation and governance, agent orchestration frameworks (LangGraph), LLM Gateways (LiteLLM, Open Router), and enterprise AI ecosystems (Vertex AI, Bedrock)
- Exposure to prompt management and versioning practices treated as code, model access management, deterministic guardrails, and evals
- Understanding of AI governance considerations including data privacy, residency, and compliance in enterprise AI deployments
- Ability to work in fast-paced, ambiguous environments with proactive, ownership-driven mindset
- Strong communication skills and comfort collaborating across engineering, product, and strategy functions
