SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
FourKites is seeking a Principal AI Architect to own the platform architecture underlying all FourKites AI agents—the execution harness, context management, model selection, and evaluation frameworks. The company operates a suite of digital workers (Tracy for track-and-trace, Sam for supplier operations, Alan for appointment management, Cassie for customer service, Polly for document management, and more) that perform real supply chain work on a shared platform.
In this hands-on individual contributor role based in Chennai (hybrid), you will write code most weeks, lead design reviews across agent teams, and make critical decisions on model choice, context strategy, and evaluation that all teams depend on.
Key responsibilities include:
- Define the reference architecture for FourKites agents: agent loop, tool execution, state and checkpointing, retries and recovery, guardrails, permissions, and human-in-the-loop handoffs
- Design for long-running, multi-step work spanning many context windows with clean failure recovery
- Decide where each problem is best solved (frontier LLM, open-weight model, fine-tuned encoder, or plain code) and own model routing, cost-per-task, and latency budgets
- Integrate agents with FourKites' event-driven platform (Kafka, microservices, Postgres) and customer systems via APIs and MCP servers
- Set standards for agent context windows (system prompts, tool definitions, retrieved documents, memory, prior turns) within explicit token budgets
- Build shared components for retrieval, compaction, summarization, sub-agent isolation, prompt caching, and persistent memory
- Diagnose context failures and turn fixes into reusable harness patterns
- Build offline eval frameworks: golden datasets from production traces, code-based checks, LLM-as-judge graders calibrated against human labels, and CI regression suites
- Build online evaluation: production traffic scoring, trace- and task-level quality metrics, drift detection, shadow and A/B rollouts, user feedback signals, and quality alerts
- Connect eval metrics to business outcomes (tracking accuracy, exceptions resolved without human, appointment fill rate)
- Review designs and PRs for agent teams, write architecture docs and playbooks, and mentor staff and senior engineers
- Track the model and tooling landscape, run experiments, and provide leadership with build/buy/wait recommendations
- Partner with Product, Solutions, and Customer Success to turn field failures into platform fixes
You will be the senior technical authority for how FourKites builds, ships, and measures agents, working closely with agent teams in Chennai and colleagues in the US and Europe.
Requirements:
- Architected and operated at least one LLM agent system with real users and real traffic (tool calling, multi-step workflows, failure handling, human escalation) and owned it post-launch through incidents, cost, and latency. Framework demos and POCs alone are not sufficient.
- Deep ML knowledge of encoders and transformers: attention, positional encodings, encoder-only vs decoder-only vs encoder-decoder architectures, tokenization, and KV cache. Experience training or fine-tuning encoder models (BERT-family, sentence embeddings, bi-encoders, cross-encoder re-rankers) for classification, extraction, or retrieval. Understanding of how contrastive training, distillation, and quantization affect cost and quality.
- Context engineering experience: managed token budgets, built retrieval and memory for agents, used compaction and sub-agents to maintain coherence in long tasks, and measured quality degradation as context windows fill.
- Harness engineering: built execution layers around models including agent loop, tool dispatch, state and checkpoints, interruption and recovery, sandboxing and permissions, guardrails, and model routing. Fluent with MCP, structured outputs, and streaming.
- Hands-on online and offline evals: built eval suites including datasets, graders, LLM-as-judge calibration and CI regression gates offline; production scoring, trace sampling, drift monitoring, shadow deployments, and feedback loops online. Used evals to block releases or justify model switches.
- 10+ years building software or ML systems, with 3+ years building on LLMs
- Strong, hands-on Python; comfortable reading and reviewing Java or TypeScript services
- Production experience on AWS or GCP, containers and Kubernetes, and event-driven systems
- Excellent oral and written communication skills: ability to explain trade-offs to engineers and business impact to executives
Nice to have:
- Supply chain, logistics, or transportation domain experience
- Document AI: OCR, layout models, extraction from bills of lading, invoices, and PODs
- Voice or email agents acting on behalf of a business
- Agent security: prompt injection defense, tool permissioning, tenant data isolation
- Hands-on experience with multiple model families (Claude, GPT, Gemini, open-weight models) and serving stacks
- Published work, talks, or open-source contributions on agents, retrieval, or evaluation
