SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Beacon AI is building an AI platform to make flying safer, more efficient, and more capable. The company is backed by top investors, has secured a dozen Department of Defense contracts, and partners with major airlines to deliver mission-critical systems.
You will ship LLM-powered product features end-to-end, designing retrieval and tool-calling flows, writing the services that run them, building evals and guardrails, and monitoring cost, latency, and quality in production. You'll partner with ML/infra teammates on embeddings, indexing, and model hosting, and with product teammates on user experience and outcomes.
Key responsibilities include:
**Build user-facing LLM features:** Design and implement retrieval-augmented generation (RAG) and tool-calling flows using frameworks like LangChain. Deliver robust JSON and schema-bound outputs with validation, retries, and fallbacks. Add function calling to integrate with internal tools, search, routing, and data services.
**Own the service layer:** Ship APIs and workers in Python or TypeScript with clear contracts, streaming, and backoff. Add caching, request shaping, prompt templates, and context packing to control latency and cost. Integrate with AWS Bedrock, OpenAI, Anthropic, or self-hosted endpoints.
**Retrieval and data prep:** Collaborate with infrastructure teammates to develop chunking, embeddings, and indexing capabilities for documents, time series, and multimedia. Choose and tune vector backends such as OpenSearch, pgvector, or Pinecone. Keep knowledge bases fresh with data syncs from S3, Aurora, DynamoDB, and external sources.
**Evaluation and quality:** Create offline evals and golden sets for prompts, retrievers, and tools. Stand up online metrics for task success, hallucination rate, retrieval precision/recall, p95 latency, and cost per request. Run A/B tests and prompt/version rollouts with guardrails and canaries.
**Safety, privacy, and compliance:** Implement content and policy checks, PII detection and redaction, access controls, and auditing. Design human-in-the-loop paths for sensitive actions. Handle aviation data with care and follow internal security standards.
**Operate what you build:** Add tracing, logs, and dashboards for model calls, token usage, errors, and saturation. Debug tricky failures across retrieval, prompts, tools, and providers.
Example problems in month one: Transform an internal knowledge base into a low-latency RAG service with explicit schemas and evaluations. Add tool-calling to automate cockpit or ops workflows with guardrails and audit trails. Reduce cost per request through improved chunking, caching, and prompt refactoring while maintaining task success rates.
The role is hybrid, based in San Carlos, CA, with 3+ days per week onsite.
**Requirements:**
- 8+ years of experience with a track record of owning systems or defining standards others build against
- Shipped LLM apps and improved them with data
- Strong production coding, testing, and documentation skills; preference for simplicity and observability
- Deep understanding of embeddings, chunking, vector search tradeoffs, and function calling
- Design evals, define success metrics, and iterate based on evidence
- Track p95 latency, hit SLAs, and reduce cost without hurting quality
- Clear communication and ability to explain tradeoffs across product, infra, and security teams
- Demonstrated technical leadership: set direction or standards that other engineers or teams built against
**Nice to have:**
- Experience with Bedrock, OpenSearch Serverless, pgvector, Pinecone, or Weaviate
- Prompt versioning, guardrails, and provider routing in production
- Multimodal work with time series or video
- Familiarity with GPU inference, Triton, or TensorRT-LLM
- Aviation or other safety-critical domain exposure
- DevOps basics for CI/CD, IaC, and secure secrets handling
