SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Salary: CAD 164,490 - 197,389 / annual
Grafana Labs is seeking a Senior Backend Engineer to join the Tempo team, the open-source distributed tracing backend powering Grafana Cloud Traces and Grafana Enterprise Traces. Tempo is at an inflection point in 2026, shifting from foundational work to product and operational excellence while evolving into a platform that powers Grafana's next-generation observability products including App Observability, Asserts, Traces Drilldown, and AI-driven assistants.
As a Senior Engineer on Tempo, you will own technical direction on the hardest problems in the roadmap and raise the bar across the team. Key responsibilities include:
- Deliver significant multi-month technical projects from problem framing through rollout, such as trace aggregation APIs, autoscaling cells, customer limits, and query engine improvements. Partner with Staff engineers and tech leads on broader cross-team direction.
- Own major components of Tempo's ingestion, storage, query, and metrics generation systems. Write and present design docs, participate in design reviews, and make sound trade-offs on performance, cost, and complexity.
- Build APIs for both humans and AI agents, shaping the next generation of Tempo's interfaces to be structured, deterministic, and discoverable for downstream products and LLM-driven assistants.
- Contribute to operational excellence by owning outcomes against concrete SLOs (P99 write latency, incident recurrence, TCO per ingested GB) and driving toward Zero Ops through automation, parameterized rollouts, and actionable alerts.
- Collaborate with Product and sibling teams (App Observability, Asserts, Drilldown, Grafana Assistant) to understand how Tempo is consumed and ship what unblocks them.
- Mentor and support engineers through thorough code review, constructive design feedback, and pairing, especially with junior and mid-level engineers.
- Participate in on-call rotations for services you help build and contribute to incident response and post-incident learning.
- Engage the open-source community, review external contributions, and contribute to Tempo in the open.
Key focus areas for 2026 include: making Grafana Cloud Traces "just work" by eliminating rough edges and hidden failure modes; achieving operational excellence at scale as the platform grows from ~50 cells to triple digits; evolving Tempo into a platform enabler with higher-density APIs and machine/LLM-friendly interfaces; pushing performance further with faster query latency at hundreds of MB/s ingestion; and preparing Tempo for an agent-driven world with larger, burstier, higher-cardinality workloads.
Grafana Labs is a 100% remote company with team members across 40+ countries, backed by leading investors including Lightspeed Venture Partners, Sequoia Capital, GIC, Coatue, and others. The company is trusted by major organizations including Anthropic, Bloomberg, NVIDIA, Microsoft, and Salesforce.
REQUIREMENTS:
- Track record of delivering complex, multi-month projects spanning design, delivery, and operations, with demonstrated ability to help teammates grow.
- Significant hands-on experience building and operating distributed data systems in production: ingestion pipelines, storage engines, query execution, or similar.
- Strong software craftsmanship: write clean, robust, performant software that others can maintain, with good judgment on optimization vs. shipping.
- Solid experience in Go, or a clear path to it. Experience in other systems languages (Rust, C, C++) translates well.
- Operational mindset: owned production services, carried a pager, reduced toil, and treated SLOs as a product feature.
- Customer focus and pragmatism: break complex problems into short feedback loops (analyze, design, deliver MVP, learn, iterate).
- Strong collaboration and clear writing skills for a fully remote, asynchronous environment.
BONUS EXPERIENCE:
- Tracing, OpenTelemetry, or large-scale observability systems.
- Designing query languages, SQL/TraceQL-like engines, or APIs for programmatic consumption by services or agents.
- Columnar storage formats (e.g., Parquet) or purpose-built on-disk formats for analytical workloads.
- Operating multi-tenant, multi-cell SaaS infrastructure at scale on Kubernetes.
- Building for AI/LLM consumers: structured APIs, metadata/discovery endpoints, deterministic outputs, evaluation harnesses.
- Open-source contribution or maintainership with community engagement.
- On-call experience with Grafana, Prometheus, Loki, or Tempo.
- Experience in fully remote, globally distributed teams.
