Staff Software Engineer, Tech Foundations
Free account needed to apply or save. It takes a minute, and we'll bring you right back to this job.
How well do you match this job?
Key details
- Location
- United States
- Work style
- Remote
- Level
- Senior
- Job type
- Full-time
- Experience
- 9–12 yrs
- Function
- Software engineering
- Focus area
- Infrastructure / Cloud
- Posted
- Yesterday
Skills
About the role
This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Staff Software Engineer, Tech Foundations based in the United States.
As a Staff Software Engineer, you will shape the technical direction of a large-scale Observability platform supporting critical production systems. You will lead the design, development, and operation of distributed infrastructure that processes millions of telemetry events every second. Working across metrics, logs, and distributed tracing, you will help engineering teams gain deeper insights into system performance, reliability, and operational costs. You will also contribute to building AI-ready observability capabilities, enabling autonomous agents to access operational data safely and efficiently. In this highly collaborative environment, you will influence architectural decisions, mentor engineers, and drive innovation across infrastructure teams. This is an opportunity to solve complex technical challenges while improving developer productivity and system resilience at global scale.
Accountabilities
-
Technical Leadership & Architecture: Define the technical vision and architecture for major areas of the Observability platform, leading initiatives from initial design and prototyping through production deployment and ongoing operations.
-
Distributed Systems Development: Design, build, and maintain highly scalable and reliable distributed systems capable of processing tens of millions of events per second.
-
Metrics Infrastructure: Scale and optimize the metrics pipeline using technologies such as OpenTelemetry, vmagent, and Grafana Mimir, while managing infrastructure costs through aggregation, sampling, and unused-metric filtering.
-
Logging & Tracing Modernization: Lead the transition from legacy log aggregation systems to OpenTelemetry-based pipelines and improve log and trace storage capabilities using technologies such as OpenSearch.
-
Query Infrastructure: Develop fast, reliable, and secure query systems for engineers and AI agents, incorporating rate limiting, access controls, isolation, and caching to protect critical incident-response workflows.
-
Telemetry Cost Optimization: Build self-service tools that provide engineering teams with visibility into telemetry costs, including cost attribution by team and service, as well as controls for data granularity, sampling, and retention.
-
Observability & Reliability: Improve incident detection through anomaly detection, automated signal correlation, and enhanced monitoring capabilities.
-
Cross-functional Collaboration: Align technical roadmaps with infrastructure and product teams, influence build-versus-buy decisions, and communicate architectural tradeoffs and investment priorities to engineering leadership.
-
Engineering Excellence: Write high-quality, maintainable, and well-tested code, participate in design reviews, improve operational readiness, and contribute to relevant open-source projects.
-
Mentorship & Knowledge Sharing: Mentor engineers, strengthen technical practices across the team, and help developers instrument services and independently investigate production issues.
-
Professional Experience: At least 9 years of software engineering experience, with a proven track record of designing, building, and operating large-scale distributed systems in production environments.
-
Technical Leadership: Demonstrated experience leading the technical direction of complex systems or multi-team initiatives, including architecture design, stakeholder alignment, delivery, and production operations.
-
Observability Expertise: Deep experience in at least one major telemetry domain at scale, including metrics platforms such as Prometheus, Grafana Mimir, or VictoriaMetrics; logging technologies such as OpenSearch or Kafka; or distributed