Benefits
- Generous PTO
- Retirement plan
Skills
Nice to have
Full job description
Description
- We are seeking a Senior Staff Software Engineer - AI/ML Inference Engineering to lead the architecture, optimization, and operational excellence of our AI inference platform
- You will own the end-to-end inference lifecycle—from onboarding cutting-edge models to deploying optimized inference engines in production across multiple backends on Kubernetes infrastructure
- You’ll also architect the intelligent gateway layer using Envoy and ext_proc to enforce access control, provide observability, and enable advanced request routing
- This role combines deep technical expertise in GPU compute, LLM APIs, inference frameworks, containerization, orchestration, and API gateway architecture with a bias toward pragmatic engineering and measurable performance impact
Benefits
- Generous family leave
- Matched donations
- Annual learning stipends
- Flexible PTO
- Competitive retirement plan
- Paid volunteer time
- Knowledge of unit testing, profiling, and code tuning
- Experience building new products that use challenging algorithms
- Knowledge of core AI/ML techniques and algorithms
- Expertise in Go or Python, OOP, Design Patterns, time and space-efficient algorithms
- Expertise in coding efficient, object-oriented, modularized and quality software
- Exposure to LLM APIs or ML-serving systems, or a clear ability to ramp up quickly
- Model-serving and runtime experience, such as vLLM, TensorRT-LLM, Triton Inference Server
- Artificial intelligence strategy is a critical skill at Experienced proficiency for IC5 — it applies to how you build and to what you build
- Confident use of AI coding assistants such as GitHub Copilot or Cursor, paired with the critical eye to review AI-generated code as rigorously as any other — catching logic errors, security anti-patterns, and missed edge cases rather than treating model output as production-ready
- AI-assisted debugging and trace analysis, and a clear understanding of the data privacy rules governing what goes into a prompt
- A master’s degree may offset up to one year of the experience minimum. Equivalent practical experience is considered where technical depth is clearly demonstrable
- Bachelor’s degree in computer science, software engineering, or a closely related technical field required
- 8+ years of Backend software engineering experience
- 4+ years of designing and operating distributed systems
- Debug and resolve production incidents independently, participate in on-call, run root-cause analysis, and operate against defined service level objectives
- Own observability for what you build — meaningful metrics, useful logs, real tracing, and alerts that fire on customer impact rather than on noise
- Deploy, scale, and operate services in production Kubernetes environments, including Helm-based deployments and CI/CD pipelines that make releases repeatable and safe to roll back
Opens ServiceNow’s own application page in a new tab.