AI Test Engineer (SME)

NeatDevs LLC is a small federal subcontractor with a long-running seat on a U.S. Customs and Border Protection (CBP) program in Ashburn, VA. We are hiring an SME-level AI Test Engineer to own testing of AI components - Gemini skills, AI agents, and MCP integrations - defining how non-deterministic AI behavior is evaluated and building the automated evaluation and regression suites that run in CI/CD. U.S. citizenship is required (federal contract); you must be able to obtain a DHS/CBP Public Trust - an active or recent clearance shortens your start date but is not a gate. Hybrid: on-site in Ashburn, VA about one day per week, remote the rest.

What you'll do

• Define and own test strategy, standards, and quality gates for AI components
• Build evaluation approaches for non-deterministic output: golden datasets, scoring rubrics, acceptance thresholds, LLM-as-judge evaluation
• Test agents end to end: task completion, tool selection and trajectory, guardrails, human-in-the-loop checkpoints
• Test MCP servers and clients: schema conformance, error handling, authentication, least-privilege access
• Perform adversarial testing: prompt injection, jailbreaks, data leakage; measure latency, token usage, and cost
• Maintain automated suites in CI/CD; track test activity in Jira/Xray; mentor the team on AI testing

Required

• U.S. citizenship and ability to obtain a DHS/CBP Public Trust; must live in the DC/MD/VA area
• Bachelor's degree (BA/BS) in Computer Science or related field
• 10+ years of software QA and testing experience, including 5+ years in test automation
• 2+ years hands-on testing AI/ML or LLM-based applications (chat assistants, RAG, agents, tool integrations)
• Strong Python test development (pytest); strong API testing (REST, JSON schemas, auth)
• Working knowledge of agent concepts and the Model Context Protocol (MCP)
• CI/CD test integration (GitLab, Jenkins)

Nice to have

• Gemini / Vertex AI; Google ADK or LangGraph agent evaluation
• LLM eval frameworks (Vertex AI GenAI eval, promptfoo, DeepEval, Ragas)
• OWASP LLM Top 10, NIST AI RMF; Section 508 (ANDI); ISTQB AI Testing certification

Pay and benefits

$104,000 - $119,000 per year depending on experience and clearance status. Full-time W-2 role with health, dental, and vision insurance, life insurance, 401(k), paid time off, and paid federal holidays.

NeatDevs LLC is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or protected veteran status.