Lead Data Engineer

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Lead Data Engineer based in the United States.

As Lead Data Engineer, you will drive the technical strategy and execution of large-scale data modernization initiatives within a federal technology environment.
You will lead the migration of legacy data warehouses and pipelines to a modern Azure and Databricks lakehouse architecture.
Working closely with data engineering teams, you will establish scalable migration frameworks, data quality standards, and reconciliation processes.
Your expertise will help ensure data accuracy, performance, and reliability throughout complex cloud transformation projects.
You will provide technical guidance, resolve challenging engineering issues, and mentor engineers to strengthen delivery capabilities.
Collaborating with Product, QA, and Platform teams, you will ensure technical solutions align with project objectives and customer expectations.
This is an opportunity to make a meaningful impact through modern data engineering solutions supporting critical public-sector services.

You will lead the technical direction of data migration and modernization initiatives, ensuring the successful transition of legacy systems to scalable cloud-based infrastructure. Your responsibilities will include:

• Define and own the technical approach for migrating legacy data warehouse environments and pipelines to an Azure and Databricks lakehouse architecture.

• Establish standardized migration frameworks, technical patterns, and best practices to guide data engineering teams.

• Design and develop reusable data migration pipelines using Azure Data Factory and Azure Databricks.

• Define data reconciliation strategies and validation processes to ensure accuracy and consistency between source and target systems.

• Establish data quality controls, testing standards, and quality gates throughout the migration lifecycle.

• Provide technical leadership in resolving complex pipeline challenges, performance bottlenecks, and data engineering issues.

• Optimize data processing workflows to improve scalability, reliability, and overall system performance.

• Mentor data engineers, provide technical guidance, and review engineering deliverables to maintain high standards of quality.

• Collaborate with Product, Quality Assurance, and Platform teams to coordinate technical activities and ensure successful project delivery.

• Promote consistent engineering practices and scalable architectural solutions across migration workstreams.

• Communicate technical decisions, progress, risks, and recommendations clearly to internal stakeholders and customers.

• Contribute to additional technical and operational responsibilities as required.

Requirements:

The ideal candidate is an experienced data engineering leader with a strong background in cloud modernization, large-scale data migrations, and distributed data processing. You should bring:

• A Bachelor's degree in a relevant field.

• Proven experience leading data engineering teams or technical workstreams within large-scale migration or modernization programs.

• Deep expertise in Databricks and/or Apache Spark, with hands-on experience designing and implementing data processing solutions.

• Strong experience with Azure Data Factory and cloud-based data pipeline development.

• Advanced SQL and Python programming skills.

• Demonstrated experience migrating legacy ETL processes and data warehouse workloads to cloud environments.

• Strong understanding of lakehouse architecture, medallion architecture, and Delta Lake.

• Experience designing reusable data pipelines, data validation frameworks, and reconciliation processes.

• Ability to troubleshoot complex data engineering problems and optimize pipeline performance.

• Proven technical leadership skills, including mentoring engineers, reviewing code, and establishing engineering best practices.

• Excellent communication and collaboration skills, with the ability to explain technical concepts and align stakeholders around technical direction.

• Experience working effectively in remote, collaborative Agile environments.

• A proactive, adaptable, and solution-oriented mindset.

Additional qualifications that would be advantageous include:

• Experience working in federal, government, or other regulated data environments.

• Familiarity with government data modernization initiatives and associated compliance requirements.

Additional Employment Requirements:

• Must be legally authorized to work in the United States.

• Depending on federal contract requirements, certain positions may require U.S. citizenship.

• Ability to obtain and maintain a federal background investigation and/or security clearance, if required.

Benefits:

• Annual salary range of $110,000–$145,000.

• Fully remote work arrangement.

• Comprehensive employee benefits package.

• Technology and education stipend.

• 401(k) retirement plan with company matching.

• Flexible paid time off (PTO).

• Paid holidays.

• Opportunities for professional development and technical growth.

• Collaborative environment within a small, impact-driven organization.

• Opportunity to contribute to meaningful technology initiatives supporting the public sector.

Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.

#LI-CL1