Requirements
- Bachelor’s degree in Computer Science, Engineering, or a related technical discipline with 8+ years of relevant experience. [Required]
- Experience supporting DISA, DISN, or other DoD network environments.
- Experience developing and deploying software applications that meet DoD security standards, including applicable STIGs.
- Knowledge of network security, encryption, and access controls within classified environments.
- Experience with big data technologies and platforms such as Kafka, Spark, NiFi, Hadoop, Elasticsearch, Logstash, Databricks, and ELK Stack.
- Experience applying big data technologies to text mining, summarization, search, and entity extraction.
Skills
Nice to have
Full job description
Clearance: Top Secret/SCI
Job Type: Full-time
Target Salary Range*: $175,000 - $200,000
- This represents the potential salary range for this position depending on education level, years of experience and/or certifications in addition to other position specific requirements which may impact salary
The Senior Data Engineer works with Confluent Kafka, Elastic Stack, AWS, Red Hat OpenShift/Kubernetes, data warehouses, and data lakes to deliver reliable and efficient data solutions. The role also supports automation, CI/CD, platform optimization, visualization capabilities, and collaboration across data, software, and network operations teams.
Key ResponsibilitiesData Pipeline Engineering and Integration
- Design, build, and maintain scalable and reliable data systems and pipelines for ingesting, processing, and transforming large, complex, and disparate data sources.
- Unify and integrate data from multiple network operations systems into a consistent and accessible platform.
- Create and maintain Kafka, Elastic, and Logstash pipelines to support enterprise data ingestion and processing.
- Automate ETL/ELT processes, data validation, cleansing, monitoring, and related data-engineering activities.
- Implement data solutions using Confluent Kafka, Elastic Stack, AWS, and Red Hat OpenShift/Kubernetes.
- Develop and manage data warehouses and data lakes supporting analytics, reporting, and operational dashboards.
- Optimize data infrastructure for scalability, reliability, performance, and cost effectiveness.
- Apply automation and engineering best practices to improve platform maintainability and operational efficiency.
- Analyze and optimize ingestion throughput, query performance, indexing strategies, storage efficiency, and cluster utilization across Elastic and Kafka platforms.
- Troubleshoot and resolve issues related to data ingestion, processing, storage, memory, partitioning, and cluster performance.
- Identify and resolve platform bottlenecks affecting data availability and downstream analytics.
- Support Kibana visualizations and dashboards.
- Support visualization capabilities using React, JavaScript, and HTML.
- Collaborate with data analysts, visualization engineers, and other stakeholders to deliver integrated and actionable data solutions.
- Ensure data quality, accuracy, security, and compliance with applicable DoD and DISA requirements.
- Support CI/CD pipelines for automated build, test, and deployment activities.
- Apply version control and DevOps practices to data-engineering workflows.
- Support secure operation of data platforms through appropriate access controls and configuration standards.
- Document data architecture, integration patterns, platform configurations, and operational procedures.
- Maintain technical documentation and runbooks supporting data platforms and pipelines.
- Collaborate across software, integration, data, analytics, and network operations teams to support enterprise data initiatives.
- Bachelor’s degree in Computer Science, Engineering, or a related technical discipline with 8+ years of relevant experience. [Required]
- Hands-on experience designing, building, and maintaining ETL/ELT data pipelines and integrating multiple disparate data sources. [Required]
- Experience with data warehouse and/or data lake technologies such as AWS Redshift, Amazon S3, Hadoop, Snowflake, or similar platforms. [Required]
- Experience with Elastic Stack, including Elasticsearch, Logstash, and Kibana, and/or Kafka for data ingestion and processing. [Required]
- Experience with Linux/UNIX system administration and automation. [Required]
- Experience with CI/CD pipelines, containerized pipelines, version control technologies such as Git and Bitbucket, and DevOps practices. [Required]
- Familiarity with network operations data, including logs, metrics, events, ticketing, CMDB data, and related data models.
- Proficiency with data engineering technologies and languages such as Python, Java, SQL, and shell scripting. [Required]
- Knowledge of data security, access controls, and compliance frameworks, including DoD, DISA, RMF, and STIG requirements. [Required]
- Strong communication and documentation skills supporting collaboration and knowledge sharing. [Required]
- Current Security+ certification or higher/equivalent certification. [Required]
- Active TS/SCI security clearance. [Required]
- Willingness and ability to complete and maintain additional customer suitability screenings throughout employment. [Required]
- Experience supporting DISA, DISN, or other DoD network environments.
- Experience developing and deploying software applications that meet DoD security standards, including applicable STIGs.
- Knowledge of network security, encryption, and access controls within classified environments.
- Experience with big data technologies and platforms such as Kafka, Spark, NiFi, Hadoop, Elasticsearch, Logstash, Databricks, and ELK Stack.
- Experience applying big data technologies to text mining, summarization, search, and entity extraction.
- Experience with cloud-based data platforms, including AWS, Azure, GCP, or AWS GovCloud.
- Experience with cloud-integrated platforms and Infrastructure as Code, including networking and security policies.
- Familiarity with Kubernetes deployments, platform upgrades, and patching.
- Experience with configuration management technologies such as Ansible, Puppet, or Chef.
- Experience with monitoring, alerting, and observability technologies such as Prometheus, Grafana, and Elastic.
- Experience with business intelligence and visualization tools such as Kibana.
- Experience developing and maintaining technical documentation and operational runbooks.
- Strong understanding and practical experience with Agile methodologies, including Scrum and SAFe.
- Experience with Atlassian Jira and Confluence.
- Confluent Developer certification.
- Elastic Certified Engineer certification.
- Experience working remotely with geographically dispersed teams and within matrixed organizations.
- Experience combining software, integration, and data engineering practices.
- Experience applying knowledge of system architecture, networking, and centralized logging using ELK to support data transformation initiatives.
Opens The Amatriot Group’s listing on LinkedIn in a new tab.