Lead Data Engineer - Databricks - $180,000 - Gain \"Productionised\" Ai

Lead Data Engineer - Databricks - Melbourne - $180,000 - Gain "productionised" Agentic AI and Gen AI

Morgan Consulting are partnering with an exciting and fast growing organisation seeking a Lead Data Engineer to join their Data function. This is an opportunity to gain Agentic and Generative AI "Productionised" experience. This role offers strong Career Development within an organisation which operates with very little red-tape and has a supportive and positive culture.

The Lead Data Engineer is a hands on technical specialist responsible for building and supporting modern Databricks Lakehouse solutions on Azure within a complex retail data landscape. Working across both project initiatives and BAU, the role delivers robust, scalable data pipelines and curated analytical datasets that underpin enterprise BI, advanced analytics, and AI. You will collaborate closely with retail stakeholders, BI developers, and broader technology teams to transform high volume retail data into trusted, production ready insights that support critical business outcomes. This is an opportunity to gain Agentic and Generative AI "Productionised" experience.

• Enable enterprise BI, analytics, and AI driven insights by delivering scalable, high quality data solutions that support retail trading, operational, and customer use cases.
• Build and optimise data pipelines and integrations that underpin reporting, advanced analytics, and AI/ML workloads.
• Ensure analytics and reporting solutions are performant, secure, scalable, and fit for enterprise and retail decision making.

• Design, develop, and evolve modern data engineering solutions on Microsoft Azure, with a strong focus on Azure Databricks, Azure Data Factory (ADF), and Azure Data Lake Storage (ADLS).
• Develop robust data transformations using PySpark, Databricks SQL, SQL, and Python.
• Design, implement, and maintain curated analytical and dimensional data models aligned to retail BI, analytics, and AI requirements.

• Develop, optimise, and support Power BI datasets, semantic models, and reports to deliver trusted, business ready insights.
• Administer Power BI environments, including workspace management, refresh orchestration, security, and access controls.
• Align BI solutions to best practice data modelling, performance optimisation, and usability standards.

• Provide BAU support across data platforms, pipelines, BI, and analytics solutions, including issue resolution and root cause analysis.
• Monitor data platforms and pipelines, implementing logging, alerting, and performance tuning.
• Proactively identify and resolve data quality, availability, and performance issues in production environments.

• Implement and maintain CI/CD pipelines for data and analytics solutions using Azure DevOps.
• Apply cloud security, authentication, and access control best practices across Azure data and analytics services.
• Contribute to design reviews, code reviews, and technical discussions to ensure robust, maintainable, and scalable solutions.
• Support the definition and continuous improvement of data engineering, analytics, and BI standards.

• Contribute to the design, adoption, and communication of data governance practices across the data platform.
• Partner closely with retail business stakeholders to translate requirements into reliable technical solutions.
• Collaborate with engineering, data science, infrastructure, and DevOps teams to deliver secure, production ready solutions.
• Balance delivery across project initiatives and ongoing BAU activities within a complex retail data environment.

• Strong experience with Databricks, including Unity Catalog, Delta Lake, Databricks SQL, and Genie AI, enabling secure data governance, self service analytics, and AI driven insights across enterprise data platforms.
• Strong hands on Data Engineering experience with SQL, Python, PySpark and Databricks SQL.
• Strong experience designing and delivering data solutions on Microsoft Azure - Azure Databricks, Azure Data Factory (ADF), Azure Data Lake Storage (ADLS).
• Proven experience designing and building data warehouses and analytical data models, including scalable lakehouse and dimensional modelling patterns.
• Strong understanding of data integration, analytics platforms, and reporting architectures in enterprise environments.
• Hands on experience administering and developing Power BI solutions, including Dataset and semantic model design, Strong DAX capability, Performance optimisation and governance.
• Experience working in DevOps environments, including - CI/CD pipelines, Source control, Release and environment management.
• General understanding of cloud authentication, security, and access control concepts, including role based access and data security principles.
• Demonstrated ability to deliver both project work and BAU/support, operating confidently in a production data and BI environment.
• Proven "enterprise scale" Data Engineering experience to Lead Data Engineer level.

• Opportunity to gain Agentic and Generative AI "Productionised" experience.
• Strong Career Development.
• Amazing culture with very little red tape.

If you are a Lead Data Engineer with strong stakeholder management, who has experience in a fast-paced retail environment, and is outcome focussed we would love to hear from you.