Join our team as a Senior Data Engineer and be part of data engineering and migration workloads for our esteemed clients. If you're passionate about Databricks, Spark, and cloud-based data engineering, this is your chance to lead a high-impact, transformative project. Immediate start available!About INGRITY:INGRITY is a Microsoft Data and AI Solution Partner dedicated to delivering transformative data and AI-driven solutions. We collaborate with some of the best ASX-listed companies and medium-sized businesses, working closely with Microsoft to create measurable value for our customers. Our success is built on innovation, customer advocacy, and a strategic partnership with Microsoft.About the job:As a Senior Data Engineer, you will be responsible for designing, developing, migrating, and optimising data pipelines using Databricks and modern data technologies. You will work closely with data scientists, analysts, and engineers to enable seamless data integration and analytics across the organisation.Key Responsibilities:Design, develop, and maintain scalable ETL/ELT pipelines in Databricks and cloud environments.Redesign and implement transformations and business logic in Databricks using Spark SQL, PySpark, or Scala.Lead data migration efforts, ensuring successful transfer of data from legacy systems to Databricks.Optimise Apache Spark workloads for performance and cost-effectiveness.Collaborate with cross-functional teams to support machine learning and advanced analytics initiatives.Work with various data sources to integrate both structured and unstructured data.Implement best practices for data governance, security, and compliance.Monitor and automate data pipeline workflows for reliability and performance.Requirements:5+ years of experience in Data Engineering in a similar role.Proven expertise in Databricks, including cluster management, Delta Lake, and performance optimisation.Strong hands-on experience in Spark (PySpark, Scala, or SQL) for data processing.Deep knowledge of ETL processes, data modelling, and data warehousing concepts.Experience with cloud platforms (Azure, AWS) and their data services, particularly in relation to Databricks.Excellent problem-solving and troubleshooting skills.Strong communication skills, with the ability to explain technical concepts to non-technical stakeholders.Additional Information:Eligibility: Candidates must have full working rights in Australia to be considered for this role.
#J-18808-Ljbffr