Lead Data Engineer Contract duration: 12 + 12 + 12 + 12 months
Location of work: Anywhere in Australia / Remote
Application closing date: Wednesday, 26 March 2025
Estimated start date: Tuesday, 01 July 2025
Security Clearance: Able to obtain baseline clearance
Our federal government client, IP Australia (IPA), is seeking an ETL Developer/Data Engineer to provide technical leadership to carry out development of and provide production support for a series of data load or data replication workflows on an AWS-based Data Lake platform.
Key Skills: Data pipelines/ETL/ELT in PySpark (demonstrated in AWS Glue)Management of data in a variety of table formats including Parquet and IcebergManagement of data ingestion/egress using REST APIsManagement of data ingestion using AWS native technologies such as DMS and OpenSearchData manipulation in SQL and PythonManagement and deployment of AWS infrastructure using CloudFormationManagement of lifecycle policies and backups in AWS S3Management of data workflows using a combination of AWS Lambda and SagemakerSelection Criteria Essential Demonstrate experience working in tight-knit Agile teams, providing data capabilities and services to technical and business stakeholders.
Communicate well (both verbally and in writing), share knowledge and experiences and document requirements.Have a strong knowledge or and demonstrate high-level experience in data analysis, problem solving and ETL development, including designing, building, testing and deployment of data workflows.Demonstrate experience with the following technologies: AWS GlueAWS DMSApache ParquetAWS LambdaAWS SagemakerOpensearch/ElasticSearchPySparkLogstashIcebergDesirable Experience in the following areas: Database designOracle DBMSs, Elasticsearch DBMSs and Azure servicesData load/migration projects including data profiling and data quality analysis.Use of Informatica IICS, PowerCenter, Developer, Analyst or MDM tools#J-18808-Ljbffr