Senior Data Engineer at SEEK

Job Description

Responsibilities

Design scalable, high-performance data architectures and enforce data modelling standards.
Deploy, operate, and support data infrastructure and services on Databricks and AWS
Maintaining system health, ensuring that our systems are secure, reliable, resilient and cost effective.
Implement stringent data quality checks and troubleshoot issues to ensure data integrity.
Establish and enforce data governance policies, ensuring compliance with standards.
Growing the technical capabilities of more junior engineers, analysts and data scientists through coaching and mentoring.
Work closely and share knowledge with our partner businesses.

Qualifications
Essential Qualification, Skills and Experience

Bachelor’s degree in engineering, computer science, statistics, mathematics, or other related discipline
3+ years relevant professional experience or postgraduate qualification (Master’s/Ph.D.)
Highly proficient in both SQL and Python
Building and operating cloud-based data processing and warehousing solutions
Big data processing experience, preferably Databricks, EMR, Spark & Hadoop
Strong software engineering, coding & scripting
Logical and physical data modelling
Ownership of deployments, infrastructure, system administration, monitoring, and operations (ie. DevOps)
Data pipeline / ETL / workflow orchestration with Airflow
Databricks platform, particularly Spark Structured Streaming, Delta Lakes & Unity Catalog
AWS services, particularly EMR & S3, but also IAM, Cloudformation, CDK, Lambda, SNS, SQS, SSM
Kafka technologies, particularly KStreams DSL, processor and producer/consumer APIs
Develop clean code using good practices like CI/CD, TDD, pair programming, code review