Databricks Engineer

As an ETL/ELT Data Engineer at our company, your responsibilities will include: - Designing, building, and maintaining scalable and efficient ETL/ELT pipelines in Azure Databricks to process various types of insurance data from internal and external sources. - Collaborating with data architects, modellers, analysts, and business stakeholders to gather data requirements and deliver data assets that meet analytics, regulatory, and operational needs. - Developing, testing, and optimizing data transformation routines, batch, and streaming solutions using tools like Azure Data Factory, Data Lake Storage Gen2, Azure Event Hubs, and Kafka to ensure timely and accurate data delivery. - Implementing rigorous data quality, validation, and cleansing procedures to enhance reliability for insurance use cases, reporting, and regulatory outputs. - Integrating Informatica tools for data governance, including capturing data lineage, metadata, and data cataloguing as required by regulatory and business frameworks. - Ensuring robust data security by following best practices for RBAC, managed identities, encryption, and compliance with Hong Kong's PDPO, GDPR, and other relevant regulatory requirements. - Automating and maintaining deployment pipelines using GitHub Actions to ensure efficient, repeatable, and auditable data workflows and code releases. - Conducting root cause analysis, troubleshooting pipeline failures, and proactively identifying and resolving data quality or performance issues. - Producing and maintaining comprehensive technical documentation for pipelines, transformation rules, and operational procedures to ensure transparency, reuse, and compliance. Your expertise in Hong Kong Life and General Insurance will be valuable in ensuring that our development captures local business needs and industry-specific standards. As an ETL/ELT Data Engineer at our company, your responsibilities will include: - Designing, building, and maintaining scalable and efficient ETL/ELT pipelines in Azure Databricks to process various types of insurance data from internal and external sources. - Collaborating with data architects, modellers, analysts, and business stakeholders to gather data requirements and deliver data assets that meet analytics, regulatory, and operational needs. - Developing, testing, and optimizing data transformation routines, batch, and streaming solutions using tools like Azure Data Factory, Data Lake Storage Gen2, Azure Event Hubs, and Kafka to ensure timely and accurate data delivery. - Implementing rigorous data quality, validation, and cleansing procedures to enhance reliability for insurance use cases, reporting, and regulatory outputs. - Integrating Informatica tools for data governance, including capturing data lineage, metadata, and data cataloguing as required by regulatory and business frameworks. - Ensuring robust data security by following best practices for RBAC, managed identities, encryption, and compliance with Hong Kong's PDPO, GDPR, and other relevant regulatory requirements. - Automating and maintaining deployment pipelines using GitHub Actions to ensure efficient, repeatable, and auditable data workflows and code releases. - Conducting root cause analysis, troubleshooting pipeline failures, and proactively identifying and resolving data quality or performance issues. - Producing and maintaining comprehensive technical documentation for pipelines, transformation rules, and operational procedures to ensure transparency, reuse, and compliance. Your expertise in Hong Kong Life and General Insurance will be valuable in ensuring that our development captures local business needs and industry-specific standards.