Digital/Data Engineer Lead

About the Role We are looking for a highly skilled Data Engineer to design, build, and optimize our data infrastructure and pipelines. You will work closely with data scientists, analysts, and product/engineering teams to ensure reliable, scalable, and high quality data delivery that supports analytics, machine learning, and business decision making. Key Responsibilities 1. Data Pipeline & ETL Development • Design, build, and maintain scalable ETL/ELT pipelines for batch and real time data processing. • Develop data ingestion frameworks from multiple sources (API, DB, message queues, cloud storage). • Ensure data quality, validation, and monitoring across the entire pipeline. 2. Data Architecture & Modeling • Design and optimize data warehouse / data lake architectures. • Build efficient data models to support BI, analytics, and ML workloads. • Implement best practices for data partitioning, indexing, and performance tuning. 3. Data Platform Engineering • Develop and maintain data platform components (workflow orchestration, metadata management, lineage tracking). • Optimize storage and compute costs in cloud environments. • Ensure high availability, reliability, and scalability of data systems. 4. Collaboration & Cross Functional Support • Work closely with data scientists to productionize ML features and datasets. • Partner with engineering teams to integrate data solutions into product systems. • Support business teams with data accessibility, documentation, and troubleshooting. 5. Governance, Security & Compliance • Implement data governance standards, including data cataloging, lineage, and access control. • Ensure compliance with data privacy and security policies. • Establish monitoring, alerting, and incident response for data pipelines. Required Qualifications Technical Skills • Strong programming skills in Python / Java / Scala. • Hands on experience with SQL and performance tuning. • Experience with modern data processing frameworks: o Spark, Flink, Beam, Kafka, Airflow, Dagster, Prefect • Experience with cloud platforms: o AWS / GCP / Azure (e.g., S3, Redshift, BigQuery, Snowflake, Databricks). • Solid understanding of data warehouse / data lake architectures. • Experience with CI/CD, containerization (Docker), and version control (Git). Soft Skills • Strong problem solving and analytical thinking. • Ability to work cross functionally with engineering, product, and data teams. • Good communication skills and documentation habits. Preferred Qualifications (Nice to Have) • Experience with machine learning pipelines or feature stores. • Knowledge of data governance frameworks (e.g., DataHub, Amundsen, Collibra). • Experience with real time streaming (Kafka, Pulsar, Kinesis). • Familiarity with dbt for data transformation. • Experience in industries such as e commerce, retail, fintech, AI, manufacturing. Education & Experience • Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, or related fields. • 5–8 years of experience in data engineering or similar roles.

职位描述

Daisy Liu