Data Engineer

Mô tả Công việc

· Lead the design and implementation of the Data Lakehouse architecture on Azure, making critical technology choices across Azure Data Lake Storage Gen2, Delta Lake, Azure Synapse, or Azure Databricks.

· Design, develop, and optimize high-throughput data processing pipelines (ETL/ELT) for a diverse range of sources (on-premise, cloud, streaming, APIs, files, etc.), ensuring resilience and fault tolerance.

· Act as a key technical liaison with AI, BI, and DevOps teams, driving the integration of data into complex analytics applications, production-grade machine learning models, and intelligent reporting systems.

· Pioneer and implement advanced optimization strategies for large-scale data storage and query systems (big data), focusing on maximizing performance, scalability, and cost-effectiveness.

· Establish and enforce best practices for data governance, including comprehensive data quality frameworks, centralized metadata management, and automated data lineage processes.

· Collaborate with infrastructure and security teams to architect and operate secure, compliant data systems, ensuring adherence to security protocols, internal regulations, and external standards.

· Mentor junior engineers, review code/design specifications, and produce high-quality technical documentation that enables knowledge transfer and efficient operations across product and analytics teams.

Yêu Cầu Công Việc

Education & Experience

· Bachelor’s degree or Master in Information Technology, Data Science, Information Systems, or equivalent.

· Minimum of 5 years of hands-on experience as a Data Engineer, with a proven track record of delivering end-to-end data solutions.

· Deep, demonstrable expertise in implementing and operating data infrastructure on Azure Cloud, with high proficiency in:

o Azure Data Lake Storage Gen2

o Architecture and implementation using Azure Synapse Analytics, Azure Data Factory, or Azure Databricks.

o Real-time/Streaming technologies like Azure Event Hub, Azure Stream Analytics, or Apache Kafka.

· Expert proficiency in SQL and a programming language such as Python or PySpark, along with a strong theoretical and practical understanding of distributed computing principles.

· Extensive experience designing and working with advanced data models (e.g., Star Schema, Snowflake Schema, Mastery of Lakehouse Architecture and Delta Lake best practices).

· Required experience in designing and managing large-scale production data systems, implementing robust real-time pipelines, and integrating data directly into AI/ML model serving layers.

Soft Skills

· Ability to work independently, proactively, and with a systematic mindset.

· Strong collaboration skills with other technical teams (AI, DevOps, Product, ...).

· Logical thinking, strong analytical, and problem-solving skills.

· Openness to learning new technologies, adapting to change, and experimenting