Employer description & source details
Design, develop, and maintain scalable data pipelines using Databricks, Apache Spark, Python, and/or Scala. Build and optimize data ingestion, transformation, and processing workflows using ETL/ELT methodologies. Implement and maintain Lakehouse architecture and Delta Lake solutions to support reliable and efficient data storage and processing. Work with Azure cloud services, particularly Azure Data Lake Storage Gen2 (ADLS Gen2), to manage and process large volumes of data. Optimize Spark jobs and data pipelines for performance, scalability, and cost efficiency. Implement data modeling practices to support analytical workloads, reporting, and downstream data consumption. Use Unity Catalog and related governance capabilities to manage data access, security, permissions, and compliance. Develop and maintain CI/CD pipelines to automate deployment, testing, and release processes for data engineering solutions. Monitor production data pipelines, troubleshoot failures, and resolve performance and data quality issues. Collaborate with data scientists, analysts, architects, and other engineering teams to deliver dependable data solutions. Follow best practices for data quality, documentation, version control, security, and production support. Strong hands-on experience with Databricks and Azure data engineering services. Experience working with Azure Data Lake Storage Gen2 (ADLS Gen2) . Proficiency in Python and/or Scala for data processing and pipeline development. Minimum 3.5 years of relevant experience in data engineering. Strong understanding of data pipeline development, data integration, and cloud-based data platforms. Experience with CI/CD workflows, data modeling, governance, and production support. Hands-on experience with Apache Spark and distributed data processing.
Extracted skills: python, azure, security