Employer description & source details
Design, develop, and maintain scalable, reliable, and high-performance data pipelines using Python, PySpark, SQL, and Databricks. Build and optimize ETL/ELT pipelines for large-scale data ingestion, transformation, and integration. Develop and manage cloud-based data engineering solutions leveraging AWS services and Databricks. Write complex SQL queries, perform data modeling, and implement query optimization and performance tuning techniques. Design and implement distributed data processing solutions using Apache Spark and PySpark. Work with data lakes, cloud-based storage solutions, and enterprise data platforms. Implement CI/CD pipelines, automated deployments, and DevOps best practices for data engineering workloads. Ensure data quality, governance, security, compliance, and reliability across data pipelines. Troubleshoot pipeline failures, identify performance bottlenecks, and implement optimization strategies. Collaborate with data architects, data scientists, business analysts, and engineering teams to deliver end-to-end data solutions. Participate in Agile/Scrum ceremonies, technical discussions, design reviews, and code reviews. Contribute to the adoption of modern cloud-native data engineering practices and emerging technologies. Strong hands-on programming expertise in Python. Experience developing production-grade data engineering applications, automation scripts, and reusable components. Ability to write clean, efficient, maintainable, and scalable code. Expert-level SQL skills, including complex joins, window functions, CTEs, and advanced data transformations. Strong experience in data modeling, query optimization, and performance tuning. Ability to design efficient data structures for large-scale analytical workloads. Strong hands-on experience with PySpa
Extracted skills: python, sql, aws, azure, gcp, docker, kubernetes, machine learning, security