Experienced Data Engineer with expertise in Apache Spark, Delta Lake, AWS, Hadoop and ETL

Summary

As a result-oriented technocrat, I have a solid understanding of Big Data technologies and have successfully built data pipelines for optimal extraction, load, and transformation (ETL) of data and analytics. I have created an end-to-end infrastructure for streaming data pipelines, achieving 60% cost savings by migrating streaming pipelines from Databricks to Amazon EMR and Airflow managed service on AWS. I have also reduced database server utilization by 50% using partitioning in Spark while loading large datasets from relational databases to data lakes. I have developed production-grade Python and Spark projects executed on Amazon EMR and have worked with various cloud platforms such as AWS, and GCP.

Expectations

A challenging opportunity in Data Engineering with good work culture and work life balance

Employment Preferences

Expected Base Salary

*,*00,000 USD

Academic Degree

Experience

Skills

• Airflow
• Amazon Elastic MapReduce
• Amazon Web Services
• Apache Hive
• Apache Spark
• AWS Glue
• Big Data
• BigQuery
• BitBucket
• Cross-functional Team Collaboration
• Data Analytics
• Data Lake
• Data Pipelines
• Data
• Processing
• Data Warehousing
• Databricks
• Delta Lake
• Git
• GitHub
• Google Cloud Dataproc
• Hadoop
• Java
• Kafka
• Map Reduce
• MySQL
• NoSQL
• PostgreSQL
• Pyspark
• Python
• Relational Database Management System
• Spark
• Streaming
• SQL
• Technical Documentation
• Strong Communication Skills