Data Engineer

Data Analytics Irving, Texas, United States

Profile
Contacts

Summary

Creating pipelines, data flows and complex data transformations and manipulations using Azure Data Factory (ADF) and PySpark with Databricks.
Developed Spark applications using PySpark and Spark-SQL for data extraction, transformation and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
Developed methods, by evaluating statics, joins, indexes and code changes.
Implemented both ETL and ELT architectures in Azure using Data Factory, Databricks, SQL DB and SQL Data warehouse
Integrated data from different sources to a single common source in Azure DB.
Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL and U-SQL Azure Data Lake Analytics. Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks.
Developed Databricks PySpark scripts using DataFrames and Spark-SQL for transformations and loading the data to different targets.
Experienced working with different source and target systems such as Azure data lake gen1/gen2.
Develop Databricks Python notebooks to Join, filter, pre-aggregate, and process the files stored in Azure data lake storage.
implemented self-hosted integration runtime in ADF.
Responsible for modifying the code, debugging, and testing the code before deploying on the production cluster.
Worked with team members to resolve any technical issue, Troubleshooting, Project Risk & Issue identification, and management.
Adapted to all stages of software life cycle like design, development, testing and deployment. Requirements translation into design and then coding according to designs and testing depending upon requirements.
Used GIT for integrating the work across the team.