Data Engineer

Data Analytics North York, Ontario, Canada

Profile
Contacts

Summary

Experience with Azure Data Bricks:
o Responsible for estimating the cluster size, monitoring, configuring, and troubleshooting of the spark data bricks cluster.
o Hands on experience with writing SQL Scripts, Python scripts in notebooks, PySpark coding etc.
o Applied the spark data frame API to complete data manipulation within spark session.
o Created several data bricks Spark jobs with PySpark to perform several tables to table operations.
Experience in Azure Data Factory:
o Built pipelines to copy data from source to destination in Azure Data Factory, created stored procedures, and scheduled them in the Azure environment using Azure Web Jobs.
o Created pipelines in ADF using linked services/datasets/pipeline to extract, Transform and load data from different sources like Azure SQL, Blob Storage, delta lakes, etc.
o Ingest data into one or more Azure services, including Azure Data Lake, Azure Storage, Azure SQL, and Azure DW, and process the data in Azure Databricks.
o Implemented workflows using Lookups, For each, Append Variable, Union, Filter tasks, conditional split, etc.
Experience in Power BI using the Azure platform:
o Used various sources to pull data into Power BI such as SQL Server, SQL Azure etc.
o Designed and developed Power BI dashboards with a rich look, filtering, drill-down, drill-through reports, and sub-reports in Power BI Dashboards.
o Worked with various DAX functions to implement Row- Level Security, Create calculated measures and calculated columns.
Involved in configuring and upgrading the on-premises data gateway between various data sources like SQL server to Azure analysis services and Power BI Service.
Created data bricks notebooks using python and SQL scripts for transforming the data that is stored in Azure data lake Gen2 from Raw to Stage and Curated Zone.
Created independent data marts from existing data warehouse as per the application requirement and updated them on bi-weekly basis.
Developed Integration Services packages to extract data from various sources such as Flat Files, XML files, CSV files, Excel files, SQL Server, and loaded it into the CRM system.
Used ETL (SSIS) to develop jobs for extracting, cleaning, transforming, and loading data into the data warehouse to ensure relevant data is readily available for the CRM system.
Collaborated with analysts and knowledge experts to design, build, and customize reporting packages to ensure the CRM system meets the organization's needs.
Maintained and debugged existing Power BI reports and have hands-on experience in designing complex reports using SQL Server Reporting Services (SSRS) and Crystal Reports.