Data Engineer candidate in Noida, Uttar Pradesh, India

Summary

I am an experienced Data Engineer with a focus on designing and implementing efficient ETL pipelines, leveraging modern technologies to improve data quality and processing times. Currently, I work as a Sr. Data Engineer at RedDoorz, where I have been responsible for building and maintaining data pipelines using technologies such as Python, AWS, Snowflake, Redshift, Meltano, Airflow, Pyspark, Docker, EKS, Kubernetes, DBT, and AWS CloudFormation.

At RedDoorz, I designed and implemented ETL pipelines that extract, transform, and load data from various sources into AWS S3, Redshift, and Snowflake data warehouses. I also leveraged Pyspark and Spark SQL to process and analyze large datasets in distributed computing environments, which improved data quality and processing times. Additionally, I developed custom ETL scripts using Python and SQL to extract, transform, and load data from a variety of sources into the data lake.

One of my major achievements at RedDoorz was deploying Airflow on AWS EKS, using Secrets Manager and EFS Driver, which allowed us to orchestrate data pipelines more effectively. I also mentored junior team members, providing guidance on best practices for data engineering and software development, and maintained version control of the codebase using GIT, enabling collaboration with other team members and ensuring code integrity. Moreover, I was responsible for performing UT cases for developed codes, logics, and pipelines, ensuring high data quality and accuracy.

Before RedDoorz, I worked at Axtria India Pvt Ltd, where I was part of the core team that developed the flagship product "DataMax." I used PySpark, Python, Redshift, and Matillion to process and transform large volumes of data and created a Data Quality Check pipeline using Dagster to ensure high data quality and accuracy. I also reduced the runtime of the Parquet file ingestion module by 10 times and conducted Data Quality Checks and Unit Tests for developed codes, Matillion jobs, and pipelines to ensure data integrity and high-quality deliverables.

In my earlier role at Cognizant Technology Solutions, I worked as a Programmer Analyst Trainee and Programmer Analyst. I worked extensively with JSON dumps to extract data and convert it to tables, while also handling complex data types present in the JSON dumps. I used Python and Spark to write logics and automate manual work, which improved efficiency. I developed PySpark and Python codes and wrote scripts to automate certain tasks, thereby reducing manual effort and increasing productivity. I scheduled jobs using Crontab and Redwood, ensuring timely completion of tasks and adherence to schedules. Additionally, I converted all Hive queries to PySpark scripts, which reduced runtime and consumption of resources. I actively participated in the development of a framework that facilitated seamless data loading into the table. As a result, data could be accessed easily even when writing data to the table.

Throughout my career, I have received several awards and recognitions for my exceptional work. These include the CoMeT R&R Q2 Hats-off Award from Cognizant Technology Services, the Bravo Award from Axtria India Pvt Ltd, and the Employee of the Quarter Award from RedDoorz.

Expectations

, I would have certain expectations from my new job. Firstly, I would like to work in a challenging and dynamic work environment that would enable me to continuously learn and develop my skills. I am looking for a position that will allow me to use my technical expertise and creativity to contribute to the growth and success of the company.

I would also appreciate a positive and supportive work culture that values teamwork, communication, and employee development. It is important for me to work in a collaborative environment where I can learn from my colleagues and share my own knowledge and experiences.

As a tech professional with expertise in Python, SQL, AWS, Docker, Kubernetes, and Git, my expectations from a new job include working on challenging projects that allow me to use and enhance my skills. I am looking for a role that allows me to work with cutting-edge technologies and innovate new solutions. I am also keen on working with a team that fosters a culture of collaboration and continuous learning.

Additionally, I would value a job that offers opportunities for growth and career advancement. I would like to work for a company that invests in its employees and provides opportunities for training, mentorship, and professional development.

Overall, I am looking for a job that provides a challenging and exciting work environment, values collaboration and employee development, aligns with my technical expertise, and offers opportunities for growth and advancement.