Data Engineer

BigData Development Chennai, Tamil Nadu, India

Profile
Contacts

Summary

I am a Data Engineer with practical experience in designing and delivering scalable, metadata-driven data pipelines on Azure. At TCS, I built pipelines in Azure Data Factory and Databricks to ingest and process over 5 TB of SAP data monthly into ADLS Gen2. I have implemented full, delta and CDC load strategies, optimized Delta Lake transformations, and improved performance using adaptive partitioning and efficient state management techniques. I work extensively with Python, SQL and PySpark, and I maintain strong CI/CD practices through Azure DevOps. I also configure data governance and access control using Unity Catalog and Azure AD groups, while implementing proactive monitoring and fault-tolerance to ensure reliability and stable production operations.

Expectations

I am looking for a role where I can work on end-to-end data engineering pipelines using Azure and Databricks, with opportunities to design solutions, optimize performance, and contribute to architectural decisions. I am especially interested in roles that allow me to work closely with AI, machine learning, and LLM engineering, where data engineering supports model training, deployment, and real-world application. I want an environment that encourages learning, ownership, and collaboration, where I can grow into a more solution-driven and technically strong engineer while contributing to impactful, production-scale data and AI systems.