Solution Architect

Solutions Architect Sunbury-on-Thames, England, United Kingdom

Profile
Contacts

Summary

With 18 years in IT consulting, I specialize in Data Engineering and Architecture, focusing on enhancing data management for business growth. My expertise lies in Information/Data Security, Governance, Cloud-based Big Data Solutions, and Regulatory Compliance, tailored to European and UK markets.

Experience as Solution Architect & Data Engineering
--------------------------------------------------------------------
Led the integration of Apache Airflow to streamline Data transformation pipelines and modeling -workflows, ensuring efficient task management and dependencies.
Utilized Anaconda Navigator for advanced data analytics, facilitating the creation of robust, Python-based data processing scripts.
Designed and managed tasks using Apache Airflow operators, enhancing workflow efficiency and automation capabilities.
Implemented XCom in Apache Airflow for effective data passing between tasks, ensuring seamless data integration and accuracy.
Employed TaskGroups in Apache Airflow for visualizing complex workflows, improving clarity and manageability for the data engineering team.
Conducted extensive data operations including reading, filtering, grouping, and processing, leveraging Apache Airflow's powerful scheduling and execution capabilities.
Administered Cron scheduling within Apache Airflow, focusing on catch-up and backfilling strategies to maintain data consistency across various projects.
Led the migration from a scikit-learn single node to a scalable Azure Databricks solution using Apache Spark's MLib, enhancing machine learning workflow efficiency.
Managed distributed ETL tasks in Databricks, optimizing data handling and processing large-scale datasets efficiently.
Directed advanced data processing in Databricks clusters, handling batch and streaming data, ensuring high performance in large-volume data analysis.
Showcased proficiency in integrating scikit-learn with Spark on Databricks, enhancing analytics and machine learning in data pipelines.
Developed sophisticated business reporting techniques in Azure Databricks, involving complex data aggregation and analysis for strategic business insights.

Expectations

I am particularly interested in positions where I can apply my skills in Apache Airflow for orchestrating complex data pipelines, utilize MS Azure Databricks for scalable and collaborative data analytics, and employ Python for scripting and automation. Additionally, my experience in business reporting has enabled me to translate complex data into actionable business intelligence, a skill I am eager to bring to a new role. Roles could be solution architect, data engineering, product developer or data architect.