BigData/Spark Principal Architect

Job Description

Job Title: Data Engineering Specialist Media & Entertainment
Location: Permanent Remote North America, United States
Experience Required: 10 - 18 years
Job Type: Full-Time
Salary Range: USD 150,000 - 200,000 per year

Job Description:
We are seeking a highly skilled Data Engineering Specialist to work in the Media & Entertainment domain. The candidate will design, develop, and optimize big data pipelines using Apache Spark (DataFrame, Spark SQL, PySpark/Scala Spark, Databricks), Kafka, and cloud platforms such as AWS Glue or Azure. This role requires hands-on experience in solutioning and consulting as a Subject Matter Expert (SME) in Big Data technologies.

The role involves working with cross-functional teams to deliver scalable, high-performance data solutions for clients in the media and entertainment sector.

Roles & Responsibilities:

Design and develop big data pipelines (batch and streaming) using Spark and related technologies

Act as SME in Big Data solutioning and consulting for complex data platforms

Ensure data quality, governance, and operational excellence across pipelines

Work on cloud-based data platforms (AWS Glue or Azure) for large-scale data workloads

Write production-level code in Scala, PySpark, or Spark SQL

Collaborate with clients and internal teams for architecture and solution design

Support cross-skilling initiatives and knowledge sharing within the team

Required Qualifications:

10 18 years of experience in Big Data engineering

Hands-on experience with Apache Spark (DataFrame, Spark SQL, PySpark/Scala Spark, Databricks)

Proven experience in Big Data solutioning and consulting (SME level)

Strong cloud experience in AWS Glue or Azure for large-scale data workloads

Familiarity with Kafka or other streaming technologies

Expertise in ETL, Data Governance, Data Quality, and Data Operations

Ability to design and architect big data systems

Strong problem-solving, learning agility, and cross-skilling capability

US Citizen or Green Card holder

Preferred / Nice-to-Have Skills:

AI/ML exposure

Media and Entertainment domain experience

Consulting experience with solution selling and practice development

Work in United States

Employment Options

Professional Experience

Skills

• Apache Spark
• Spark SQL
• PySpark
• Scala Spark
• Databricks
• Kafka
• Big Data Solutioning
• Big Data Consulting
• AWS Glue
• Azure
• ETL
• Data Governance
• Data Quality
• Data Operations
• Big Data Architecture
• Production-level Coding
• Problem Solving
• Learning Agility
• Cross-skilling