Python + PySpark - Junior Developer Job at Compunnel, Inc., New York, NY

SDhZbm5BdXNoUzNSUG5hd0V3bjBiMVVDVEE9PQ==
  • Compunnel, Inc.
  • New York, NY

Job Description

The Python + PySpark + AWS Consultant will be responsible for designing, developing, and optimizing data engineering solutions using Python and PySpark. The role requires hands-on experience in building scalable ETL pipelines, working with cloud platforms such as AWS or Azure, and ensuring data integrity and performance optimization. The candidate should have strong SQL skills and a deep understanding of data engineering principles. Key Responsibilities

  • Develop and maintain data processing workflows using Python and PySpark
  • Design and implement ETL pipelines for structured and unstructured data
  • Optimize Spark-based data processing for efficiency and scalability
  • Deploy and manage data solutions on AWS or Azure
  • Write and optimize SQL queries for data transformation and analysis
  • Troubleshoot and resolve performance issues in data pipelines
  • Work closely with cross-functional teams to ensure data reliability and integrity
Required Qualifications
  • 5+ years of experience in data engineering
  • Strong proficiency in Python and object-oriented programming
  • Hands-on experience with PySpark for large-scale data processing
  • Proficiency in SQL for data manipulation and query performance tuning
  • Experience with AWS or Azure for cloud-based data solutions
  • Knowledge of ETL processes and data pipeline automation
  • Experience with Hadoop is acceptable
Preferred Qualifications
  • Experience in optimizing Spark jobs for performance and cost efficiency
  • Familiarity with DevOps practices for data engineering
  • Understanding of data governance, security, and compliance best practices
#J-18808-Ljbffr Compunnel, Inc.

Job Tags

Similar Jobs

Amentum

TDY Electrician Job at Amentum

 ...credentials/authorization at this work location. Starting Hourly Rate for this position is $45/hour based on a 40-hour workweek Amentum is proud to be an Equal Opportunity Employer. Our hiring practices provide equal opportunity for employment without regard to race,... 

Reddit, Inc.

Android Engineer, Ad Formats Build Innovative Ads (San Francisco) Job at Reddit, Inc.

A leading social media company is seeking an experienced Android Engineer to build ad experiences for its app. The ideal candidate will have at least 3 years of Android engineering experience with proficiency in Kotlin or Java. This role entails collaborating across teams...

Community Health Systems

Histology Supervisor Job at Community Health Systems

 ...Job Description Job Description Histology Supervisor Position Type: Full Time Benefits: Health Insurance (Medical, Dental, Vision), 401(k) with matching Student Loan Repayment: Up to 10,000 Tuition Reimbursement Job Summary The Supervisor... 

Point Park University

Plumber/Pipe Fitter Job at Point Park University

 ...Job Description Job Description POSITION : Plumber/Pipe Fitter DEPARTMENT : Physical Plant REPORTS TO : Director of Operations POSITION SUMMARY : The Plumber/Pipe Fitter is responsible for the installation, maintenance, and repair of various piping systems... 

Nethermind

Blockchain Data Engineer Job at Nethermind

 ...builders and researchers on a mission to empower enterprises and developers worldwide to access and build on decentralized systems. Our...  ...Working to solve some of the most challenging problems in the blockchain space, we frequently collaborate with renowned companies, such...