Python + PySpark - Junior Developer Job at Compunnel, Inc., New York, NY

SDhZbm5BdXNoUzNSUG5hd0V3bjBiMVVDVEE9PQ==
  • Compunnel, Inc.
  • New York, NY

Job Description

The Python + PySpark + AWS Consultant will be responsible for designing, developing, and optimizing data engineering solutions using Python and PySpark. The role requires hands-on experience in building scalable ETL pipelines, working with cloud platforms such as AWS or Azure, and ensuring data integrity and performance optimization. The candidate should have strong SQL skills and a deep understanding of data engineering principles. Key Responsibilities

  • Develop and maintain data processing workflows using Python and PySpark
  • Design and implement ETL pipelines for structured and unstructured data
  • Optimize Spark-based data processing for efficiency and scalability
  • Deploy and manage data solutions on AWS or Azure
  • Write and optimize SQL queries for data transformation and analysis
  • Troubleshoot and resolve performance issues in data pipelines
  • Work closely with cross-functional teams to ensure data reliability and integrity
Required Qualifications
  • 5+ years of experience in data engineering
  • Strong proficiency in Python and object-oriented programming
  • Hands-on experience with PySpark for large-scale data processing
  • Proficiency in SQL for data manipulation and query performance tuning
  • Experience with AWS or Azure for cloud-based data solutions
  • Knowledge of ETL processes and data pipeline automation
  • Experience with Hadoop is acceptable
Preferred Qualifications
  • Experience in optimizing Spark jobs for performance and cost efficiency
  • Familiarity with DevOps practices for data engineering
  • Understanding of data governance, security, and compliance best practices
#J-18808-Ljbffr Compunnel, Inc.

Job Tags

Similar Jobs

CommonSpirit Health

Orthopedic Spine Surgeon Job at CommonSpirit Health

 ...Michael Medical Center a proud member of Virginia Mason Franciscan Health, is actively seeking a dedicated and skilled Orthopedic Spine Surgeon to join our growing team in Silverdale, WA. This is an exceptional opportunity to practice alongside an established Orthopedic... 

Chime

Director of Corporate Editorial & Storytelling Hybrid (San Francisco) Job at Chime

 ...A financial technology company is seeking a Director of Corporate Editorial to lead innovative storytelling and strategic narratives for various stakeholders. This role demands expertise in corporate communications and digital media with a focus on leveraging AI to enhance... 

Six Flags Fiesta Texas

Justice League Character Actor- Wonder Woman Job at Six Flags Fiesta Texas

Justice League Character Actor- Wonder Woman (Audition & Warner Brother Approval Required) Audition Information - Online submissions are currently accepted. Please submit a headshot and the following information: current measurements - height, pant/ shirt/ shoe size, etc...

Core Medical Group

School-Based Occupational Therapist Job at Core Medical Group

 ...Job Description Core Medical Group is seeking a travel Occupational Therapist for a travel job in Dekalb, Illinois. Job Description & Requirements...  ...week ~ Shift: 8 hours ~ Employment Type: Travel School Occupational Therapist needed for a contract in Dekalb... 

NHBlacklabs Delivery

Step Van Amazon Delivery Driver Job at NHBlacklabs Delivery

 ...NHBlacklabs Delivery LLC Are you looking for a job with true career growth? Does the prospect...  ...) is looking for responsible Delivery Drivers to distribute products safely to our...  ...to 50 lbs. Able to get in and out of van and walk up and down stairs through your...