Python + PySpark - Junior Developer Job at Compunnel, Inc., New York, NY

SDhZbm5BdXNoUzNSUG5hd0V3bjBiMVVDVEE9PQ==
  • Compunnel, Inc.
  • New York, NY

Job Description

The Python + PySpark + AWS Consultant will be responsible for designing, developing, and optimizing data engineering solutions using Python and PySpark. The role requires hands-on experience in building scalable ETL pipelines, working with cloud platforms such as AWS or Azure, and ensuring data integrity and performance optimization. The candidate should have strong SQL skills and a deep understanding of data engineering principles. Key Responsibilities

  • Develop and maintain data processing workflows using Python and PySpark
  • Design and implement ETL pipelines for structured and unstructured data
  • Optimize Spark-based data processing for efficiency and scalability
  • Deploy and manage data solutions on AWS or Azure
  • Write and optimize SQL queries for data transformation and analysis
  • Troubleshoot and resolve performance issues in data pipelines
  • Work closely with cross-functional teams to ensure data reliability and integrity
Required Qualifications
  • 5+ years of experience in data engineering
  • Strong proficiency in Python and object-oriented programming
  • Hands-on experience with PySpark for large-scale data processing
  • Proficiency in SQL for data manipulation and query performance tuning
  • Experience with AWS or Azure for cloud-based data solutions
  • Knowledge of ETL processes and data pipeline automation
  • Experience with Hadoop is acceptable
Preferred Qualifications
  • Experience in optimizing Spark jobs for performance and cost efficiency
  • Familiarity with DevOps practices for data engineering
  • Understanding of data governance, security, and compliance best practices
#J-18808-Ljbffr Compunnel, Inc.

Job Tags

Similar Jobs

Inkitt

Creative Strategist, Social Media & Growth Marketing Job at Inkitt

 ...ideas into engaging stories across various media platforms Oversee creative production...  ...of creatives that align with key marketing beats and/or evergreen initiatives Manage...  ...movies, etc ~ Expertise in working with social platforms and influencers is a plus... 

ANGELS CARE HEALTH SYSTEMS MAGNOLIA GARDENS

Receptionist/Concierge Job at ANGELS CARE HEALTH SYSTEMS MAGNOLIA GARDENS

 ...Job Description: Receptionist/Concierge SUMMARY: Performs assigned duties that protect the safety and dignity of residents. Work together with all staff to ensure that the resident care goals are being met and that all building goals are being met. The Receptionist... 

Inyo Legal Recruiting

Native American Law Advocacy | 1600 Hours Job at Inyo Legal Recruiting

 ...This purpose-driven boutique law firm is comprised mainly of attorneys who previously practiced at Cravath-scale firms, but who desired...  ...dedicated leaders in their tribes. A background in land use, environmental law, infrastructure, energy, government regulations, and/or... 

Professional Security Consultants, Inc.

Installation Tech - IP Cameras/Network-Based Security/Access Control Job at Professional Security Consultants, Inc.

 ...installing, repairing, and programminga wide variety of electronic security: Network-Based Security Systems /IP Surveillance Cameras/Access Control Systems. Must have ability to read and translate schematics and wiring diagrams; install, repair & test hardwire, wireless,... 

Fresh Origins

Food Safety Supervisor Job at Fresh Origins

 ...Job Summary: The Food Safety Supervisor oversees food safety and compliance, ensuring the production of safe, high-quality products. This role is responsible for implementing and managing food safety policies specific to the greenhouses, outdoor growing areas and packinghouses...