PySpark - Mumbai - confidential

Description

• Key Responsibilities:
• PySpark Development:
• Design, implement, and optimize PySpark solutions for large-scale data processing and analysis.
• Develop data pipelines using Spark to handle data transformations, aggregations, and other complex operations efficiently.
• Write and optimize Spark SQL queries for big data analytics and reporting.
• Handle data extraction, transformation, and loading (ETL) processes from various sources into a unified data warehouse or data lake.
• Data Pipeline Design & Optimization:
• Build and maintain ETL pipelines using PySpark, ensuring high scalability and performance.
• Implement batch and streaming processing to handle both real-time and historical data.
• Optimize the performance of PySpark applications by applying best practices and techniques such as partitioning, caching, and broadcast joins.
• Data Storage & Management:
• Work with large datasets and integrate them into storage solutions such as HDFS, S3, Azure Blob Storage, or Google Cloud Storage.
• Ensure efficient data storage, access, and retrieval through Spark and other tools (e.g., Parquet, ORC).
• Maintain data quality, consistency, and integrity throughout the pipeline lifecycle.
• Cloud Platforms & Big Data Frameworks:
• Deploy Spark-based applications on cloud platforms such as AWS (Amazon EMR), Azure HDInsight, or Google Dataproc.
• Work with cloud-native services such as AWS Lambda, S3, Google Cloud Storage, and Azure Data Lake to handle and process big data.
• Leverage cloud data processing tools and frameworks to scale and optimize the PySpark jobs.
• Collaboration & Integration:
• Collaborate with cross-functional teams (data scientists, analysts, product managers) to understand business requirements and develop appropriate data solutions.
• Integrate data from multiple sources and platforms (e.g., databases, external APIs, flat files) into a unified system.
• Provide support for downstream applications and data consumers by ensuring timely and accurate delivery of data.
• Performance Tuning & Troubleshooting:
• Identify bottlenecks and optimize Spark jobs to improve performance.
• Conduct performance tuning of both the cluster and individual Spark jobs, leveraging Spark's in-built tools for monitoring.
• Troubleshoot and resolve issues related to data processing, application failures, and cluster resource utilization.
• Documentation & Reporting:
• Maintain clear and comprehensive documentation of data pipelines, architectures, and processes.
• Create technical documentation to guide future enhancements and troubleshooting.
• Provide regular updates on the status of ongoing projects and data processing tasks.
• Continuous Improvement:
• Stay up to date with the latest trends, technologies, and best practices in big data processing and PySpark.
• Contribute to improving development processes, testing strategies, and code quality.
• Share knowledge and provide mentoring to junior team members on PySpark best practices.
• Required Qualifications:
• 2-4 years of professional experience working with PySpark and big data technologies.
• Strong expertise in Python programming with a focus on data processing and manipulation.
• Hands-on experience with Apache Spark, particularly with PySpark for distributed computing.
• Proficiency in Spark SQL for data querying and transformation.
• Familiarity with cloud platforms like AWS, Azure, or Google Cloud, and experience with cloud-native big data tools.
• Knowledge of ETL processes and tools.
• Experience with data storage technologies like HDFS, S3, or Google Cloud Storage.
• Knowledge of data formats such as Parquet, ORC, Avro, or JSON.
• Experience with distributed computing and cluster management.
• Familiarity with Linux/Unix and command-line operations.
• Strong problem-solving skills and ability to troubleshoot data processing issues.