Sahil Arora

As a Staff Data Engineer with over 8 years of experience working on Spark and other big data technologies, I have a proven track record of designing and implementing high-performance, scalable data systems. I hold an M.Tech degree from BITS Pilani, which has equipped me with a strong foundation in computer science and software engineering.

Currently, I am pursuing an MS in AI from IIT, where I am deepening my knowledge and skills in machine learning, deep learning, and other AI technologies. My academic background, combined with my professional experience, gives me a unique perspective on the challenges and opportunities of working with data at scale.

Throughout my career, I have played a pivotal role in building and optimizing data pipelines that can handle terabytes or even petabytes of data, ensuring that data processing is efficient, reliable, and scalable. My expertise in Spark has allowed me to optimize Spark jobs for maximum performance and efficiency, using techniques such as partitioning, caching, and serialization to streamline data processing.

In addition to Spark, I have a deep understanding of big data technologies such as Hadoop, Kafka, and Flink, as well as SQL and NoSQL databases. I am adept at designing data architectures that can handle a wide variety of data types and formats, and I am able to communicate complex technical concepts to stakeholders of all levels.

What sets me apart is my ability to solve complex data engineering problems with ease. I thrive on tackling complex challenges and finding innovative solutions that optimize data processing workflows. Whether working independently or as part of a team, I bring a wealth of knowledge, expertise, and problem-solving skills to any data engineering project.

Overall, I am a true expert in the field of data engineering, and I am passionate about helping organizations of all sizes and industries to manage their data effectively and efficiently.

As a data engineering professional, I have worked for leading organizations such as Visa and IBM Labs, designing and implementing data pipelines to manage large-scale transaction data. At Visa, I leveraged my expertise in Hadoop, Spark, Jenkins, and Kafka to build and optimize data processing workflows. Prior to Visa, I worked for IBM Labs where I was responsible for handling transaction data pipelines using the big data technology stack. My experience working with Hadoop, Spark, Kafka, and other big data technologies has equipped me with a deep understanding of data engineering principles and best practices. I am passionate about using my skills and experience to help organizations manage their data effectively and efficiently.

• M.Tech degree in a relevant field (e.g., computer science, software engineering) from BITS Pilani
• Currently pursuing an MS in AI from IIT