- Building next-generation scalable, reliable, flexible, high-performance data pipeline capabilities for ingestion of data from multiple sources containing complex dataset.
- Continuously improve services you own, making them more performant, and utilising resources in the most optimised way.
- Collaborate closely with engineering, data science team and product team to propose an optimal solution for a given problem statement
- Working closely with DevOps team on performance monitoring and MLOps
- Good understanding of distributed computing principles
- Experience in Apache Spark
- Hands on programming with Python
- Knowledge of Hadoop v2, Map Reduce, HDFS
- Experience with building stream-processing systems, using technologies such as Apache Storm, Spark-Streaming or Flink
- Experience with messaging systems, such as Kafka or RabbitMQ
- Good understanding of Big Data querying tools, such as Hive
- Experience with integration of data from multiple data sources
- Good understanding of SQL queries, joins, stored procedures, relational schemas
- Experience with NoSQL databases, such as HBase, Cassandra/Scylla, MongoDB
- Knowledge of ETL techniques and frameworks
- Performance tuning of Spark Jobs
- General understanding of Data Quality is a plus point
- Experience on Databrick/snowflake or similar lake houses would be a big plus
- Nice to have some knowledge in DevOps
Data Engineer - Bengaluru, India - Rakuten India
Description
About Rakuten:
Rakuten India is the Development Centre and key technology hub of the Rakuten Group, Inc. We enable our businesses with our depth of knowledge in multiple streams of technology such as Mobile and Web Development, Web Analytics, Platform Development, Backend Engineering, Data Science, Machine Learning, Artificial Intelligence and much more. Our unique 24/7 support center ensures reliability and sustenance of the Rakuten Ecosystem. With dedicated centres of excellence for Mobile Application Development, Data Analytics, Engineering, DevOps and Information Security, we ensure the success of multiple units of Rakuten Group, Inc. With 1700+ employees and growing, Rakuten India is housed in Crimson House Bangalore in the heart of the city.
Our History:
In 1997, Rakuten first began with Rakuten Ichiba, a B2B2C marketplace, with just six employees, one server, 13 merchants. With the mission of "empowering people and society through innovation and entrepreneurship," the Rakuten group rapidly grew with regional headquarters across the world.
In 2016, Rakuten India opened it doors in Bangalore, India, the tech city known as the Silicon Valley of India This research and development center became a key technology hub of the Rakuten Group, championing some of the products and platforms that run the businesses. With and growing) Rakutenians working on the very same mission as Rakuten Group, Inc, we believe that technology and business needs must challenge each other for true innovation to rise and make a telling business impact We have team members who work support Rakuten's global strategy across businesses such as e-Commerce, Digital, Marketing Platforms, Ecosystem Services and so on. "Walk together" is our guiding philosophy and together we continue to grow stronger by taking Rakuten's businesses to the next level with not just existing products but also create some in the relevancy of Artificial Intelligence and Machine Learning.
Responsibilities:
Required Skills:
5+ Years of Data related technology experience
"Rakuten is committed to cultivating and preserving a culture of inclusion and connectedness. We are able to grow and learn better together with a diverse team and inclusive workforce. The collective sum of the individual differences, life experiences, knowledge, innovation, self-expression, and talent that our employees invest in their work represents not only part of our culture, but our reputation and Rakuten's achievement as well. In recruiting for our team, we welcome the unique contributions that you can bring in terms of their education, opinions, culture, ethnicity, race, sex, gender identity and expression, nation of origin, age, languages spoken, veteran's status, color, religion, disability, sexual orientation and beliefs"