- Design, develop, and maintain data pipelines using Spark, Python, Scala, and related technologies on Hadoop/Clouderra Platform
- Work with high volume data and ensure data quality and accuracy
- Implement data security and privacy best practices to protect sensitive data
- Develop and maintain documentation on data pipeline architecture, data models, and data workflows
- Monitor and troubleshoot data pipelines to ensure they are performing optimally
- Stay up-to-date with the latest developments in Azure, AWS, Spark, Python, Scala, and related technologies and apply them to solve business problems
- Optimize data pipelines for cost and performance
- Automate data processing tasks and workflows to reduce manual intervention.
- Ability to work in Agile Feature teams
- Provide training and educate other team members around core capabilities and helps them deliver high quality solutions and deliverables/documentation
- Self-Motivator to perform Design / Develop user requirements, test and deploy the changes into production
- Hands-on experience in the following on Spark Core, Spark SQL, SQL/Hive/Impala
- Exposure on Hadoop Ecosystem(HDP/Cloudera/MapR/EMR etc)
- Experience of working on File formats (Parquet/ORC/AVRO/Delta/Hudi etc.)
- Experience with high volume data processing and data streaming technologies
- Experience of using Orchestration tools like Control-m
- Strong experience in data modeling, schema design, and ETL development using SQL and related technologies
- Familiarity with data security and privacy best practices
- Good exposure on TDD
- Exposure on using CI tools like Git, Bitbucket, Github, Gitlab, Azure DevOps
- Exposure on using CD tools like Jenkins, Bamboo, Azure DevOps
- Cloud exposure (Hadoop)
- Experience and knowledgeable on the following: Azure data offerings - ADF, ADLS2, Azure Databricks, Azure Synapse, Eventhubs, CosmosDB etc, Presto/Athena
- Exposure of working on Power BI
- Prior experience in building or working in team building reusable frameworks
- Good understanding of Data Achitecture and design principles. (Delta/Kappa/Lambda architecture)
- Exposure to Code Quality - Static and Dynamic code scans
- Good knowledge of NoSQL Databases/ HBase/ MongoDB / Cassandra / Cosmos DB
- Good knowledge of GraphDB(Neo4J)
- Experience with enterprise data management, Datawarehouse, data modelling, Business Intelligence, data integration.
- Expertise in SQL and stored procedure.
- Experience in designing solutions for multiple large data warehouses with a good understanding of cluster and parallel architecture as well as high-scale or distributed RDBMS and/or knowledge on NoSQL platforms.
- Experience in working on Azure SQL Data warehouse (Synapse) and Azure analysis service or Redshift or Synapse
- Propose best practices/standards
- Translate, load and present disparate datasets in multiple formats/sources including JSON, XML etc.
- Should be able to provided scalable and robust solution architecture depending on the business needs.
- Should be able to compare tools and technologies and recommend a tool or technology
- Should be well versed with overall IT landscape, technologies and should be able to analyse how different technologies integrates with each other
-
Development Engineer
3 weeks ago
Rohde & Schwarz hyderabad, IndiaMake ideas real - with the perfect job. · Right now we have several job openings for you. · Functional area All Administration 7 · Application Development 28 · Construction 5 · Controlling 6 · Cybersecurity 2 · Finance 18 · Hardware Development 42 · Human Resources 9 ...
-
Development Engineer
3 weeks ago
Unison Consulting Pte Ltd hyderabad, IndiaFull stack developer will be primarily responsible for development of critical modules in Digibank using Java/MS and React. · As a part of team you will play a key part in the bank's digital transformation journey. · Technology stack: · Java 8 · Spring Boot · MySQL Database ...
-
Cloud Development Engineer
1 week ago
Pegasystems Hyderabad, IndiaMeet Our Team: · As a member of one of the most innovative and fastest growing groups at Pega, you will work closely with talented engineers and product owners across the globe to build a highly scalable Software as a Service (SaaS) offering. · Picture Yourself at Pega: · In t ...
-
Software Development Engineer
4 weeks ago
ADCI HYD 13 SEZ - H84 hyderabad, IndiaAmazon's Customer Service organization is looking to expand its software engineering team to help drive automation, self-service and defect reduction for customers. We are searching for smart, energetic and passionate software development engineers to solve complex technical prob ...
-
Software Development Engineer
3 weeks ago
ADCI HYD 13 SEZ hyderabad, IndiaThe Amazon Enterprise Engineering team builds highly-scalable enterprise products that meets Amazon's unique security needs to protect Amazon's data. We are now working on a new multi-tenant platform fully based on native AWS to enable various internal organizations and new acqui ...
-
Software Development Engineer
4 weeks ago
Thomson Reuters Hyderabad, India Full timeAs an employee at Thomson Reuters, you will play a role in shaping and leading the global knowledge economy. Our technology drives global markets and helps professionals around the world make decisions that matter. As the world's leading provider of intelligent information, we wa ...
-
Software development Engineer
2 weeks ago
Pi Square Technologies hyderabad, IndiaJob Description PSTI 1795 · Client Name · UST · Client Location · Hyderabad · Position Name · Softwaredevelopment Engineer · Job Description · Required Qualifications: · Strong knowledge of ObjectOriented Analysisand Design Software Design Patterns REST architecture and ...
-
Software Development Engineer
3 weeks ago
Pan Asia Man power Consultants Recruiting agency hyderabad, IndiaSDET - Software Development Engineer in Test · Experience Range: 6 to 9 Years. · Key skills - Core Java, Selenium, Rest API, SQL, Manual Testing · Location: Pune/Hyderabad (Hybrid - 3 Days Work From Office in a Week) · Develop and maintain automated test scripts using Java and S ...
-
Software Development Engineer
3 weeks ago
Medtronic hyderabad, IndiaCareers that Change Lives At Medtronic, we contribute to human welfare and wellbeing through biomedical engineering.Everyday we're involved in meaningful work to change people's lives and health for the better.Now is your chance to join a talented team of engineers focused on p ...
-
Software Development Engineer
3 weeks ago
ADCI HYD 13 SEZ Hyderabad, India Full timeThe International Seller Growth (ISG) Tech vision is to build foundational technology to empower every Selling Partners (SPs) on Earth to grow their business in international marketplaces and adhere to rapidly evolving regional compliance regulations. ISG Tech also extends core A ...
-
Software Development Engineer
2 weeks ago
Planful hyderabad, IndiaAbout Us · Planful is the pioneer of financial performance management cloud software. The Planful platform, which helps businesses drive peak financial performance, is used around the globe to streamline business-wide planning, budgeting, consolidations, reporting, and analytic ...
-
Software Development Engineer
3 weeks ago
Squareroot Consulting Pvt Ltd. Hyderabad, IndiaTitle: Senior Software Development Engineer in Test (SDET) · Location - Hyderabad · Notice Period - Immediate Joiners · Position Overview : · We are looking for a skilled Automation Engineer with expertise in building automation frameworks using Python. The ideal candidate will ...
-
Software Development Engineer
2 weeks ago
ADCI HYD 13 SEZ Hyderabad, India Full timeWould you like to work on one of the world's largest transactional distributed systems? How about working with customers and peers from the entire range of Amazon's business on cool new features? Whether you're passionate about building highly scalable and reliable systems or a s ...
-
Software Development Engineer
3 weeks ago
ADCI HYD 13 SEZ hyderabad, IndiaSoftware Engineer - Amazon International Seller Services · Amazon International Seller Services mission is to set our worldwide sellers up for success across our global marketplaces. We deliver needle-moving initiatives which provide a seamless experience to Amazon sellers at eac ...
-
Engineer - .NET Development
3 weeks ago
DISH Network hyderabad, India Full timeCompany Summary DISH Network Technologies, an EchoStar Company, has been reimagining the future of connectivity for more than 40 years. Our business reach spans satellite television service, live-streaming and on-demand programming, smart home installation services, mobile plans ...
-
System Development Engineer
3 weeks ago
ADCI HYD 13 SEZ hyderabad, India Full timeAre you interested in taking a front-seat in the innovative technology that powers Amazon's award-winning Customer Service? If so, come join us · We are part of the Worldwide Customer Service Capacity Planning (WWCP) organization that enables end-to-end customer service workforce ...
-
Systems Developer/Engineer
3 weeks ago
AT&T hyderabad, IndiaJob Description · : · • 24 x 7 Production support and second level trouble shooting of application incidents. · • 24 x 7 second level outage response for applications · • 24 x 7 Application performance monitoring, troubleshooting and corrective actions for applications. · ...
-
Software Development Engineer, Amazon
3 weeks ago
ADCI HYD 13 SEZ hyderabad, India Full timeWould you like to work on one of the world's largest transactional distributed systems? How about working with customers and peers from the entire range of Amazon's business on cool new features? Whether you're passionate about building highly scalable and reliable systems or a s ...
-
Software Development Engineer in Test
3 weeks ago
HealthEdge hyderabad, IndiaOverview · What You'll Do · At HealthEdge, we are looking for talented Software Development Engineers in Test (SDETs) who will work with an Automation Architect and QA engineers to evaluate existing automation framework, build new test frameworks, write, and design automation ...
-
Development Tools Software Engineer
3 weeks ago
Intel Hyderabad, IndiaJob Description · The Analyzer Engineering team within the Intel takes care of Analyzers product group. The group is looking for a college graduate/Post graduates to join in the validation of software tools to collect and analyze platform performance data from the latest Intel CP ...
Data Engineering Developer - hyderabad, India - Telstra
Description
Employment Type Permanent Closing Date 29 Sept :59pm Job Title Data Engineering Developer - Python & PysparkJob Summary
As a Data Management Analyst, you value and steward strong governance, architecture and data exchange. You play an important role in supporting the implementation of effective data management strategies to ensure that Telstra's data is accessible, better understood, managed and organised which maximise business and customer outcomes.Job DescriptionTelstra is Australia's leading telecommunications and technology company with a rich heritage that's been built over 100 years. From our humble beginnings in the Postmaster General's Office to the global business we are today our people have been at the forefront of technology innovation. More recently, we have the largest in Australia and are And this is just the beginning of what we're hoping to achieve together.
We offer a full range of services and compete in all telecommunications markets throughout Australia and are the most well-known brand in technology and communications industry.
We have operations in more than 20 countries, including in India. In India we are a licensed Telecom Service provider (TSP) and have extended our global networks into India with offices in Bangalore, Mumbai and Delhi. We've opened an Innovation and Capability Centre (ICC) in Bangalore, and have a presence in Pune and Hyderabad. In India, we've set out to build a platform for innovative delivery and engagement that will strengthen our position as an industry leader. We're combining innovation, automation and technology to solve the world's biggest technological challenges in areas such as Internet of Things (IoT), 5G, Artificial Intelligence (AI), Machine Learning, and more.
We're growing, fast, and for you that means many exciting opportunities to develop your career at Telstra. Join us on this exciting journey, and together, we'll reimagine the future.
Being part of Data Engineering means you'll be part of a team that focuses on extending our network superiority to enable the continued execution of our digital strategy. With us, you'll be working with world-leading technology and change the way we do IT to ensure business needs drive priorities, accelerating our digitisation programme.
We are seeking a highly skilled Data Engineer with expertise in Spark, Python, Scala. The successful candidate will be responsible for will be responsible for designing, developing, and maintaining data pipelines using Spark, Python, Scala, and related technologies. The Data Engineer will also be responsible for ensuring data quality, data security, and optimal performance of the data pipelines. Any new engineer would be mostly into developing reusable data processing and storage frameworks that can be used across data platform.
Hadoop Data Engineering – Hive, Oozie, Yarn / MapReduce, Spark (must) with SCALA (Pref) or Python, Strong SQL
Key Responsibilities
Data Engineer role is to coordinate, and execute all activities related to the requirements interpretation, design and implementation of Data Analytics applications. This individual will apply proven industry and technology experience as well as communication skills, problem-solving skills, and knowledge of best practices to issues related to design, development, and deployment of mission-critical systems with a focus on quality application development and delivery.
This role is key to the success of the Data Engineering capability at Telstra and will be responsible and accountable for the following:
Technical Skills (Essential)
Programming / Databases (Tools) - Java /Python/Scala/ SQL Procedure, Multi tenanted databases / Spark
Frameworks & Methodologies - Scaled Agile, DevOps, High Performance Data Engineering
Organisational skills
Proven high level of initiative, drive and enthusiasm – essential.
Time management and an ability to adhere to set schedules.
Communication Skills
Demonstrated high level of written and oral communication skills
Excellence in customer service and consulting skills.
Strong verbal communication skills.
Strong listening & questioning skills.
Conflict Handling
Ability to work under pressure – essential.
Ability to manage differences in opinions or solution approaches between business and technology stakeholders.
Ability to identify and rationalise issues and provide leadership in progressing to a beneficial outcome.
Problem Solving
High level of problem solving and interpersonal skills.
Ability to view problems and potential issues from all perspectives (think out of the box).
Qualifications/experiences :
Degree level IT qualifications in Software or Systems Engineering.
Minimum 3 years to 8 Years of experience in IT of building and supporting Data Engineering pipeline on Big Data/ Cloud Data platforms.
Values
We are changemakers, We are better together, We care, We make it simple