Datastage ETL Developer - Bengaluru, India - PureSoftware Ltd

Description

This is an ETL application development position that requires data requirements analysis, design and development of ETL applications using IBM Infosphere DataStage.

ETL tool : IBM Infosphere DataStage Suite (Designer, Director, Administrator)

Database: DB2, Postgres, Azure SQL DB

Scripting: Unix shell script

Requirements / Qualifications:

· Bachelor's Degree; computer science, software engineering, information systems or a related technical discipline preferred.

· 4+ years of technical experience using DataStage (preferably working with version 11.7 for at least 1-2 years).

· Good to have of insurance/banking/financial domain.

· Preference for an overall technical background in Data Management, Data Integration and/or Data Warehousing

· Strong experience with Postgres, Azure SQL DB and IBM DB2 enterprise databases.

· Experience with performance tuning of DataStage jobs working in conjunction with a DBA.

· Strong production support experience

· Experience working in a large data warehousing initiative.

· Experience working on applications running in UNIX environments.

· Experience with XML, IBM MQ, Real Time and Near Real Time integrations with IBM DataStage

· Advanced proficiency with SQL and related-RDBMS query tools and manage complex SQL queries, Stored Procedures

· Ability to clearly communicate solutions to both technical and non-technical teams.

· Experience gathering and translating functional requirements.

· Ability to manage working across concurrent projects at different stages of completion.

· Project implementation experience using DataStage.

· Developing and Implementing Data Integration ETL Solutions with DataStage Should be a good team player as the role would need collaboration.

· Analysis and trouble shooting skills.

· Experience in performance tuning of ETL code.

Optional Nice to Have any of the following: -

As a Data Engineer, you will work with multiple teams to deliver solutions on the Azure Cloud using core cloud data warehouse tools (Azure Data Factory, Azure Databricks, Azure SQL DW and other Big Data related technologies). In addition to building the next generation of application data platforms (not infrastructure) and/or improving recent implementations. Note: This is a data engineer from the application side. Must be able to analyze data and develop strategies for populating data lakes. This is not an infrastructure position. This person may be called upon to do complex coding using U-SQL, Scala or Python and T-SQL.

Data Engineers with Batch processing

• Experience with at least one Hadoop distribution (Hortonworks / Clourdera / MapR)
• Batch processing using Apache Spark (Spark SQL, DataFrame / DataSet API)
• Knowledge of ETL processing tools (Sqoop, Hive)
• Hive / HBase experience
• Experience of working with multiple Hadoop file formats (Avro / Parquet / ORC)

Data Engineers with Stream processing

• Experience with at least one Hadoop distribution (Hortonoworks / Clourdera / MapR)
• Knowledge of Stream Processing frameworks (Spark Streaming / Kafka Streams / Flink)
• Stream processing using Apache Spark Streaming / Kafka Streams(Spark SQL, DataFrame / DataSet API, Structured Streaming, KSQL)
• Knowledge of Kafka
• Hive / HBase experience