- Data Strategy Development: Collaborating with senior management and other stakeholders to develop a comprehensive data strategy aligned with the organization's business goals and objectives.
- Data Modeling: Designing and implementing data models that represent the structure, relationships, and constraints of the organization's data assets. This involves creating conceptual, logical, and physical data models to support various business processes and applications.
- Database Design and Optimization: Designing and optimizing database systems to ensure efficient storage, retrieval, and manipulation of data. This includes selecting appropriate database technologies, configuring database servers, and tuning database performance for optimal throughput and responsiveness.
- Data Integration: Designing and implementing data integration solutions to facilitate the seamless flow of data between different systems, applications, and databases. This may involve using Extract, Transform, Load (ETL) tools, data replication techniques, or application programming interfaces (APIs) to integrate disparate data sources.
- Data Governance: Establishing data governance policies, standards, and procedures to ensure the integrity, security, and quality of the organization's data assets. This involves defining data management practices, enforcing data compliance requirements, and implementing data quality controls.
- Metadata Management: Developing and maintaining metadata repositories to catalog and organize metadata about the organization's data assets. This includes capturing metadata attributes such as data lineage, data definitions, and data ownership to facilitate data discovery and understanding.
- Data Security and Privacy: Implementing data security measures and privacy controls to protect sensitive and confidential data from unauthorized access, disclosure, or misuse. This involves implementing encryption, access controls, and data masking techniques to safeguard data assets.
- Data Architecture Documentation: Creating and maintaining documentation of the organization's data architecture, including data models, data flow diagrams, and system architecture diagrams. This documentation serves as a reference for stakeholders and helps ensure consistency and alignment across the organization.
- Data Quality Management: Establishing data quality standards and processes to monitor, measure, and improve the quality of the organization's data assets. This involves identifying data quality issues, implementing data cleansing and enrichment techniques, and establishing data quality metrics and KPIs.
- Collaboration and Communication: Collaborating with cross-functional teams, including business analysts, software developers, and database administrators, to understand data requirements and ensure alignment with business needs. Effective communication skills are essential for conveying complex technical concepts to non-technical stakeholders.
- Technology Evaluation and Selection: Researching and evaluating emerging technologies and tools in the field of data management and analytics. This includes assessing the suitability of different technologies for specific use cases, conducting proof-of-concept evaluations, and making recommendations for technology adoption.
- Continuous Learning and Development: Keeping abreast of industry trends, best practices, and advancements in data architecture and data management. This involves attending conferences, participating in training programs, and pursuing professional certifications to enhance skills and knowledge in the field.
- Develop and maintain data ingestion pipelines.
- Extract data from various sources (databases, APIs, logs, etc.).
- Transform and clean data as needed before loading it into Redshift.
- Schedule and automate data ingestion processes.
- Monitor and optimise data ingestion performance.
- Set up and configure Redshift clusters based on workload requirements.
- Tune and optimize query performance through indexing and distribution strategies.
- Monitor and manage Redshift performance, including workload management and query optimization.
- Implement security measures and access controls for Redshift.
- Ensure high availability and disaster recovery for Redshift clusters.
- Develop ETL workflows using AWS Glue, Apache Spark, or other relevant tools.
- Transform and enrich data during the ETL process to meet business requirements.
- Handle schema evolution and data versioning in ETL pipelines.
- Monitor ETL job performance and troubleshoot issues.
- Implement data lineage and metadata management.
- Strong understanding of data engineering concepts and best practices.
- More than 13 years of experience in similar technical roles
- Proficiency in SQL and experience with data modeling techniques.
- Familiarity with AWS services, particularly Redshift, S3, and Glue.
- Knowledge of ETL (Extract, Transform, Load) processes and tools.
- Excellent problem-solving and troubleshooting skills.
- Strong communication skills to collaborate with cross-functional teams.
- Strong experience with Python or similar and Javascript highly preferred
- Enjoy your best years with our retirement savings plan
- Have peace of mind and body with our health insurance
- Make yourself a priority with flexible schedules, parental leave and our holiday purchase scheme
- Drive forward your career through professional development opportunities
- Achieve your personal goals with our Employee Assistance Programme
Technical Specialist, Data Engineering - Bengaluru, India - Carrier
Description
About the role
Experienced Cloud Operations professional. Oversees the cost/benefit analysis to determine the financial, operational, and technical feasibility of adopting external cloud-based IT platforms and infrastructure services. Manages the implementation of cloud services and integration with in-house applications and systems.
Key Responsibilities:
As a Technical specialist, you'll be responsible for:
Requirements
We are looking for people who are
Work on Data ingestion, handling database and work on ETL workflows
Data Ingestion:
AWS Redshift:
ETL (Extract, Transform, Load):
Minimum Requirements:
Benefits
We are committed to offering competitive benefits programs for all of our employees, and enhancing our programs when necessary.
Our commitment to you
Our greatest assets are the expertise, creativity and passion of our employees. We strive to provide a great place to work that attracts, develops and retains the best talent, promotes employee engagement, fosters teamwork and ultimately drives innovation for the benefit of our customers. We strive to create an environment where you feel that you belong, with diversity and inclusion as the engine to growth and innovation. We develop and deploy best-in-class programs and practices, providing enriching career opportunities, listening to employee feedback and always challenging ourselves to do better. This is .