Senior Data Engineer - Bengaluru, India - GyanSys Inc.
Description
At least 8+ years' experience, ideally within a Data Engineer role.
Demonstrated experience working with large and complex data sets as well as experience analyzing volumes of data.
Excellent experience working Python, Pandas, Flask/Fast/Django API, Middleware, Scheduler, SQL, Databases.
Prior experience with Python frameworks such as Django or Flask and a strong knowledge of SQL (queries, joins, etc.).
Good to have some experience in AWS/Azure.
Capability of developing highly-scalable RESTful APIs.
Excellent team player and can work well in an individual capacity as well.
Detail-oriented and possess strong analytical skills.
Pays strong attention to detail and deliver work that is of a high standard.
Highly goal-driven and work well in fast-paced environments.
Python, RestAPI, Redis Cache, SQl – ostgres/MariaDB/Clickhouse, Kubernetes, Linux/Window scheduler, Shell Script
1. DataOPS - Python Core / Advanced - Development and Data Pipelining - Data Structures, Pandas, Numpy, sklearn, concurrency, design patterns
2. DevOPS - App Deployment using CI/CD tools – like Jenkins, Jfrog, Docker, Kubernetes, Openshift Container Platform
3. Microservices & REST APIs - FastAPI, Flask, Tornado
4. Cloud – how apps are build and deployed using cloud
5. Databases & SQL – Postgres, Clickhouse, MongoDB
6. Caching & Queuing - Pub/Sub(RabbitMQ), Redis
7. Operating system - Linux and windows
8. Monitoring and Logging – Splunk
Good to have skills -
1. Generative AI knowledge - Langchain framework knowledge, ChatGPT knowledge
2. MLOPS knowledge - Databricks, MLFlow, Kubeflow, ClearML
3. Testing knowledge - Integration testing, Python Behave, Pytest
4. Maintaining code quality standards - working knowledge of Pylint
5. Logging – Kibana, Elastic search