Role &amp; responsibilities:

<ul>
<li style="margin-left:0px;">Own end-to-end technical design and architecture of batch and streaming data pipelines</li>
</ul>

<ul>
<li style="margin-left:0px;">Lead development of high-throughput, low-latency data processing systems using Java</li>
</ul>

<ul>
<li style="margin-left:0px;">Design and implement real-time streaming solutions (event ingestion, processing, aggregation)</li>
</ul>

<ul>
<li style="margin-left:0px;">Drive code quality, performance optimization, and scalability across systems</li>
</ul>

<ul>
<li style="margin-left:0px;">Review code, mentor engineers, and set engineering best practices</li>
</ul>

<ul>
<li style="margin-left:0px;">Collaborate with product, platform, infra, and security teams to deliver production-grade solutions</li>
</ul>

<ul>
<li style="margin-left:0px;">Ensure data reliability, fault tolerance, and observability (metrics, logs, alerts)</li>
</ul>

<ul>
<li style="margin-left:0px;">Automate operational workflows and deployment pipelines</li>
</ul>

<ul>
<li style="margin-left:0px;">Participate in capacity planning, cost optimization, and system tuning</li>
</ul>

<ul>
<li style="margin-left:0px;">Stay current with evolving Big Data and streaming ecosystems and evaluate new technologies</li>
</ul>

Requirements:

<ul>
<li style="margin-left:0px;">Strong proficiency in Java (multithreading, concurrency, JVM tuning)</li>
</ul>

<ul>
<li style="margin-left:0px;">Hands-on experience with distributed data processing frameworks:</li>
</ul>

<ul>
<li style="margin-left:0px;">Apache Spark (Core, SQL, Streaming / Structured Streaming / Apache Flink)</li>
</ul>

<ul>
<li style="margin-left:0px;">Apache Kafka (producers, consumers, partitions, offsets, exactly-once semantics)</li>
</ul>

<ul>
<li style="margin-left:0px;">Solid understanding of batch + streaming architectures (Lambda / Kappa patterns)</li>
</ul>

<ul>
<li style="margin-left:0px;">Experience with Hadoop ecosystem components (HDFS, Hive, YARN or equivalents)</li>
</ul>

<ul>
<li style="margin-left:0px;">Strong knowledge of relational databases (PostgreSQL, MySQL)</li>
</ul>

<ul>
<li style="margin-left:0px;">Experience with NoSQL / distributed datastores (HBase, Cassandra, MongoDB, Pinot etc.)</li>
</ul>

<ul>
<li style="margin-left:0px;">Understanding of data modeling, partitioning, and schema evolution</li>
</ul>

<ul>
<li style="margin-left:0px;">Experience with Linux-based systems and production deployments</li>
</ul>

<ul>
<li style="margin-left:0px;">Exposure to containerization and orchestration (Docker, Kubernetes preferred)</li>
</ul>

<ul>
<li style="margin-left:0px;">Familiarity with monitoring and observability tools (Grafana, Prometheus, ELK, etc.)</li>
</ul>

<ul>
<li>Experience with CI/CD pipelines and automated testing frameworks</li>
</ul>

<ul>
<li style="margin-left:0px;">Proven experience leading technical teams and driving delivery</li>
</ul>

<ul>
<li style="margin-left:0px;">Strong problem-solving and debugging skills in complex distributed systems</li>
</ul>

<ul>
<li style="margin-left:0px;">Ability to take ownership of critical systems and make architectural decisions</li>
</ul>

<ul>
<li style="margin-left:0px;">Excellent communication skills to work with cross-functional stakeholders</li>
</ul>

<ul>
<li style="margin-left:0px;">Comfortable working in a fast-paced, evolving data ecosystem</li>
</ul>

Nice to Have:

<ul>
<li style="margin-left:0px;">Experience with Flink / Kafka Streams / real-time analytics</li>
</ul>

<ul>
<li style="margin-left:0px;">Exposure to cloud-native data platforms (AWS, GCP, Azure)</li>
</ul>

<ul>
<li style="margin-left:0px;">Knowledge of data governance, security, and access control</li>
</ul>

<ul>
<li style="margin-left:0px;">Experience in telecom, fintech, or large-scale consumer data platforms</li>
</ul>

Big Data Engineering

India

Role &amp; responsibilities: Own end-to-end technical design and architecture of batch and streaming data pipelines Lead development of high-throughput, low-latency data processing systems using Java Design and implement real-time streaming solutions (event ingestion, processing, aggregation) Drive code quality, performance optimization, and scalability across systems Review code, mentor engineers, and set engineering best practices Collaborate with product, platform, infra, and security teams to deliver production-grade solutions Ensure data reliability, fault tolerance, and observability (metrics, logs, alerts) Automate operational workflows and deployment pipelines Participate in capacity planning, cost optimization, and system tuning Stay current with evolving Big Data and streaming ecosystems and evaluate new technologies Requirements: Strong proficiency in Java (multithreading, concurrency, JVM tuning) Hands-on experience with distributed data processing frameworks: Apache Spark (Core, SQL, Streaming / Structured Streaming / Apache Flink) Apache Kafka (producers, consumers, partitions, offsets, exactly-once semantics) Solid understanding of batch + streaming architectures (Lambda / Kappa patterns) Experience with Hadoop ecosystem components (HDFS, Hive, YARN or equivalents) Strong knowledge of relational databases (PostgreSQL, MySQL) Experience with NoSQL / distributed datastores (HBase, Cassandra, MongoDB, Pinot etc.) Understanding of data modeling, partitioning, and schema evolution Experience with Linux-based systems and production deployments Exposure to containerization and orchestration (Docker, Kubernetes preferred) Familiarity with monitoring and observability tools (Grafana, Prometheus, ELK, etc.) Experience with CI/CD pipelines and automated testing frameworks Proven experience leading technical teams and driving delivery Strong problem-solving and debugging skills in complex distributed systems Ability to take ownership of critical systems and make architectural decisions Excellent communication skills to work with cross-functional stakeholders Comfortable working in a fast-paced, evolving data ecosystem Nice to Have: Experience with Flink / Kafka Streams / real-time analytics Exposure to cloud-native data platforms (AWS, GCP, Azure) Knowledge of data governance, security, and access control Experience in telecom, fintech, or large-scale consumer data platforms

Big Data Engineering · Gurugram

Job highlights

Skillset required

Job Description

Job Description for Big Data Engineering

More at AIRTEL