Event Streaming & Messaging Systems

Discipline: Data Engineer | Module: Module 3: ETL/ELT Pipeline Engineering, Streaming & Orchestration | Estimated Study Time: 20 Hours

Welcome to Event Streaming & Messaging Systems. This topic delivers foundational and advanced concepts designed for production engineering and real-world workflows.

Key Learning Objectives

1. Distributed Log Architecture (Apache Kafka, Confluent, Apache Pulsar)
2. Cluster Internals (Brokers, Zookeeper/KRaft, Topics, Partitions, Consumer Groups)
3. Real-time Processing Engines (Spark Structured Streaming, Apache Flink)

Detailed Curriculum Breakdown

Distributed Log Architecture (Apache Kafka, Confluent, Apache Pulsar)

Explore the fundamental principles, real-world patterns, and best practices for Distributed Log Architecture (Apache Kafka, Confluent, Apache Pulsar). Practice hands-on implementations to master these concepts.

// Code Example: Distributed Log Architecture (Apache Kafka, Confluent, Apache Pulsar)
// Implement verified patterns for production use
console.log("Mastering Distributed Log Architecture (Apache Kafka, Confluent, Apache Pulsar)");

Cluster Internals (Brokers, Zookeeper/KRaft, Topics, Partitions, Consumer Groups)

Explore the fundamental principles, real-world patterns, and best practices for Cluster Internals (Brokers, Zookeeper/KRaft, Topics, Partitions, Consumer Groups). Practice hands-on implementations to master these concepts.

// Code Example: Cluster Internals (Brokers, Zookeeper/KRaft, Topics, Partitions, Consumer Groups)
// Implement verified patterns for production use
console.log("Mastering Cluster Internals (Brokers, Zookeeper/KRaft, Topics, Partitions, Consumer Groups)");

Real-time Processing Engines (Spark Structured Streaming, Apache Flink)

Explore the fundamental principles, real-world patterns, and best practices for Real-time Processing Engines (Spark Structured Streaming, Apache Flink). Practice hands-on implementations to master these concepts.

// Code Example: Real-time Processing Engines (Spark Structured Streaming, Apache Flink)
// Implement verified patterns for production use
console.log("Mastering Real-time Processing Engines (Spark Structured Streaming, Apache Flink)");

Practical Application & Exercises

1. Architecture Review: Evaluate how Event Streaming & Messaging Systems integrates with upstream and downstream systems.
2. Implementation Challenge: Build a functional prototype demonstrating each of the subtopics.
3. Validation & Testing: Verify performance and error handling under edge-case scenarios.

Summary Checklist

• Studied foundational architecture for Event Streaming & Messaging Systems
• Completed practical coding challenge
• Validated edge cases and error handling routines