Apache Flink

systemdesign distributedsystem

Introduction

Apache Flink is an open-source framework for distributed stream processing (and batch processing too, though streaming is its core strength). It’s used to process large amounts of data continuously, in real time, as it arrives — think log analysis, fraud detection, real-time analytics dashboards, IoT sensor processing, etc.

Batch Processing and Stream Processing

Core idea: everything is a stream

Flink treats data as an unbounded stream of events. Even “batch” jobs are just a special case — a stream that has a known end. This is different from older frameworks (like classic Hadoop MapReduce, or even early Spark) that were built batch-first and bolted streaming on afterward.

Basic Concepts

Sources/Sinks

A Flink job starts at a source (where data enters the pipeline) and ends at a sink (where results leave the pipeline).

• Sources: Apache Kafka topics, Kinesis streams, file systems, databases (via Change Data Capture (CDC)), sockets, or even custom generators. A source is responsible for reading raw data and turning it into a stream of records Flink can process.
• Sinks: databases, Kafka topics, data warehouses, files, or external APIs. A sink writes the final processed output somewhere durable or consumable.

Between source and sink sit the transformations that make up the actual logic of the job.

Streams

A stream is Flink’s core abstraction: an unbounded (or bounded) sequence of records flowing through the system over time. Everything in Flink — even a “batch” job — is modeled as a stream; a batch is just a stream with a known, finite end.

Key properties of a stream:

• Records typically carry a timestamp (when the event occurred).
• Streams can be keyed (partitioned by some attribute, e.g., userId) so that all records for the same key are routed to the same parallel task — this is essential for stateful, per-key logic like counting clicks per user.
• Streams flow through a DAG (directed acyclic graph) of operators.

Operators

Operators are the transformation steps applied to a stream. Common operator types:

• map — transform each record independently (1-to-1)
• filter — keep/drop records based on a predicate
• flatMap — turn one record into zero or more records
• keyBy — partition the stream by a key, enabling stateful per-key processing
• window — group records over time into finite chunks for aggregation
• reduce / aggregate — combine records within a key/window
• connect / join — combine two streams

Operators are chained together to form the job’s processing graph, and Flink runs many parallel instances of each operator across the cluster.

State

State is what makes stream processing hard — and what Flink is built to manage well. State is any information an operator needs to remember across records (e.g., a running count, a list of recent events, a session’s data).

Flink provides:

• Keyed state: scoped to a specific key (e.g., per-user counters), accessible only within operators on a keyBy’d stream.
• Operator state: scoped to an operator instance, not tied to a key (used in sources/sinks, e.g., tracking Kafka offsets).

Because state can grow large and must survive failures, Flink stores it in pluggable state backends (covered later) and periodically snapshots it via checkpoints, so a crashed job can resume from the last consistent snapshot instead of losing everything or replaying from the very beginning.

Watermarks

Events don’t always arrive in the order they occurred — network delays, retries, and mobile connectivity mean data can be late or out of order. Flink uses event time (when something actually happened) rather than processing time (when Flink received it) for correctness, but this requires a way to know “how much event time has passed” so windows can be closed.

A watermark is essentially a moving marker that says: “I believe all events with a timestamp earlier than W have now arrived.” Operators use watermarks to decide when it’s safe to close a time window and emit results, even though the stream itself never truly “ends.” Watermarks involve an inherent tradeoff: wait longer for more complete (accurate) data, or emit sooner for lower latency, with some risk of dropping/handling late data separately.

Windows

Since a stream is infinite, you can’t aggregate “over the whole stream” the way you would with a batch dataset — you need to bound the aggregation using windows. Common types:

• Tumbling windows: fixed-size, non-overlapping (e.g., “every 5 minutes”).
• Sliding windows: fixed-size, but overlapping (e.g., “the last 5 minutes, updated every 1 minute”).
• Session windows: dynamic, based on gaps of inactivity (e.g., “group events that occur within 10 minutes of each other for the same user”).

Windows work hand-in-hand with watermarks: the watermark tells Flink when a window’s data is “complete enough” to be evaluated and emitted.

Basic Use

Defining a Job

A Flink job is defined as a pipeline: source → series of operators/transformations → sink. In code (Java/Scala/Python via the DataStream API), this typically looks like reading from a source, applying keyBy, window, and aggregation operators, then writing to a sink. The job definition is essentially a description of the DAG Flink should construct and execute.

Submitting a Job

Once defined, a job is compiled into an execution graph and submitted to a Flink cluster (either a long-running session cluster or a per-job cluster). Submission can happen via:

• The Flink CLI (flink run)
• The Flink Web UI / REST API
• Deployment tooling (e.g., Kubernetes operators, Flink on YARN)

The cluster’s JobManager receives the submitted job, schedules its tasks, and distributes them across available TaskManagers for execution.

Sample Jobs

Basic Dashboard Using Redis

A common real-world pattern: consume events from Kafka, perform a windowed aggregation (e.g., “page views per minute”), and write the running aggregate into Redis so a dashboard can read fast, up-to-date metrics without querying the raw event stream directly.

Fraud Detection System

Another canonical use case: consume a stream of transactions, keep per-user state (e.g., recent transaction patterns), apply rules or pattern-matching logic (e.g., using Flink’s ProcessFunction or Flink CEP for complex event patterns) to flag suspicious activity in near real-time, and emit alerts to a sink (e.g., another Kafka topic or alerting system).

How Flink Works

Cluster Architecture

Job Manager and Task Managers

Flink clusters follow a master/worker model:

• JobManager: the “brain” of the cluster. It receives submitted jobs, builds the execution graph, schedules tasks onto TaskManagers, coordinates checkpoints, and handles failure recovery.
• TaskManagers: the “workers.” Each one runs one or more task slots, executing the actual operator logic and exchanging data with other TaskManagers as the stream flows through the DAG.

Task Slots and Parallelism

Each TaskManager is divided into task slots, which represent a fixed slice of resources (e.g., CPU/memory) where one parallel instance of a task can run. The parallelism of a job determines how many parallel instances of each operator are created — higher parallelism means more slots are used and the workload is split across more parallel subtasks, enabling horizontal scalability.

State Management

State Backends

A state backend determines where and how an operator’s state is stored. Common options:

• HashMapStateBackend: state lives in memory on the TaskManager’s JVM heap — fast, but limited by available memory and not ideal for very large state.
• RocksDBStateBackend: state is stored on local disk using an embedded RocksDB instance — supports much larger state than memory allows, at the cost of some latency, and integrates well with incremental checkpointing.

Checkpointing and Exactly-Once Processing

Flink achieves fault tolerance through checkpointing: it periodically takes a consistent, distributed snapshot of all operators’ state across the entire job, along with the position in each source (e.g., Kafka offsets). These snapshots are stored in a durable location (e.g., distributed file system or object store).

If a failure occurs, Flink restores all operator state from the most recent checkpoint and replays the source from the recorded position — ensuring no data is lost. Combined with transactional sinks (e.g., Kafka’s transactional producer), this enables exactly-once processing semantics: each event affects the final result exactly once, even across failures — critical for use cases like billing, fraud detection, or financial systems where double-counting or dropped events are unacceptable.

In Your Interview

Using Flink

In a system design interview, reach for Flink when a problem genuinely requires continuous, stateful, real-time processing — not just “data needs to move from A to B.” Good signals: windowed aggregations, per-key running state, low-latency requirements, or out-of-order/late event handling. If the problem can tolerate periodic batch jobs instead, say so — proposing Flink when a simple batch job (Spark, cron job, etc.) would suffice is often seen as over-engineering.

Lessons from Flink

Even when you don’t use Flink directly, the concepts it embodies are valuable to articulate in interviews:

• Stateful stream processing is hard because of failure recovery, rebalancing/scaling, and out-of-order data — naming these challenges shows depth.
• Event time vs. processing time and the role of watermarks is a subtlety that distinguishes a strong stream-processing design from a naive one.
• Exactly-once semantics require coordination between state checkpoints and sink transactions — it’s not free, and it’s worth explicitly calling out when your design needs it (and when it doesn’t).