prnsh's notes

computation

5 min read

The Computation stage is where data undergoes transformation and processing, whether in batch or real-time. This step typically involves cleaning, aggregating, filtering, or enriching data to prepare it for consumption, analytics, or machine learning.

1. Types of Data Processing

a) Batch Processing

  • Definition: Processes large volumes of data in scheduled intervals (e.g., daily, hourly). Typically involves high-latency but large-scale transformations.
  • Use Cases: Financial reporting, ETL jobs, daily aggregation tasks.
  • Tools:
    • Apache Hadoop: A distributed processing framework using the MapReduce paradigm.
    • Apache Spark (Batch mode): In-memory processing framework for large datasets.
    • AWS Batch: Managed batch processing on AWS infrastructure.
    • Google Cloud Dataproc: Managed Spark and Hadoop cluster for batch jobs.

b) Stream Processing

  • Definition: Processes data in real-time as it arrives. Data is processed in small chunks or events, enabling immediate actions or insights.
  • Use Cases: Fraud detection, recommendation engines, real-time analytics.
  • Tools:
    • Apache Kafka Streams: Lightweight stream processing library within Kafka.
    • Apache Flink: Stateful stream processing framework for real-time data.
    • Google Cloud Dataflow: Managed service for stream and batch processing.
    • Amazon Kinesis Data Analytics: Real-time data stream processing on AWS.

2. Key Processing Models

a) MapReduce (Batch)

  • Definition: A distributed processing model that splits tasks into independent chunks (Map) and processes them in parallel, followed by aggregation (Reduce).
  • Tools:
    • Apache Hadoop MapReduce: The original implementation of MapReduce.
    • Apache Spark (Batch Mode): Implements a similar distributed computing model but is more efficient due to in-memory processing.

b) Event-Driven (Stream)

  • Definition: Each event (or data record) is processed individually as it arrives, enabling low-latency processing. Event-driven architectures are often used for real-time decision-making.
  • Tools:
    • Apache Kafka Streams: Streams data in real-time and allows the building of lightweight processing applications.
    • Apache Flink: Stateful processing with event-time semantics, windowing, and complex event processing.

c) Lambda Architecture (Hybrid)

  • Definition: A hybrid processing model that combines batch and real-time stream processing to provide both low-latency and high-throughput capabilities.
  • Use Cases: Real-time analytics with historical data context.
  • Components:
    • Batch layer: Handles large-scale, batch-based data processing (e.g., Hadoop).
    • Speed layer: Handles real-time stream processing (e.g., Kafka, Flink).
    • Serving layer: Provides access to processed data for querying.

3. Data Transformation Techniques

a) ETL (Extract, Transform, Load)

  • Batch ETL: Data is extracted, transformed (cleaned, aggregated), and loaded in bulk to storage or databases.
    • Tools: Apache Nifi, Talend, Airflow.

b) ELT (Extract, Load, Transform)

  • Stream ETL/ELT: Data is ingested and loaded into the system first, and transformation is done on-demand or as the data is consumed.
    • Tools: Apache Kafka, AWS Kinesis, Flink.

4. Real-Time Stream Processing Techniques

a) Windowing

  • Definition: Organizing incoming data into fixed or sliding windows to perform calculations over a group of data points.
  • Use Cases: Real-time metrics, aggregations over a time period.
  • Tools:
    • Apache Flink: Advanced windowing support (tumbling, sliding windows).
    • Kafka Streams: Simple windowing capabilities (time-based).

b) Stateful vs. Stateless Processing

  • Stateful Processing: Maintains state across events for continuous calculations (e.g., tracking counts, aggregates, or windows).
    • Tools: Apache Flink, Kafka Streams.
  • Stateless Processing: Processes each event independently without maintaining any state.
    • Tools: Apache Kafka, Amazon Kinesis.

c) Event Time vs. Processing Time

  • Event Time Processing: Uses the timestamp of when an event occurred to process it (ideal for late-arriving data).
  • Processing Time Processing: Uses the timestamp of when the event was processed (simpler, but may have latency issues with out-of-order events).

5. Tools for Distributed Computation

a) Apache Spark (Batch and Streaming)

  • Batch Mode: Processes large datasets in parallel using RDDs (Resilient Distributed Datasets) or DataFrames.
  • Streaming Mode: Structured Streaming provides real-time data processing, handling both event-time and processing-time semantics.
  • Use Case: ETL jobs, machine learning, real-time analytics.
  • Supported Languages: Scala, Python, Java.
  • Stateful Stream Processing: Flink provides advanced capabilities like state management, event time processing, and windowing.
  • Use Case: Complex event processing, real-time analytics.
  • Supported Languages: Java, Scala, Python.

c) Apache Beam (Unified API for Stream & Batch)

  • Definition: A unified programming model that allows defining both batch and stream processing pipelines, with support for multiple runners like Spark, Flink, and Google Dataflow.
  • Use Case: Cross-platform data processing pipelines.
  • Supported Languages: Java, Python.

d) Cloud Native Tools

  • AWS Lambda: Serverless event-driven processing, ideal for small data chunks or microservices.
  • Google Cloud Dataflow: Unified stream and batch processing using Apache Beam.
  • Azure Stream Analytics: Real-time data stream analytics.

6. Challenges and Optimization Techniques

a) Latency

  • Solution: Use real-time processing tools (e.g., Flink, Kafka Streams) and optimize data pipelines for low-latency execution.

b) Scalability

  • Solution: Distribute workloads across multiple nodes. Use cluster-based frameworks like Spark and Flink that scale horizontally.

c) Data Consistency

  • Solution: Implement mechanisms for handling out-of-order data and late arrivals using watermarks and event-time processing.

Example: Real-Time Fraud Detection Pipeline

Scenario:

A real-time fraud detection system for credit card transactions.

Tech Stack:

  • Source: Transactions streamed via Apache Kafka.
  • Stream Processing: Apache Flink for real-time event-driven processing, applying machine learning models for fraud detection.
  • Storage: Processed data is stored in a data warehouse (Snowflake) for further analysis.
  • Analytics: Alerts generated for suspicious transactions, and data is ingested into the analytics dashboard.

Workflow:

  1. Kafka streams transaction data.
  2. Flink processes transactions in real-time, checking for anomalies.
  3. If a fraud pattern is detected, an alert is triggered and sent to a monitoring system.
  4. The transaction data is loaded into Snowflake for further aggregation and reporting.

Graph View