prnsh's notes

ingestion

3 min read

The Ingestion stage focuses on transferring collected data into a centralized environment where it can be processed. This stage ensures data arrives in a reliable, scalable, and timely manner. Here’s a detailed breakdown:

1. Purpose

  • Move raw or partially processed data into storage or processing systems.
  • Ensure data consistency, quality, and availability for downstream tasks.

2. Types of Data Ingestion

a) Batch Ingestion

  • Data is moved at scheduled intervals or in chunks.
  • Use Case: Periodic reports, daily transaction summaries.
  • Tools:
    • Apache Sqoop: Transfers bulk data between RDBMS and Hadoop.
    • AWS Glue: ETL service with batch processing.
    • AzCopy: Transfers large datasets to Azure Blob Storage.

b) Streaming Ingestion

  • Real-time data transfer as it is generated.
  • Use Case: IoT sensor data, real-time analytics.
  • Tools:
    • Apache Kafka: Distributed streaming platform.
    • Amazon Kinesis: Real-time data streaming service.
    • Google Pub/Sub: Messaging service for real-time pipelines.

3. Industry-Standard Tools

a) ETL Tools (Extract, Transform, Load)

  • Automates ingestion, transforming data before loading into storage or compute layers.
  • Examples:
    • Talend: Offers connectors for databases, cloud services, APIs.
    • Informatica PowerCenter: Enterprise-grade ETL tool.
    • Fivetran: Extracts and loads data with minimal transformations.

b) Data Pipeline Frameworks

  • Apache NiFi: Automates data movement with flow-based programming.
  • Luigi: Orchestrates batch data pipelines.
  • Airflow: Schedules and monitors ingestion jobs.

c) Cloud-Specific Ingestion Services

  • AWS Data Pipeline: Orchestrates data movement on AWS.
  • Azure Data Factory: Transfers data between on-prem and cloud.
  • Google Dataflow: Real-time and batch data processing.

4. Protocols and Formats

  • Data Transfer Protocols:

    • HTTP/HTTPS: Common for APIs.
    • SFTP/FTP: Secure file transfers.
    • gRPC: High-performance RPC protocol.
    • Kafka Connect Protocol (KIP): Kafka connector framework.

  • Data Formats:

    • Structured: CSV, JSON, Parquet.
    • Semi-structured: Avro, ORC.
    • Binary: Protocol Buffers, Apache Thrift.

5. Workflows for Ingestion

a) Direct Ingestion

  • Raw data is directly ingested into storage or compute environments.
  • Example: Logs shipped from Filebeat to Elasticsearch.

b) Buffered Ingestion

  • Data is first buffered in message brokers or streaming platforms before loading.
  • Example:
    • Logs ingested via Apache Kafka, stored temporarily, and then loaded into Snowflake for analytics.

6. Monitoring and Optimization

a) Monitoring Tools

  • Prometheus + Grafana: Monitors throughput and failures.
  • Datadog/CloudWatch: Tracks ingestion latency and resource usage.

b) Challenges & Mitigations

  • High Latency: Use streaming tools like Kafka to reduce delays.
  • Data Loss: Implement durable storage or replication (e.g., Kafka’s replication factor).
  • Backpressure: Manage flow control using tools like Apache Flink.

Example: Real-Time Sensor Data Ingestion

Scenario: Ingest IoT sensor data from edge devices into a cloud system for real-time analytics.

Tech Stack:

  • Source: Edge devices emitting temperature data.
  • Buffer: Kafka handles real-time streaming.
  • Storage: Data is loaded into AWS S3 for batch analysis.
  • Workflow:
    1. Devices push data to Kafka topics.
    2. Kafka Connect transfers data from Kafka to S3.
    3. AWS Glue processes the data for analytics.

Graph View