prnsh's notes

storage

3 min read

The Storage stage involves organizing and storing ingested data so it can be processed, queried, and analyzed efficiently. The choice of storage depends on the type of data (structured, semi-structured, unstructured), performance needs, and integration with downstream systems.

1. Types of Storage Systems

a) Data Lakes

  • Purpose: Stores raw, unstructured, or semi-structured data in its native format.
  • Key Features:
    • Scalability for large datasets.
    • Schema-on-read: Schema is applied during query time.
    • Flexible for advanced analytics and machine learning.
  • Popular Tools:
    • Amazon S3: Core service for AWS data lakes.
    • Azure Data Lake Storage Gen2: Optimized for big data workloads.
    • Google Cloud Storage (GCS): Supports various data formats and integrations.

b) Data Warehouses

  • Purpose: Optimized for structured, tabular data and analytical queries.
  • Key Features:
    • Schema-on-write: Data is transformed and structured during ingestion.
    • High performance for SQL-based analytics.
    • Suitable for business intelligence (BI) applications.
  • Popular Tools:
    • Snowflake: Cloud-native, supports semi-structured data with high concurrency.
    • Google BigQuery: Serverless data warehouse for ad hoc querying.
    • Amazon Redshift: Columnar storage for fast query performance.
    • Microsoft Synapse Analytics: Combines warehouse and big data capabilities.

c) Data Lakehouses

  • Purpose: Hybrid solution combining features of data lakes and warehouses.
  • Key Features:
    • Handles structured and unstructured data.
    • Unified storage layer for batch and real-time processing.
    • Low-cost storage with high-performance query engines.
  • Popular Tools:
    • Databricks Lakehouse Platform: Built on Apache Spark and Delta Lake.
    • Apache Hudi: Supports data versioning and incremental updates.
    • Apache Iceberg: Handles large-scale table format management.

2. Storage Formats

  • Text-Based Formats: CSV, JSON, XML.
    • Simple but inefficient for large datasets.
  • Optimized Formats:
    • Apache Parquet: Columnar storage for analytical workloads.
    • Apache Avro: Row-based, schema evolution-friendly.
    • ORC (Optimized Row Columnar): Efficient for large-scale processing in Hadoop.

3. Partitioning and Indexing

  • Partitioning divides data into smaller segments based on key fields like date or region.
    • Example: /data/sales/2025/01/17/
  • Indexing accelerates queries by creating metadata structures.
    • Example: Delta Lake’s Z-order indexing.

4. Storage Layer Technologies

a) On-Premise Systems

  • HDFS (Hadoop Distributed File System): Distributed file system for big data storage.
  • Ceph: Distributed object storage for flexible data formats.

b) Cloud-Native Systems

  • Amazon S3: Supports storage tiers (e.g., Glacier for archiving).
  • Google BigQuery Storage API: Direct integration with analytical tools.
  • Azure Blob Storage: Compatible with tools like Synapse and Data Factory.

5. Access and Security

  • Role-Based Access Control (RBAC): Manage permissions for users and applications.
    • Tools: AWS IAM, Azure Active Directory.
  • Data Encryption: Protect data at rest and in transit.
    • Standards: AES-256 encryption.
  • Data Governance:
    • Apache Atlas: Metadata management.
    • Collibra: Tracks data lineage and compliance.

6. Integration with Compute Engines

  • Big Data Frameworks: Apache Spark, Hadoop.
  • Query Engines:
    • Presto/Trino: High-performance SQL on distributed storage.
    • Athena (AWS): Serverless query engine for S3.

Example: Hybrid Storage System for E-commerce Data

Scenario:

  • Data Types:
    • Raw clickstream data (JSON) → Stored in a data lake (S3).
    • Aggregated sales reports (Parquet) → Stored in a data warehouse (Redshift).
  • Workflow:
    1. Raw data ingested into Amazon S3 via Kafka Connect.
    2. S3 bucket is partitioned by date for efficient access.
    3. ETL jobs (AWS Glue) transform and load data into Redshift.
    4. Delta Lake on S3 manages transactional data updates.

Graph View