prnsh's notes

collection

3 min read

The Collection stage involves gathering raw data from a variety of sources and preparing it for ingestion. Below are industry-standard tools and methodologies for this stage, focusing on scalability, reliability, and efficiency.

1. Data Sources and Industry-Standard Tools

a) Databases

  • Relational Databases (RDBMS): MySQL, PostgreSQL, Oracle, SQL Server.

    • Extraction Tools:
      • Apache Sqoop: Transfers bulk data between RDBMS and Hadoop systems.
      • Debezium: Change Data Capture (CDC) for real-time replication.
      • Airbyte: Open-source tool supporting numerous database connectors.

  • NoSQL Databases: MongoDB, Cassandra, DynamoDB.

    • Extraction Tools:
      • Kafka Connect: Pre-built connectors for NoSQL databases.
      • Custom Python Scripts: Using libraries like pymongo for MongoDB or cassandra-driver for Cassandra.

b) Data Streams

  • Real-Time Streams: Apache Kafka, Amazon Kinesis, Apache Pulsar.
    • Data from event-driven systems, IoT devices, or clickstreams.
    • Key Tools:
      • Confluent Kafka Connectors: Out-of-the-box connectors for ingesting and exporting.
      • Fluentd: Aggregates logs and event data into Kafka or Kinesis.

c) Applications (APIs)

  • APIs from SaaS platforms: Salesforce, Zendesk, Shopify, Stripe.
    • Extraction Tools:
      • MuleSoft Anypoint: Connects APIs with customizable logic.
      • Fivetran: Automates API data syncing into data warehouses.
      • Python + REST libraries: Custom integrations using requests or httpx.

d) File Systems and Cloud Storage

  • On-Premise: Local file servers, shared drives.
    • Tools: Rclone, SFTP scripts.
  • Cloud Storage: AWS S3, Google Cloud Storage, Azure Blob Storage.
    • Tools: AWS DataSync, Google Transfer Service, Azure Data Factory.

e) Third-Party Data Providers

  • Examples: Social media (Twitter API, Facebook Graph API), financial data providers (Quandl, Bloomberg).
    • Extraction Tools: Python libraries like tweepy for Twitter, direct integration with vendor-provided SDKs.

2. Data Collection Frameworks

a) Log Aggregators

  • Elastic Beats (e.g., Filebeat, Metricbeat):
    • Collect and ship log files or metrics to Elasticsearch or Kafka.
  • Logstash:
    • Processes unstructured log data from servers or applications.

b) Event Trackers

  • Snowplow Analytics: Collects web and mobile app event data for behavioral analysis.
  • Google Tag Manager (GTM): Captures events from websites or apps.

3. Data Formats

  • Structured Data: CSV, JSON, Parquet.
  • Semi-Structured Data: XML, Avro, ORC.
  • Unstructured Data: Plain text, images, videos.

Tools to Standardize Formats:

  • Apache Avro/Parquet: Optimized for big data and Hadoop ecosystems.
  • Schema Registry (e.g., Confluent Schema Registry): Ensures consistent schema management.

4. Workflows and Protocols

a) Data Pull vs. Push

  • Pull-Based (On-Demand):
    • REST API integrations, database querying.
  • Push-Based (Streaming):
    • Webhooks, Kafka Producers.

b) Protocols

  • HTTP/HTTPS: For APIs.
  • SFTP/FTP: For file transfers.
  • Message Queues (AMQP): RabbitMQ, ActiveMQ.
  • gRPC: High-performance RPC for internal services.

5. Monitoring and Error Handling

  • Tools:
    • Prometheus + Grafana: Tracks data collection latency and failures.
    • Datadog/CloudWatch: Monitors API performance and database connectivity.
  • Retry Logic: Built into tools like Kafka Connect or custom scripts.

Example: Log Data Collection System

Scenario: Collect web server logs in near real-time.

  • Source: NGINX logs on multiple servers.
  • Tech Stack:
    • Filebeat: Sends logs to Kafka.
    • Kafka: Streams logs to a central processing cluster.
    • Elasticsearch: Stores logs for search and analysis.
    • Monitoring: Prometheus tracks Kafka lag and Filebeat throughput.

Workflow:

  1. Filebeat reads logs from /var/log/nginx/access.log and forwards them to Kafka.
  2. Kafka partitions the logs for parallel processing.
  3. Logs are written to Elasticsearch for visualization in Kibana.

Graph View