Scribd Logo
Upload

Welcome to Scribd!

  • Data Pipeline Flow Sample

    Uploaded by

    paulocampo29
    2 views1 page
    The document describes a data pipeline architecture that involves collecting sensor data via SFTP, processing and masking the data through various stages including landing, raw, and data vault zones, and executing data pipelines to curate, transform, and aggregate the data for storage in BigQuery and Business Vault warehouses. The process is orchestrated through the use of Cloud Composer and Data Fusion technologies.

    Original Description:

    Sample data pieline

    Original Title

    Data Pipeline Flow sample

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    The document describes a data pipeline architecture that involves collecting sensor data via SFTP, processing and masking the data through various stages including landing, raw, and data vault zones, and executing data pipelines to curate, transform, and aggregate the data for storage in BigQuery and Business Vault warehouses. The process is orchestrated through the use of Cloud Composer and Data Fusion technologies.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    2 views1 page

    Data Pipeline Flow Sample

    Uploaded by

    paulocampo29
    The document describes a data pipeline architecture that involves collecting sensor data via SFTP, processing and masking the data through various stages including landing, raw, and data vault zones, and executing data pipelines to curate, transform, and aggregate the data for storage in BigQuery and Business Vault warehouses. The process is orchestrated through the use of Cloud Composer and Data Fusion technologies.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    of 1

    Curate/Transform/A

    ggregate
    File Processing
    SFTP Data Pipeline
    Execution Landing Data Raw
    Sensor
    Zone Masking Zone

    SFTP
    Cloud DLP
    Data BigQuery
    Composer Storage
    Fusion Data
    Vault

    API
    Data Pipeline
    Execution
    API Call
    Curate/Transform/A
    API ggregate
    source
    Composer Data
    Fusion

    BigQuery

    Business
    Vault

    You might also like