Data Processing Pipeline

From discovery, through processing to data driven decisions http://freshdata.sk

Data Sources

Discovery and Acquisition

Extraction

Cleansing, Transformation and Integration

Analytical Modeling

Presentation, Exploration and Publishing

web pages

Audit

Loading to Data Store

Data Formats and Standards

Data Granularity

Using Reference Data

Analytical Model Development

Online Analytical Processing

Visualization Method Selection

Visualization and Plotting

text documents

Data Pipes Crawling Merging, Joining Mapping Handling Manual Corrections

Business Rules

Graph/Network Metrics

Report Development

Publishing Online

structured documents

Manual Digitization

Regression Scraping Normalization Treating Duplicates Entity Uniqueness Segmentation and Clustering

Outliers

Story Telling

Map Geo-Tagging

Bulk Digitization databases

Parsing

Changing Dimensions

Shopping Basket Analysis

Decisioning

Automation Crowd Sourcing scientic data Natural Language Processing Indexing and Optimization

Customer Value Computation

Behavior and Impact

Campaign Management

Automated Decisioning

Simulation

Governance
ETL Process Management Data Quality Management Auditability and Provenance Master Data Management Metadata

cbna

Stefan Urbanek @Stiivi 2013 v0.3