Chapter 8

Spatial analysis and geoprocessing

Instructor
Chapter objectives

• Understand the web-based analysis tools provided in ArcGIS

• Know the collection of variables available from ArcGIS Online
• Understand the basics of ArcGIS Insights
• Create web apps that use ArcGIS Online analysis
• Author and share web tools with ArcGIS Enterprise
• Use web tools in web apps
• Understand the workflow for big data analysis
• Understand ArcGIS Notebooks

Getting to Know Web GIS, fourth edition

Data sources Desktop tools ArcGIS Online / ArcGIS Enterprise Client apps
CSV files Feature layers Ready-to-use apps and
configurable templates
Shapefiles Raster tile layers
Web Configurable app templates
KMLs Vector tile layers maps/
web ArcGIS StoryMaps
File
File geodatabases
geodatabases Scene layers
scenes Experience
Experience Builder/Web
Builder/WebAppBuilder
AppBuilder
Enterprise geodatabases ArcMap Map image layers
ArcGIS Dashboards
Nonspatial data ArcGIS Pro Image layers ArcGIS Insights
Tools ArcGIS Feeds ArcGIS Collector, Survey123,
Photos/Imagery Drone2Map Stream layers Explorer, QuickCapture, Tracker,
Navigator, and Workforce …
Raster function templates ArcGIS Custom web
web tools
tools
ArcGIS AppStudio
Sensor and real-time data CityEngine Standard
Standard tools
tools ArcGIS Earth
Big
Big data
data GeoAnalytics
GeoAnalytics tools/big
tools/big ArcGIS Indoors viewer
data analytic
analytics
… ArcGIS VR 360
Real-time analytics
AuGeo
360 VR Experience
…
Geometry services
Custom apps
Living Atlas
Atlas of
of the
theWorld
World
ArcGIS API for JavaScript
… ArcGIS Runtime SDKs
ArcGIS
ArcGIS API
API for
forPython
Python
… Getting to Know Web GIS, fourth edition
Web GIS is more than mapping

Understanding and Determining how places Finding the best locations

measuring places are related

Detecting and quantifying Making predictions Directions, routing,

patterns and logistics

Getting to Know Web GIS, fourth edition

Spatial analysis and spatial data science

• Spatial analysis holds great practical value for our personal life and for enterprise operations

• Spatial data science allows analysts to extract deeper insight from data using a comprehensive
set of analytical methods and spatial algorithms, including machine learning and deep learning
techniques

• Web-based spatial analysis allows more users to discover relationships, patterns, and trends in
geospatial data

Getting to Know Web GIS, fourth edition

ArcGIS provides web tools and clients for spatial analysis
Web tools Clients

For standard-sized ArcGIS Map Viewer

Standard tools
data ArcGIS Web AppBuilder
Feature ArcGIS Experience Builder
analysis Insights
For big data in vector
Geoanalytics tools
format ArcGIS Pro
Ready-to-use ArcGIS Notebooks
web tools
Raster ArcGIS API for Python
For raster data
analysis ArcPy
ArcGIS API for JavaScript
…
Custom Publish toolboxes or Python to ArcGIS
web tools Enterprise as geoprocessing services
Custom analysis
ArcGIS Your python code run in an isolated
Notebooks environment
Getting to Know Web GIS, fourth edition
 Compare ArcGIS Enterprise and ArcGIS Online analysis

ArcGIS Enterprise
ArcGIS Online
analysis tools
Ready-to-use tools Custom web tools
Created and hosted by Esri. Comes with software install. Hosted on your To be created and hosted by organizations
Source of tools Ready to use infrastructure. Ready to use

Low. Easy to use Relatively high. Requires knowledge of

Difficulty level ModelBuilder, Python, or other scripts to
create services

Both non-professionals and professionals Professionals who provide web tools or

Target users services

Yes (you can use your own No (you can use the data from ArcGIS Online)
Data provided? data too)

Yes No, unless you are referencing ArcGIS Online No

Cost credits tools and premium content
Costs organizations to purchase, install and
host
Publishers, administrators, and custom accounts with certain privileges Administrators, and publishers (if enabled to
System requirements publish geoprocessing tools)

Getting to Know Web GIS, fourth edition

Standard analysis tools provided by ArcGIS Online and ArcGIS Enterprise

Getting to Know Web GIS, fourth edition

Living Atlas of the World supports ArcGIS Online analysis

Getting to Know Web GIS, fourth edition

Workflow to use ArcGIS Online analysis

Review and
Prepare data Add to map Perform analysis
interpret results

The first two steps are optional if you use only the data provided by ArcGIS Online
Getting to Know Web GIS, fourth edition
 Big data

• Datasets that are so large and complex that

traditional data-processing application
software is inadequate to deal with them

Large scale Different forms

of data of data
Volume Variety

Big
data
Velocity Veracity

High speed Uncertainty

of data flow of data http://www.washingtonpost.com/wp-dyn/content/article/2011/02/10
/AR2011021004916.html
Getting to Know Web GIS, fourth edition
Geospatial big data

• Smart sensors
- Electrical meters (AMI), SCADA, UAVs
• GPS telemetry
- Vehicle tracking, smartphone data collectors, workforce tracking, geofencing
• Internet data
- Social media streams, web log files, customer sentiment
• Sensor data
- Weather sensors, stream gauge measurements, heavy equipment monitors
• Imagery
- Satellites, frame cameras, drones

Getting to Know Web GIS, fourth edition

ArcGIS GeoAnalytics Server: For big data (vector) analysis

Too big to fit in one computer and to be processed by one

computer

Big
data ArcGIS GeoAnalytics Server leverages distributed analytics and storage

Getting to Know Web GIS, fourth edition

Big data analysis tools provided by ArcGIS Enterprise

Getting to Know Web GIS, fourth edition

Use GeoAnalytics Server to analyze big data

Register and edit

Prepare data Perform analysis Review results
big data file share

Getting to Know Web GIS, fourth edition

Perform big data analysis

For browsing and using

custom web tools

For standard analysis For big data analysis

Getting to Know Web GIS, fourth edition

New York City taxi pickup locations (170 million) aggregation

1 3 5

Bin time series

Now
Time step

Bin

Then

2 4 6 Getting to Know Web GIS, fourth edition

Custom web tools and geoprocessing services

• Geoprocessing services expose the analytic capability of ArcGIS to the web

Toolbox GP service

Results
Publish
Tools
(Models, GP tasks
Python)

Run inside Run inside

ArcMap or ArcGIS Pro ArcGIS Server

Getting to Know Web GIS, fourth edition

Steps to create custom web tools

Author your tool

Run your tool in Share the result
using ModelBuilder
ArcGIS Pro (execution plan)
or Python

• Python & ArcPy

Execution mode
• ModelBuilder • Synchronous
• Asynchronous

Getting to Know Web GIS, fourth edition

ModelBuilder

• ModelBuilder
- Available in ArcMap and ArcGIS Pro
- A visual programming language for building GP workflows
- Can create, edit, and manage models
- Easier than Python and other scripting languages

• Model
- Created with ModelBuilder
- A diagram that chains together sequences of GP tools and connects them with inputs
and outputs

Getting to Know Web GIS, fourth edition

Python and ArcPy

• Another way to create GP tools

• Python is a free, powerful, cross-platform, and open-source scripting language

• ArcGIS extends Python by providing ArcPy, a module that facilitates GIS data analysis, data
conversion, data management, and map automation

• Python and ArcPy are included in ArcMap, ArcGIS Pro, and ArcGIS Enterprise install

Getting to Know Web GIS, fourth edition

Synchronous and asynchronous execution modes

Request
Synchronous

Server
•

Client
- Client waits for the server to finish executing and then gets the result Result
- Typically for fast services

Request

Job id
• Asynchronous Status?

- Client submits a request, and checks server for status repeatedly processing

Server
Client
- Client gets the result once the server finishes the job Status?
- Typically for services that tale longer time to run finished

Request result

Result
• .
Getting to Know Web GIS, fourth edition
Share custom web tool from ArcGIS Pro to ArcGIS Enterprise

Portal for
ArcGIS
to ol
Web

Geopr
ocessi
ng ser
ArcGIS Pro v ice ArcGIS
Server
renced
Data refe
or
Data copie
d
ArcGIS
Data Store

Getting to Know Web GIS, fourth edition

 For browsing and
Access ArcGIS web tools using custom web
tools

For standard analysis For big data analysis

• ArcGIS Map Viewer
• ArcGIS Web AppBuilder
• ArcGIS Experience Builder
• Insights
• ArcGIS Pro
• ArcGIS Notebooks
• ArcGIS API for Python
• ArcPy
• ArcGIS API for JavaScript
• … ArcGIS
Web AppBuilder
Analysis Widget

Getting to Know Web GIS, fourth edition

 Perform analysis in ArcGIS Insights

• An analytics workbench that allows you to perform iterative and exploratory data analysis

• Drag-and-drop driven
• Visualize and analyze data in maps, tables, and charts
o Bubble, histogram, scatter, time-series graph, and treemap

Spatial analytics o Bar, line, donut, stacked, and combo charts

ArcGIS Insights
Business intelligence • Analysis and visualization at the same time
• Stimulates questions and rapid iterations of analysis
• Interact with more than one map or chart at a time
• Record steps of analysis

www.esri.com/en-us/arcgis/products/insights-for-arcgis/overview
Getting to Know Web GIS, fourth edition
ArcGIS Insights

ArcGIS Insights Gallery

Getting to Know Web GIS, fourth edition
Perform analysis in ArcGIS Notebooks
• Create and share notebooks that contain live Python code and visualizations
• Provides a Jupyter notebook experience with which data scientists can combine ArcGIS spatial analysis
algorithms with open-source Python libraries to build precise spatial data science models

ArcGIS
ArcGIS Notebooks • Jupyter
• pandas
• ArcGIS API for Python Open Source • Conda
• ArcPy Python & Data Science • Keras
• Analytics • Scipy
• Geoprocessing • PyTorch
• Web GIS analysis • NumPy
• GeoAnalytics • SymPy
• Raster Analytics

Getting to Know Web GIS, fourth edition

Perform analysis in ArcGIS Notebooks

• Include hundreds of Python libraries

- ArcGIS Python API: a Python library for performing GIS visualization and analysis, spatial data management, and GIS
system administration tasks in Web GIS (https://developers.arcgis.com/python/guide/)
- ArcPy
- Common open source data science libraries
• Conveys analysis results with intuitive, interactive maps and apps for data storytelling that drives insight and
action
• Increases cross-team collaboration. You can save your notebooks in ArcGIS Online and ArcGIS Enterprise and
can share them with other users the same way you share other items. Other users can open and run the
analysis or workflow implemented in your notebook
• Accessible anywhere with a desktop or a mobile web browser

Getting to Know Web GIS, fourth edition

Artificial intelligence, machine learning, and deep learning

Artificial Intelligence Popular research areas:

• Computer vision
• Natural language processing
• Speech recognition
• Video game behavioral AI
Machine • ...
Learning

Deep
Learning
Convolutional
Neural Networks

Getting to Know Web GIS, fourth edition

Machine learning in ArcGIS

Classification Prediction
Deciding to which category Using the known to
an object should be assigned estimate the unknown

Integration with
Clustering
Grouping observations based on ArcGIS external frameworks
similarities of values or locations

Getting to Know Web GIS, fourth edition

Richness of Imperiled Species in the United States

Created using
ArcGIS machine
learning analysis

Getting to Know Web GIS, fourth edition

Deep learning in ArcGIS
ArcGIS Python API provides arcgis.learn model for deep learning

Case study: ArcGIS Python API provides deep-learning tools that can
effectively identify road cracks from dashcam videos
• Much cheaper than visual inspection or inspection using sophisticated
instruments
• ArcGIS Python API arcgis.learn module was trained with photos that have
various types of cracks labeled
• The trained tool then detected road cracks on images and in video feeds
• The analysis results included the locations, types, and severities of road
cracks and helped authorities to plan and prioritize timely repairs

Getting to Know Web GIS, fourth edition

Deep learning in ArcGIS
Integration with third-party libraries

Case study: Georgia Power combined ArcGIS Python API with

IBM Watson machine-learning API to automatically detect
damaged insulators and locate them on the map
• Human manual inspection would be a huge and tedious task

• George Power flew drones to take photos along the transmission lines

• Trained IBM Watson machine-learning with sample photos that had known
damaged insulators

• Used the trained tools to automatically detect damaged insulators from the
hundreds of thousands of photos

• ArcGIS Python API saved the locations of the photos to a feature layer for
visualization to guide field repairs

Getting to Know Web GIS, fourth edition

Tutorial demos
8.1-8.2: Restaurant site selection using ArcGIS Online analysis
8.3-8.5: Factory site selection by publishing and using custom web tools
8.6: New York taxicab drop-off and pick-up big data analysis using
ArcGIS GeoAnalytics Server
8.7: Crime hot spot analysis using ArcGIS Notebooks

Getting to Know Web GIS, fourth edition

Summary

• Web-based spatial analysis holds great practical value for our personal life and for enterprise
operations
• ArcGIS provides standard analysis tools, big data analysis tools, raster data analysis tools, and the
capability for users to publish custom web tools
• You can author tools using ModelBuilder and Python in ArcMap or ArcGIS Pro, run the tools, and
then publish the results (the execution plan) to ArcGIS Enterprise to create geoprocessing services
or custom web tools
• ArcGIS provides tools for machine learning, including tools for classification, clustering, predication,
and the capability to integrate other data science packages
• ArcGIS Python API arcgis.learn module can perform deep learning analysis, including computer
vision
• ArcGIS Notebooks provides an insolated environment with rich pre-installed python libraries for
users to create analysis notebooks, which can be saved, shared, and rerun
Getting to Know Web GIS, fourth edition
Assignment 7.1: Use ArcGIS Online analysis to select your new home
Select one from 7.1 and 7.2

You are in the market to look for a new home. You are considering many factors, such as the distance
between your home and work or your children’s schools, neighborhood income, median house value,
median family income, unemployment rate, and crime rate

Create an app to help you narrow the area in which you would like to buy your new home

What to submit:
- The URL of the web app you created using Web AppBuilder
- Screenshots of your analysis processes and your final choice of neighborhood areas

Getting to Know Web GIS, fourth edition

Assignment 7.2: Create a web app to clip, zip, and ship GIS data
Select one from 7.1 and 7.2

A survey and mapping bureau would like to improve its data-sharing workflow. Instead of extracting and
copying data manually to serve its customers, the bureau wants to automate the workflow with a web
app, which should allow its users to:
- Select the layers they need
- Draw the area of interest
- Select the desired data format
- Have the data clipped and zipped for them to download

• What to submit:
- The URL or REST endpoint of your web tool
- The URL to your web app

Getting to Know Web GIS, fourth edition

Discussion

Getting to Know Web GIS, fourth edition