STG214

Data migration and edge computing

with the AWS Snow family
Ramesh Kumar Matt Johannessen
Principal Product Manager Director, Cloud Services
Amazon Web Services Novetta

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
Responding to customer needs for migration and edge computing

Edge computing with AWS Snowball Edge

Guest speaker: Using AWS Snowball Edge for disaster response

Moving data at scale with the AWS Snow family

Q&A
 Edge

Migration
AWS capabilities
Continuum
Scale
across a continuum of deployments
of capabilities

Fully managed, fully elastic,

fully featured cloud scale

Capabilities
The AWS Snow family evolution

Data movement Edge computing

Edge sites Edge sites

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why computing at the edge?

Content Maritime and Healthcare

distribution tactical edge IoT
computing

Industrial Transportation and Autonomous vehicles and

IoT logistics fleets drone data collection
Compute at the edge
For non-traditional, rugged, For fully connected, data
austere, mobile environments center rack environments

AWS Snowball Edge AWS Outposts

Compute and high volume Cloud-based compute,
data ingestion at the edge storage, and database for
for disconnected scenarios connected scenarios
2 Snowball Edge options

Compute optimized Storage optimized

• 42 TB usable S3 compatible storage
• 52 vCPUs, 208 GiB of memory
• Optional NVIDIA Tesla V100 GPU
• sbe1, sbe-c, and sbe-g instances
(equivalent to C5, M5a, G3, P3)
• Cluster-able with compute optimized
• 80 TB usable S3-compatible storage
• sbe1, sbe-c instances
(equivalent to C5)
• Up to 24 vCPUs, 32 GiB of memory
• Cluster-able with storage optimized

Long-term deployment options: 1- and 3-year discounted pricing

Edge computing example workflow
Air-gapped or intermittent connection for
AWS
periodic transfers or updates

Edge computing environment

Hardened, secure, and highly performant computing outside of the data center
suitable for hot factories, facilities with variable security

Benefits Collect, analyze, and enable local actions faster

Consistent programming and DevOps model: Develop & test in-cloud, run at the edge
AWS delivery model: Hardware-as-a-service
Snowball Edge device manager GUI
Simplified Device Unlock
• Graphical UI to manage multiple
Snowball Edge devices
• Fast and easy device unlock
Snowball Edge Device Dashboard View
• Drag and drop file transfer
• Reboot or update devices
• Monitor resources: memory,
Storage, job metrics, IP addresses
• Manage EC2 instances: create,
start, stop and reboot
• Manage virtual network interfaces

• Expected availability by Q1’20

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Changing the way the intelligence
community moves data
12 million images 50 million indexed observations
Reduced timelines 75% automated
Automated extractions Automate imagery observations
Automate text reports related
to those observations

Approved for public release, 19-132

Excerpted from AWS re:Invent session: National Geospatial-Intelligence Agency: Changing the Way the Intelligence Community Moves Data – WPS315, led by
Timmy Richardson, Nov 2018 https://youtu.be/KXelfBpJtDY
Odyssey
Ingest and dissemination

• Delivers cloud architecture at the edge

• Built on AWS Snowball Edge
(storage, compute, and hosts virtual machines)
• Provides constant tech refresh AWS Snowball Edge
• Leverages existing enterprise licenses 100 TB storage each
Clusters up to 1.4 PB

Approved for public release, 19-132

Excerpted from AWS re:Invent session: National Geospatial-Intelligence Agency: Changing the Way the Intelligence Community Moves Data – WPS315, led by
Timmy Richardson, Nov 2018 https://youtu.be/KXelfBpJtDY
AWS Disaster Response Team and Snowball Edge

• AWS technical professionals will

provision, transport, implement, and
redeploy AWS Snowball Edge devices
for customers deployed to disaster zones
• Critical data will be provisioned with
predefined software stacks to allow the
customer to run workloads on the edge
where connectivity is limited/non-existent
• AWS DRT conducts field simulation exercises
with customers to ensure the technology
works when its needed most
Edge computing in the Bahamas: Help.NGO
Global Disaster Immediate Response Team (D.I.R.T.)
Global D.I.R.T. image collection and processing
architecture in Bahamas
AWS Snowball Edge

Amazon S3 and Amazon EBS

compatible storage, and EC2
AMIs

Open-source maps

Drones (UAS)
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
About me

Operation Convergent Response

Solution architecture

IoT, Ageon, and machine learning

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
About me

• Physics/Engineering background
• Alliance lead for Novetta
• Builder & passionate innovator
with a prototyping habit
• Father of two
• Enjoy anything to do with being
outdoors
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Every year, Verizon hosts Operation
Convergent Response at Guardian
Centers in Perry, GA.

OCR gives first responders, disaster

management, and law enforcement
groups an opportunity to test new
technologies in the field.

AWS invited Novetta to participate to

solve a hard problem, keeping track of
first responders and critical equipment.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Solution architecture
We used the Snowball Edge to collect and visualize data from inexpensive
IoT sensors and traditional security cameras
Solution architecture

AWS IoT Greengrass IoT camera

Internet
AWS Cloud

VPC
Local users
EC2

Data is sent to a mature sensor

fusion platform running on a
powerful Windows Server 2016
Lora GPS instance
Sensors
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
IoT sensors
We used LoRaWAN GPS sensors that run on AAA batteries and can
transmit over long distances reliably
Ageon
Disconnected
Connected environment

• We used the Medium

Earth Orbit backhaul
connection to stream data
to Ageon in Amazon EC2
• Ageon provided Common
Operating picture to the
Joint Operations Center
Machine learning
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data collection and movement challenges

Limited online data

transfer options due to
Time to transfer
@100Mbps @1Gbps @10Gbps
• Connectivity limitations
• Bandwidth constraints
10 TB 12 days 30 hours 3 hours
• Restrictions on protocol support
100 TB 124 days 12 days 30 hours • Legacy environments
1 PB 3 years 124 days 12 days • Data properties and usage patterns
• Data collected in remote/
austere locations
AWS Snow family for data collection and movement

Snowball Snowball Edge Snowmobile

Migration Up to
Up to exabytes offline
size petabytes, offline

Form Rugged 8.5 G impact cases that are rain and dust 45-foot container,
factor resistant, e-ink label for shipping automation scheduled delivery

Encryption, Encryption, security staff, GPS tracking,

Security
tamper detection video surveillance, alarms, etc.

Capacity 42 TB or 72TB usable 80 TB usable <100 PB

Amazon EC2 or AWS IoT GreenGrass

Compute processing to use applications or
functions to load or pre-process data
AWS Snowball Edge import workflow

1–3 days 1–2 days 1–N days 1–2 days 1–3 days
(file transfer
speed dependent)
Large-scale migrations with Snowball Edge
Learned from multi-petabyte customer migrations
Run a proof
of concept (POC)
• Early discovery and remediation of
environmental issues
• Sets more realistic migration and edge
compute timelines
• Deploy staging workstations
• Ensure low network latencies (<1ms)
• Ensure larger files (>5MB)
• Benchmark & optimize data transfer
(target 300-500 MBps)

Plan devices and scheduling with your

account team/TAM before ordering jobs
Resources
• White paper: AWS Snowball Edge data migration guide
• Blog: Data migration best practices with Snowball Edge
ISV tools for Snowball Edge: SBManager
GUI for Snowball Edge management:

• Automate unlock, start up, stop,

reboot, & configuration
management
• One-click bulk or single
management control over
multiple devices
• Monitoring, log access, &
software version management
• Cluster management
• Emulate all available AWS Boto
commands

By Quantum Edge Services

Booth #1217 in the expo

quantumedgeservices.com
ISV tools for Snowball Edge: Pixspan PixMover

www.pixspan.com

Speed up offline migrations to Amazon S3 with Snowball

Edge
Simplified GUI avoids CLIs and scripting
Can accelerate time to load Snowball Edge by up to 2-3x,
depending on file size
Enables easy simultaneous loading of multiple devices
Data migration at scale with Snowball Edge
Europe’s #1 online photo service

Migrated to AWS from two data centers

• Used Snowball Edge to move 10 PB (5.7 billion) of photos from
Dell EMC Isilon and IBM Cleversafe to Amazon S3
• Needed S3 data durability; higher than colos and other clouds

Shifted investments and focus to innovation and product

development for customers, away from IT infrastructure

“We’ve reduced costs. We’ve improved our customer experience.

Generally, we’ve made our website faster. And that’s because AWS
manage that infrastructure in a way we could never do internally.”
—Chris Astall, Group Director of Architecture
aws.amazon.com/solutions/case-studies/photobox/
Next steps: Learn more & try it out
More sessions later today and tomorrow
STG353 – Chalk Talk – Edge computing, IoT & machine learning in disaster response
STG310-R1 – Chalk Talk – Edge computing use cases: Deep dive on AWS Snowball Edge
STG352 – Chalk Talk – Edge computing in disaster response with Snowball Edge
STG316 – Workshop – Get hands-on & learn best practices for AWS data migrations
STG219 – Chalk Talk – AWS Snow family: Why we built it, how to migrate with it
STG354 – Chalk Talk – Large-scale file migrations with AWS DataSync

Find on YouTube
STG204 – Get your data to AWS: How to choose and use data migration services
STG213 – Storage for hybrid cloud and edge computing: Bring AWS to you
STG305-R1 – Build hybrid storage architectures with AWS Storage Gateway
Thank you!

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.