Object Storage

Content

1 Object Storage Overview

2 Object Storage Features

3 Application Scenarios

4 Object Storage Products

2 Huawei Confidential
Storage Trend – Unstructured Data Burst Growth
500

44 ZB Users > 800 million

375
80% unstructured data DAU (Daily Active Users) > 300 million
MAU (Monthly Active Users) > 1.5 billion
Data characteristics:
250
• Massive videos need to be stored and retrieved quickly;
• The number of users, the number of videos, and capacity
125
80% cloud burst.
• Videos need to be tagged, categorized, and analyzed
• User preferences will be recorded based on history record,
and the appropriate video will be recommended to user.
• Most searched topic, Hottest Topics will be updated
20% dynamically.
2009 2010 2011 2012 2013 201
4
2015 2017 2017 2018 2019 2020 • The storage should be accessed by Mobile App or Web
 According to IDC reporter, the data will grow to 44 ZB in 2020, need to access to storage directly.
80% of which will be unstructured data.
 Unstructured data including images, video, audio, radar, Traditional NAS storage CANNOT meet the
sensor data. requirements of performance, expansion, and
 10% of data comes from device connections. retrieving.
 35% of data requires user data analysis.
ByteDance's self-developed Object Storage is
used to store the videos of TikTok.
3 Huawei Confidential
 Why Is Object Storage Born for Internet and Massive Data?

Unstructured data burst growth Limitless capability

Storage of tens of billions of objects
On-demand linear expansion of capacity
• Be able to store large amounts of data. and performance
• Should be on-demand scaling to meet the
unexpected growth.
• Unstructured data should contain the
description about the data for marking, Customized Attribute
categorizing, and analyzing. Identified by Unique ObjectID, User-
defined attribute fields
Object Object
Internet applications require new
interactive access Anytime/Anywhere
• Data will be accessed across regions, not Access Object Storage
only inside the data center. Accessed by Any Device via Internet
anytime/anywhere
• File system (NAS) is designed for Human-
computer interaction. It is difficult to program
file sharing and management in file systems.
Friendly access
REST API, HTTP, HTTPS

4 Huawei Confidential
Object Storage Is More Suitable for Massive Data than NAS
Object Storage
NAS Server

Data Access
APP APP
Shared as
Directory Access directly
Directory Requiring servers Accessible by Web IOT Mobile
Apps access the different devices
shared directory.

Protocols
NFS/CIFS/POSIX CIFS/NFS
Complex protocol
HTTP/HTTPS HTTP/HTTPS (RESTful API)
Based on URL address
Incompatible with Easy & Flexibility
different OS

Network
Private Public/private Internet
LAN Cannot be No network restrictions
accessed in
Internet

Bucket Bucket

Data Organization
Bucket
Tree Structure Flat Structure
As the size of a file or No complex file Object
directory grows, the directories
retrieval performance of Better Scalability
the file or directory
degrades significantly

NAS storage used as local file directory Object storage used as a function

5 Huawei Confidential
What Is Object and Object Storage?
Object storage is a Cloud-Orient Storage with hierarchy-free method, which is designed to
store practically limitless amount of unstructured data cost effectively.
OID

Data System
Metadata
User
Bucket Bucket Bucket
Object Metadata

Item Description
OID Object Identifier is a name used to identify an object.
• Data is stored as objects.
• Data lives in a flat address space System System metadata are attributes of objects such as
Metadata object size, access permissions, and creation time.
• (Bucket/Object).
• Easy to scale User metadata extends the information stored with
User
each object by adding application specific
Metadata
information.
Data Service data, can be text, image, video etc.

6 Huawei Confidential
An Example of Object – Object Containing Data and Metadata
Data
User Metadata
File = Image

System Metadata
System Metadata (object attribute)
 Filename: CA12345.JPG
 Created: June 2, 2009
 Last modified: July 4, 2009
File (Data)
User Metadata (user-defined attribute)
 Name: XXXXX
 Category: Chest X-ray
 Save Time: XX years
 Doctor: Dr. ABC
 Result: No abnormalities were found in the lungs, heart
shadows, and palate; a dense nodular image was seen
Object below the right palate.
7 Huawei Confidential
 History of Object Storage – CAS and Object Storage

CAS: Content Addressable Storage

CAS era
(Focus on massive unstructured data)
2006, Amazon release AWS (S3)

Object Storage era

(Improve Internet access) Huawei's first object
S3 has emerged as the de- storage product UDS
facto market standard

8 Huawei Confidential
 Implementation – Distributed Architecture Is the Better One

Original Object is designed based on distributed storage.  Original object storage

It can provide object storage services without additional
Objects do not need to be mapped to a
gateways and has a better architecture of object storage.
file system, shortening access delay.

Object Gateway Original  Concurrent access

Compared with the object gateway and
disk array architecture, its network is
Object gateway fully redundant with no single point of
Object APIs failure, enhancing the concurrent
access capability.
Object APIs
 Scale-out
File system The capacity and performance increase
linearly with the increase of nodes,
Disk array which is not restricted by the size of the
Object Storage file system.

9 Huawei Confidential
Mainstream Object Storage APIs – Amazon S3 Is De Facto Standard

 Amazon S3 is an open object  OpenStack Swift is a type of the  CDMI is a type of cloud
storage protocol of Amazon object storage APIs under the storage APIs released by
AWS, which is the de facto OpenStack cloud computing Storage Network Industry
standard for object storage. framework. Association (SNIA).

10 Huawei Confidential
RESTful API – Suitable for Object Storage

Get PUT
HTTP
POST DELETE
Client Object Storage

• Each URI represents a resource, which can be a piece of text, a picture, a

song, or a service.
• Clients operate the resource on the server side via four HTTP verbs
 GET: Get a resource
 POST: Create a new resource (or Update a resource)
 PUT: Update a resource
 DELETE: Delete a resource
REST: Representational State Transfer
URI: Uniform Resource Identifier

11 Huawei Confidential
AWS S3 – De Facto (Real) Standard Protocol of Object Storage

• Amazon S3 is an open object storage protocol Tenant

User
for Amazon AWS.
• It is the de facto standard for object storage.
Bucket Bucket
• S3 simplifies the complex directory structure of
the file system, using the Bucket/Object model.
• HTTP/HTTPS RESTful interface Object … Object Object … Object

Concept Description
Bucket A bucket is a container for object stored and does not support nesting, thus avoiding a hierarchical container structure.

Object An object consists of object data and metadata. Objects are always stored in a bucket

12 Huawei Confidential
OpenStack Swift – Similar to AWS S3
Account
 Swift is a distribute module of OpenStack, a cloud User-1
operating system. User-2
 Three-layer model: Account/Container/Object
A container is equivalent to a bucket in AWS S3.
Container Container
 HTTP/HTTPS RESTful interface

Object … Object Object … Object

Concept Description
Represents the top-level of the hierarchy. Your service provider creates your account and you own all resources in that account. The
Account account defines a namespace for containers. A container might have the same name in two different accounts.

Container Defines a namespace for objects. An object with the same name in two different containers represents two different objects.

Object Stores data content such as documents and images, also stores custom metadata with an object.

13 Huawei Confidential
 AWS S3 API Formats and Example at Glance

Request Formats Example Request

Put
Object

Get
Object

14 Huawei Confidential
How to Use Object Storage - Required Tools or Self-Developed Software

Object storage provides APIs for access, so it cannot be accessed by users directly, but needs
software or a tool, such as backup software for backup scenarios and S3 browser or self-develop
software.
Service host
Invokes the
Web service.
Self-developed APP
Bucket
software Invokes local Bucket
APIs.
Object
eSDK Object
Object

Object Object Object

User Object
Key Data

User-defined Metadata
metadata

Object storage system

15 Huawei Confidential
Operation – Connecting to Object Storage Using a Tool
Exercise: Access FusionStorage Object or access Huawei Cloud (https://www.huaweicloud.com) with S3 browser or online tools

Create a bucket in Put an object in

Huawei Cloud. Huawei Cloud.

Bucket list in Huawei Cloud:

Object list in Huawei Cloud:

16 Huawei Confidential
Content

1 Object Storage Overview

2 Object Storage Features

3 Application Scenarios

4 Object Storage Products

17 Huawei Confidential
Global Namespace – Client Access Storage Via Unified Domain Name
Object Storage supports multiple regions and Availability Zones (AZ). Storage resources in different regions need to be
virtualized into a global namespace to implement domain name resolution, location services (LSs), and load balancing.

Global
Namespace

Region Region Region

1 2 N

LAN/WAN
AZ 1 AZ 2 AZ_N

Object
Cluster 1 Cluster 2 Cluster 3 Cluster N Object
Object-based storage Object
Object
Node 1 Node 1 Node 1 Node 1 Object
resource pool Object

Node 2 Node 2 Node 2 Node 2

…… …… …… …… Guangzhou Beijing Shanghai Shenzhen
Node N Node N Node N Node N

Global namespace across regions, AZs, and clusters.

The client accesses the object storage space and resources through the unified domain name, and determines the
home region based on the bucket. Using the bucket and global domain name, the system automatically routes to the
home region and cluster, so the client does not need to record multiple endpoints.

18 Huawei Confidential
Cross-Region Replication: Supports Disaster Recovery

Cluster C

Bucket 3 Cluster B
Asynchronous data
Region 1 synchronization
Region 2

Object Storage 25/10GE

25/10GE
Bucket 1 Bucket 2
Cluster A Object Storage
Bucket 2 … …
… …
Cluster A Cluster B
Bucket 1

Object
Object Storage Storage

• In the global namespace, the system automatically • Clusters of different regions support one-to-one
or
and asynchronously copies objects in bucket 1 of many-to-one asynchronous replication,
Region1 to bucket 2 in another region based on achieving remote DR and local access.
• Cross-region transmission uses SSL encryption to ensure
policies.
data security.

19 Huawei Confidential
Erasure Code: Improves Disk Usage

Erasure code: distributed data error tolerance

technology with the highest utilization EC: Erasure code is the core encoding and error
tolerance technology of cloud storage. It replaces the
multiple-copy technology with data redundancy
verification and provides the same or even higher
data reliability for cloud storage pools with lower cost
Data fragments and more advanced technology.
Redundant
fragments
PP PP PP PP

• The disk usage is improved from 33% in three-copy

mode to 80% in EC mode, saving hardware costs and
improving disk usage by 40%.
• Based on different hardware media, data availability
can reach 99.9999%.
... • Generally, the EC is provided by mature enterprise-
grade cloud storage products.

20 Huawei Confidential
Multi-Tenant: Isolates the Data of Storage for Different Tenants
Multi-tenant is used to isolate tenants so that they always
have access to specific levels of performance and
capacity that they can change over time.
Each tenant's data is isolated and remains invisible to  Single system, providing multi-client
other tenants.
services, reducing initial investment.

 Online storage. One system is

Tenant A Tenant B Tenant C
centrally managed, and data is
logically isolated.

 Transmission encryption: HTTPS

transmission encryption and user
authentication are supported to
Object Storage ensure data transmission security.
Data center

21 Huawei Confidential
 Quota: Limits the Resources Occupied
Quota is a way to limit the resources occupied by Quota
buckets and tenants. • Bucket-level quota: Maximum capacity of a bucket. When
the bucket capacity reaches this quota, data cannot be
Object Storage written to the bucket.
• Tenant-level quota: Maximum account capacity. When the
100 TB total bucket capacity in an account reaches the configured
account quota, the account cannot be written.
• Notify users about the resources they occupied by alarm or
event.
Tenant 1
General Tenant 2 Tenant 3
manager Financial Personnel Resource statistics
20 TB 40 TB dept. 40 TB dept. • Bucket resource statistics: Bucket resources include the
size of the bucket and the number of objects in the bucket.
Users can query their own bucket resources.
10 TB • Account resource statistics: Account resources include the
account quota, number of buckets, number of objects, and
total capacity. An account can be used to query account
resources.
2 TB 10 TB 8 TB 5 TB
Notify
Employee a Employee b Employee c Employee d • Notify users about the resources they occupied by alarm or
Bucket 1 10 TB
Bucket 2 Bucket 3 Bucket 4 event.

22 Huawei Confidential
Billing Service: Supports Billing APIs by Operation or Capacity

Object storage support to pay only for you want, so billing

service is mandatory.
Billing center Generally, object storage provides open APIs such as SFTP.
Through these APIs, the upper-layer billing system can access
object storage to obtain required data for fee calculation.

 Two types of bills can be provided:

 Detailed record: Records end users' operations, such as
downloading/uploading objects, creating or deleting
API Billing APIs buckets. The record operation attribute includes the
operation type and the data capacity information so that the
Object Storage upper-layer billing system can calculate and charge the fee.
 Inbound bill: Records the resource usage of end users.
Object storage records the capacity of all the objects that
have been uploaded by users and saves them to the
inbound bill file.

23 Huawei Confidential
Content

1 Object Storage Overview

2 Object Storage Features

3 Application Scenarios

4 Object Storage Products

24 Huawei Confidential
Application Scenario of Object Storage – Massive Resource Pool

Cloud Cloud Advantages

Cloud
Mail Video backup disk archiving
Seamless capacity expansion: Unlimited scalability on
demand. Performance linearly increases with capacity.
Video Storage Data reliability: Erasure code algorithm is used to
Music Picture TV Surveillance leasing
provide high reliability of redundant protection, data
transmission encryption, and E2E data security.

Cross-site access: The data can be accessed directly

Multi-tenant from offsite.

Efficient: Machine-to-machine data management and

storage, supporting artificial intelligence and analytics,
and reducing the cost and time of the design process.
Object Storage

25 Huawei Confidential
Application Scenario of Object Storage – Backup and Achieving

Online data
Office environment Advantages

Backup
Production and
Backup Massive storage space with backup software,
and
environment archiving
archiving composed of massive data backup solutions,
data
data Supports backup of multiple types of applications
and data.
Application 1 Backup media server

Allocate backup resources on-demand: Avoids

space waste. The performance is improved in line
with capacity expansion, preventing performance
Online data Office environment Object Storage bottlenecks caused by increasing data.

Reliability: The system adopts cluster deployment,

Backup
and Backup
multiple data protection technologies including EC,
Production
environment archiving and and automatic fault detection and recovery
data archiving
data
technologies, enhancing system data reliability.

Application 2 Backup media server

26 Huawei Confidential
Content

1 Object Storage Overview

2 Object Storage Features

3 Application Scenario

4 Object Storage Products

27 Huawei Confidential
Huawei FusionStorage – Fully Distributed Storage System,
Supporting Object, Block, File, and HDFS

Traditional
New Apps
Apps
Reliable Elastic Open

OpenStack

iSCSI SCSI NFS CIFS FTP HDFS NDMP S3/Swift

 On-demand block, file, HDFS

Block storage Object storage and object storage services
File storage service
service service
 Fully distributed architecture
FusionStorage  Based on Huawei's general-
purpose hardware platform
General-purpose hardware platform

28 Huawei Confidential
Thank you. Bring digital to every person, home, and
organization for a fully connected,
intelligent world.

Copyright © 2020 Huawei Technologies Co., Ltd.

All Rights Reserved.

The information in this document may contain predictive

statements including, without limitation, statements regarding
the future financial and operating results, future product
portfolio, new technology, etc. There are a number of factors that
could cause actual results and developments to differ materially
from those expressed or implied in the predictive statements.
Therefore, such information is provided for reference purpose
only and constitutes neither an offer nor an acceptance. Huawei
may change the information at any time without notice.