Cloud Storage Infrastructures

Assignment 2

Company XYZ
Company XYZ is a global software solutions company specializing in the creation of
e-commerce solutions for Business to Customer transactions. Company XYZ has a requirement
to design a storage solution which will be used primarily for handling the VDI workload for their
1000 employees. This solution will be using VMware Horizon View 7. Company XYZ has 3
primary groups of users which are categorized as Task Workers, Office Workers and
Developers. The first set of users named Task Workers are mostly made up of support staff
which respond to customer queries. The second named Office workers are primarily made up of
back office workers from departments such as Finance and Operations. The third set of workers
are the developers who are responsible for the development and maintenance of the software.

Part1

Prequalification Interview Questions and Answers Showing Assumptions

How many users will be using VDI?
• Task worker: 350
• Office worker: 400
• Developers: 250
Will there be concurrent users
Yes there are users from all groups that will be accessing the VDI across the APAC, EMEA and
NA regions
• Task worker: 120
• Office worker: 140
• Developers: 90
What type of applications are used
• Task workers: Email, Chrome browser, FS, AD and Salesforce
• Office worker: The office worker will be using Office 2019 as the majority of their work will be
email and Office 2019 applications. They also have access to Salesforce, FS and AD are also a
requirement
• Developers: will be using Chrome browser to access internal Intranet, AD, FS. They also
required access to developer tools such as GitHub, visual studio
What type of applications do developers require?
• The developer will be using a web browser to access internal Intranet, AD, FS. They also
required access to developer tools such as GitHub, visual studio
What storage space will office workers and support staff require?
 • Task worker: They will need 5GB of non-persistent storage
• Office worker: They will need 10GB of non-persistent storage
• Developers: They will need 50GB of persistent storage
How much compute will knowledge workers require?
• Task worker: They will need 2GB RAM, 2 vCPU and will be running Windows 10
• Office worker: They will need 4GB RAM, 2 vCPU and will be running on windows 10
• Developers: They will need 8GB RAM, 4 vCPU and will be running on windows 10
How will the they be accessing the VDI
• For now we just want a solution that allows users to connect internally using PCoIP
Regarding the storage array, is there a preference for a particular brand?
• Yes Dell as the engineers already have vast experience with this brand and would prefer to
stick with this vendor to avoid a new learning curve.
Is there currently a disaster recovery solution and/or backup solution?
• Task worker 5GB personal storage and Office workers 10GB is out of scope as this is
backed up on local NAS. In relation to the developers this 50GB of personal should be backed
up by the in guest OS.
Have you used an assessment tool to find out the IOPS of the VM’s, what were the
results?
Yes we used a tool to assess the IOPS, results are as follows
• Task worker: Avg. of 15 IOPS with Peak of 20 IOPS (80W/20R) (4k W/32k R)
• Office worker: Avg. of 25 IOPS with peak of 40 IOPS (80W/20R) (4k W/32k R)
• Developers: Avg. of 50 IOPS with peak of 80 IOPS (80W/20R) (4k W/32k R)
What is the potential growth over the next 4 years?
We expect a 10% growth over the next 3 year for all user groups

Part2

Expected I/O Patterns and the impact when using Horizon on the I/O Load

Based on the interviews held we can predict the max number of IOPS for the virtual desktop.
There are also several other factors that will impact the performance and increase the IOPS
such as spikes when users load applications. We know from the interviews that there are 3
different time zones so be boot storms, shutdown storms and logon storms so we should expect
high IOPS during this time. This can be mitigated by performing proactive boots where the
desktops are started in batches before a new shift starts. [1]

Boot storms can impact the overall VDI performance when there are a lot of VDI’s starting at the
same time when the datastores allocated to the desktop pool are not able to handle the IO load,
when a flood of DHCP lease requests come at the same time. The logon storm is caused when
there are a lot of users logging on at the same time such as shift change when all the profiles
are created, if the performance of AD is slow which decreases the overall VDI performance.
Another measure to help prevent boot storms for linked clones is disable provisioning on the
desktops by setting the refresh OS disk on logoff to never so that if there is an
maintenance/network outage this will prevent boot storms when services come back online. [2]
View Storage Accelerator can be used to improve performance reducing the need for additional
storage IO bandwidth to help manage both anti-virus scanning IO and boot storms by cashing
common VM disk data on the ESXi host, this will lower the demand on the storage array. This
feature should be enabled on the vCenter server and can be managed when creating the
desktop pools. [3]
For non-persistent desktops antivirus updates should be run on the master image so that it will
provide a clean image for the linked clones prior to deploying to production. The master image
will need to be updated with antivirus patching to ensure that they are fully up to date and uses
get a clean image. Disable auto updating of antivirus as any updates will be lost anyway when
the user logs off. [4]

Support Staff and Office workers

Virtual Machine Specs for the office worker and task works specifications are taken from the
initial interviews. They will be using non persistent desktops which means that they might be
connected to different desktops from the pool each time they connect. Applications and user
data does not persist between sessions as the desktop will be reset each time the user logs off.
This is the best model as you don’t need to build all of the desktop models upfront for each and
every user. The desktop will get personalized before delivery. This helps with concurrency rates
so for this solution means that although we have x amount of users at one time. Non persistent
is more efficient as you will not need to build a virtual desktop for every user and is the best
option as less infrastructure is required and also easier to manage.
Instant clones will be used for the office worker and the task worker as it will use less storage as
it will not create a full copy of the parent virtual desktop, it will be linked back to the parent
image in order to operate. This will allow for easier updating of the OS and the master image.
When a linked clone is deployed then it’s the same as the parent image. A master virtual
desktop image will be created which will have the OS and core applications installed, and the
horizon view agent component so it can talk to the connection server. This is typically called the
Gold image. A snapshot is taken of the parent image which is used to create a replica, this
replica will need to be placed on a high performance data store and set to ready only. When a
linked clone is created a unique copy of the replica will be created for each user. Through
persona management user settings and association to local data storage is preserved so each
time a new linked clone session is created nothing will be lost.

Developers
Virtual Machine Specs for the developers are taken from the initial interviews. They will be using
persistent virtual desktops which retain all of their applications, documents and settings between
each session. Once the user connects for the first time then this virtual desktop will be assigned
each time the user connects. Access is only permitted to the specific user and cannot be
accessed by another user. Persistent desktops are typically used for developers who may need
to install applications on their desktops or for compliance purposes. User profile management is
more difficult as the same user experience is required every time the user logs on. [5]
Full clones will be used for the developers where an exact full size copy of the parent virtual
desktop with no links back to the parent image.

Proposal 1: Traditional Storage Array

High Level Design

Here an internal user connects through their end point device using PCoIP, first through the
connection server, then to the connection server which connects that user to their virtual
desktop which is running on the hosted infrastructure.

View Composer works with VCenter by taking the parent image by creating a linked clone/full
clone virtual desktop machine. They will be presented with the applications that are part of the
replica. The linked clones and full clones are utilizing the storage from the EMC VNX7600
Storage array. The storage is external to the ESXi hosts
Pool 1 – Task Workers
I took the below input from the interviews and used the VDI Calculator by Andre Leibovici to get
the total requirements for the task workers, Office workers and Developers. Boot IOPS were left
at 600 which is what was used by the calculator.
POOL 2 Office Workers
POOL 3 Developers
Total storage calculation is shown below

Traditional Storage Sizing

Storage Sizing: ​When​ ​using the sizing tool​ I used the s​izing with a guide function. I entered
41TB of storage, IOPS of 14000 and selected 2 Tier Storage. I selected the VNX7600 model.

The calculation give me a total of 8 * 400 SSD’s and 104 * 900 SAS 10K both will be Raid 10.
The VNX model uses only 16.5% based on the current workload.

Looking at the IOPS below, there is sufficient IOPS on the SSD layer to accommodate the
linked clones and desktop clones. The SSD Layer will provide sufficient IOPS for the master
images and the Replicas, much higher IOPS will be required at this layer to load the VM’s
Growth of 10%

The VNX 7600 is showing that its using 16.5% based on the current required workload so I think
that there should be enough but I may need to add more SAS Drives. To check this I have done
the calculations.

Task workers growth 10%

Office workers growth 10%

Developer’s growth 10%

The VNX Calculator recommends to add an additional 24 SAS drives to accommodate the
growth of 10%
Storage Tiers VNX7600

VNX7600 will provide fully automated storage tiering for data storage by segregating disk drives
into different categories called storage tiers. Typically when data is first created the data is used
more and over time that data is accessed less often. FAST VP is fully automated storage tiering
for virtual pools is a solution used to dynamically manage storage as data as its used frequently
to less frequently through the data access lifecycle.

For this solution I will not be using FAST VP and storage tiering will not be automated for virtual
pools.

The VNX7600 solution chosen for this project is using two tiers.
Tier 1 - Extreme Performance Tier​: This tier will be used when both performance and
response times are critical. This solution will be using 16 400GB SSD which have no moving
parts so no rotational latency leading to increased performance for critical workloads. This tier is
suitable for high performance resources such as the replicas for the VM’s

Tier 2 - Performance Tier​: This tier will be using 104 900GB SAS 10 disks to provide a
combination of performance and capacity. This tier is suitable for storing the linked clones and
full desktops for this solution

[6]

RAID Level and Data Protection

RAID 10 has been chosen due to the level of protection and performance that this level can
provide. With RAID 10 there will be a write penalty of 2 and will have better read and write
performances when compared to RAID 5 and RAID 6. RAID 10 is the best option here for
mission critical services as it will offer greater performance and redundancy. It does not use
parity so rebuilding of data is faster, if a disk fails then data can be rebuilt quickly by copying
data from the remaining disks to the new disk. ​[7]

Raid 10 provides data protection by copying data on one disk to another disk so data is fully
protected. For raid 10 to work we will basically need double the disks, for this solution we will
have 16 SSD @ 400GB – 8 of these will be used to mirror data. There are 104 SAS @ 900GB
and 52 of these will be used to mirror data.
LUN’s
VSphere uses virtual volumes vVols so there is no need to create the Lun’s at the storage level,
the lun’s can be created on vSphere. This simplifies the delivery of storage as you can quickly
provision lun’s and associate it with VMFS datastores.

The tool used recommended to create the following datastores. The first thing to do here is to
create a lun’s for each of the datastores, then create a VMFS datastore of the same size.

Pool 1 Task workers 6 VMFS datastores @ 636 GB

Pool 2 Office Workers 7 VMFS datastores @ 857GB

Pool 3 Developers 3 VMFS Datastores @ 10.7

Replica for the task workers and office workers each of 700GB total 1400GB so I will create a
LUN of 5GB and create a VMFS datastore and allocate 5TB to allow for further growth.

Clusters
AS you can see from the image below there are a total of 4 clusters. Each cluster is routing to
its VNX storage and the Management cluster is connected to each of the clusters.

Cluster 1:​ This cluster is for the task workers with 6 ESXi hosts, 6 datastores and 6 LUN’s. The
LUN’s in the cluster are routing to the SAS disks in the VNX 7600

Cluster 2:​ This cluster is for the office workers with 7 ESXi hosts, 7 datastores and 7 LUN’s.
The LUN’s in the cluster are routing to the SAS disks in the VNX 7600

Cluster 3:​ This cluster is for the developers with 4 ESXi hosts, 3 datastores and 3 LUN’s. The
LUN’s in the cluster are routing to the SAS disks in the VNX 7600

Cluster 4:​ This cluster is the vSphere management cluster, I have also stored my replicas here
and it has direct access to the SSD on the VNX 7600 storage array. It is through the
management cluster that the linked clones and full clones get access to the services such as FS
and domain controller.
VM mapping to LUN
The below diagram shows ESXi hosts that provide CPU and Memory to the VM’s, the storage
will come from the storage array chosen earlier. A vSCSI is assigned to the VM operating
system so that it can communicate with the storage so that the VM can read and write to the
storage. When the VM issues a read/write request it uses SCSI commands through the vSCSI
then hits the ESXi hypervisor layer which sends the command to the storage adaptor HBA and
across the storage network to a dedicated physical storage with luns, then to the datastore that
was created on that LUN. If the VM needs to be migrated to ESXi host 2 for some reason then it
will still be able to access the datastore and any VM files required to run such as VMDK, VMX,
VSWP​ [8]
Storage Area Network

As discussed during the interview stage the in house engineers want to use FC to connect to
the Dell VNX7600.The Dell VNX7600 documentation specifics that a 4 port 8/16 GB Fiber
Channel Module is compatible. As outlined below, each ESX1 host will have at least 2 HBA
ports, HBA 1 and HBA2 and is used to connect to the fabric switch 1 and fabric switch 2. The
fabric switch will then connect to SPA and SPB on the VNX7600.

This design will allow for redundancy and load balancing as both SPA and SPB are directly
connected to each fabric switch. ​[9]
Proposal 2 Hyper Converged
Infrastructure – VMware vSAN

High Level Design Overview Solution 2

Here an internal user connects through their end point device using PCoIP, first through the
connection server, then to the connection server which connects that user to their virtual
desktop which is running on the hosted infrastructure.

View Composer works with VCenter by taking the parent image by creating a linked clone/full
clone virtual desktop machine. They will be presented with the applications that are part of the
replica. The linked clones and full clones are utilizing the storage from the vSAN datastore
which basically takes its storage from disk groups which are part of the ESXi hosts.

VMware vSAN is a hyper converged solution designed for the use with either hybrid or all flash.
vSAN is basically software which will allow you to manage both the compute and storage from a
single platform. This is achieved by joining all of the storage devices into a shared data pool
called vSAN datastore. The main advantage is that the design can be easily scaled up or scaled
down to suit business requirements at much lower costs when compared to traditional storage
and server designs.
Disk Groups:​ Disk groups is a unit of compute in a vSAN cluster, this group is the collection of
disks that exist on the host. There are two levels of architecture for the disk groups. The first
level is the cache which is a single SSD where the SSD is the cache. The second level is the
capacity tier, which can be either SSD or HDD. Now, you can use either one of them, but for
performance reasons it’s probably better to go with an SSD as it will give you the best
performance. This second level is the disk capacity which can have up to 7 SSDs or HDD. A
single host can have up to 5 disk groups so a host and can have a maximum of 40 disks
providing storage to the vSAN datastore. ​[10]

When a write comes into a disk group it first goes to the cache, the cache acknowledges the
write taking advantage of the write performance of the cache speed all of the time. When a read
comes in if the block that you are reading is not in the cache then the host will look for that block
in the capacity. ​[10]

The main design choice with vSAN is whether to use all SSDs or HDDs but in a VDI
environment deduplication is one of the main features and it will only work with all flash so this is
the solution I will be proposing for Company XYZ.

This design will provide high performance when compared with the hybrid model for all use
cases. The cost of SSD is also declining and VMware are driving the use of all flash
configuration by providing new features which are not available in the hybrid configuration such
as de-duplication, compression, Raid 5 which retain the redundancy and availability of data but
without using extra space which can reduce the cost of an all flash storage array. ​[11]
VSAN Storage Policies
VSAN uses storage policies to store objects,

●
Witness:​ This is used to ensure that both disk groups are in sync. If an ESXi host
fails the witness will be used to express the VDMK that is the one to be used.
Raid 5 and Raid 6 do not need the witness component as there is already be the data
component and the parity component, used for RAID 1

●
FFT: ​vSAN will create different mirrors of the data to prevent loss in the event of a
failure. N+1

● Erasure encoding​: vSAN uses this technique where the data will be broken down
into fragments allowing fragments of data to be recovered

● FFT1 without erasure coding: vSAN will create 2 mirrors plus a witness

● FFT1 with erasure coding vSAN is basically RAID 5 which requires 4 hosts in the
configuration, 3 disks for the data and the third is required for parity.

● Object Space Reservation: ​An object is a logical volume which contains data and
metadata associated with each VM such as Swap object which is created with a
VM is powered on, VM home namespace, VMDKs and snapshot delta-disk
objects. The objects are thinly provisioned by default but for developers this will be
at 100% because they are full clones. 100% is basically thick provisioned

● Stripe Width: ​This is the number of disk stripes per object, defines the min number
of capacity devices which each replica of the storage object is distributed.
Default value is 1 ​[12] [13]
vSAN Storage Policies. ​The following storage policies will be created

FTT1-FULL​: Should include FTT1, Stripe Width 1 (SW1) and Object space reservation as 100%

FTT1-REP-MASTER​: Should include FTT1 and SW1

FTT1EC-FLOAT​: Should include FTT1 with erasure coding.

Task workers storage policy​: FTT1 with Erasure encoding SW1

Office workers storage policy​: FTT1 with Erasure encoding SW1

Developer’s storage policy: ​FTT1 without erasure encoding Stripe Width 1 (SW1) and Object space
reservation as 100%

Replicas and master images: ​FTT1

I will need to ensure when doing the sizing that I have a minimum of 4 ESXi hosts in each cluster to
accommodate Raid 5. On vSAN Raid 5 works in the same way as with traditional storage using parity and
data bits. With PFTT=1 I will be able to tolerate the failure of 1 ESXi host.

Raid 5 with erasure encoding will protect the data against loss and also increases the efficiency. Using
both will provide the same level of data protection as when using Raid 1 but it will use less storage
capacity, the drawback to this is that will incur an additional overhead cost. On vSAN Raid 5 works
differently than with traditional storage arrays. During the process of creating a block and is written to
vSAN it is divided up into components which along with the parity. This consumes more compute and
also incurs a write latency as the object will need to be distributed across the all hosts on the vSAN
cluster. To have a uniform distribution of the blocks all the block data will need to be first verified and
then rewritten with each new write for the failure toleration and for the rebuild process.​[7]
Deduplication and Compression:​ To save storage space vSAN performs block level
deduplication and compression by reducing redundant data on each disk group. Deduplication
is applied on the disk group level reducing redundant data to a single copy. ​[8]

Duplication on vSAN the data blocks are kept on the cache tier while its active for increases
performance and when it’s not active then its moved to the capacity tier, deduplication and
compression then happens by using the SHA-1 hash algorithm which will create a hash which is
unique to each block so that when a new block arrives from the cache tier to the capacity tier
the hash is compared to a table of the hashes. If the hash is already in the table then there is no
need to store the new block and a new reference is noted. If the hash is not in the hash table
then the block is persisted to the capacity tier and the unique hash added to the hash table. ​[9]

Compression happens when a new block is not in the hash table then vSAN applies LZ4
compression on 4K blocks and tries to compress to blocks of 2K in size which is then persisted
to the capacity tier storage. If the block cannot be compressed to a block of 2K or less then the
full size block is persisted to the capacity tier. ​[9]
Design and sizing of vSAN
For the vSAN sizing I used the vSAN online tool by VMware. I took the below sizing which is the
same inputs as for Solution 1 Traditional Storage

POOL 1 Task Worker Linked Clone

POOL 2 Office Worker Linked Clone
POD 3 Developers Full Clone
Cluster Decision
Initially I considered putting all the clones in one cluster as thought it might be easier from a design
perspective. Then I decided to separate out into separate clusters so that each pool of VDI’s can be
managed separately. I could not really find a definite answer on which design choice is better. As this is a
design for only 1000 uses one cluster may be sufficient but if deploying a solution for say 20,000 users I
would definitely use different clusters for each pool of VDI’s. When I was creating one cluster for all
pools of VDI’s I would have used 1 less ESXi host and 3 less Disk groups for this cluster.

From the online VDI calculator I got the below results for each pool of VDI’s. The VDI calculator already
took the 10% growth for the next 4 years into account.

The Below diagram gives an overview of how the clusters are laid out. There is also a separate
cluster for vSphere Management. The replica and master images are also on the management
cluster and storage policy will be used to allocate space in the each of the disk group cluster
SSD disk
VSAN Storage Tiring

Cache Tier:​ For the cache tier we will choose an enterprise grade fast highly durable SSD due to the fact
that it will be handing a lot of write operations and it’s important that this SSD does not fail.
Performance is improved with all flash because there is no read cache so it can handle more writes, the
cache is dedicating 100% to write buffering. If a read come in and if stored on the cache then the read is
performed from the cache, if not in the cache then it’s read from the capacity tier. Since all flash SSDs
can effectively handle reads then there is no need to have a dedicated read cache which in turn frees up
space for write buffering on the cache. The cache is dedicated 100% to writes buffering up to the max of
600GB but VSAN uses the entire disk even if the size is larger than 600GB where writes are spread across
the entire disk. ​[9]

Capacity Tier​: The Capacity tier can be a cheaper high capacity SSD, this will lower the overall cost as a
lot of high capacity of SSD are required.

Initially when setting up disk groups each ESXi host must be identified and then identify which SSD is
used for cache tier and capacity tier, so as each ESXi host and its SSD and capacity SSD are identified
vSAN organizes into disk group, the disk group can then the added to the vSAN datastore increasing the
vSAN datastore in size. If more capacity is required you can add more disks to the capacity tier or you
can add more hosts to the cluster which will increase the capacity for the existing vSAN datastore. ​[9]
There will be 3 Disk groups associated with each ESXi host in the cluster. With this design if a cache
device fails then that specific disk group is offline until the cache disk is replaced. Whereas if you had a
disk group of say 1 cache SSD and 6 capacity SSD then if the cache SSD fails then the whole disk group is
offline. Point he is that it’s not always better to have more capacity disks to build up the capacity of a
disk group. The VM is not interrupted if a disk group fails as there is an additional copy of the VM on
another disk group. ​[10]

VM Mapping

On vSAN a virtual machine is broken down into a series of objects which are stored on the local
physical storage of the ESXi host, one of these objects is called the VMDK. You must setup a
vSAN kernel port on each of the ESXi hosts so that the hosts can communicate with each other,
this can also be used for other traffic such as management traffic. For example, below VM1 is
hosted on EXSi host 1, one of the objects of VM1 is a VNDK object is stored on Disk group 2.
When a read/write command is issued by VM1 it is captured and sent to the VSAN kernel port
on ESXi host 1, interpreted by the ESXi hypervisor layer and then sent over the vSAN network.
It then reaches the vSAN Kernel port on ESXi host 2 to be stored on ESXi host 2 storage. When
the cache of Disk group receives the read/write request it sends an acknowledgement back to
VM1 to acknowledge the read/write request. From the beginning of this process the VMDK file is
also written to a second Disk group, going through the same process as above. ​[12]

Networking Design for vSAN​: For the all flash storage array the minimum spec for a switch is 10GB
network card and for redundancy we will require a secondary 10GB Switch. As the 10GB is the minimum
recommended we will go with a 40GB Switch. Each EXSi host will require 2 40GB Ethernet adaptors,
named below vmnic0 and vmnic1. Each adaptor will be connected to a different physical switch. Each
ESXi host will have a VM kernel port VMK2 which will handle all vSAN traffic. If one or the virtual
network adaptors or 40GB switch fails then vSAN traffic will use the 2​nd​ vmnic or 40GB switch. [​ 12]
PhotoBox And AWS

PhotoBox had 2 Data Centers where user’s photos were uploaded to and as their business was
scaling it was taking a long time to scale their storage arrays to meet the business demands.
They used AWS Snowball Edges to move the data from their DC’s to AWS S3, in total 84TB of
usable storage per 110 snowball edges to transport all this data.
When PhotoBox was moving to AWS S3 storage they needed to develop a storage strategy,
how to choose the right name for files. They had to consider how AWS partitions data in AWS
S2 and their read and write throughput. They used the MD5 hash of the photo name to generate
a random prefix. They also had to get the design of the bucket right, whether they wanted to
group by bucket name or group by prefix.

AWS Lambda:​ Lambda allows PhotoBox to run code without having the need to provision or
manage servers on a pay as you consume basis. It allows images uploaded by users to be
instantly available in thumbnails.
This is achieved by backend code that runs in response to events such as image uploads. Once
code is uploaded to Lambda AWS takes care of the scaling, patching and administration. Once
functions are loaded the event can be located to monitor such as S3 bucket
In house this would require PhotoBox to size, provision and scale a lot of servers and manage
the maintenance, security and then monitor the infrastructure for availability and performance.
There is no upfront cost to use AWS Lambda, charges are a fee per request and for the time
your code runs in increments of 100ms. Code can be easily uploaded as a zip file or can be
designed in the integrated development environment via the management console. The console
also has pre-built function samples which are prebuilt for common functions such as image
conversion and file conversion. Other AWS sources such as S3 bucket and DynamoDB can
also be called which trigger functions when a specific event occurs. [19]
AWS DynamoDB: this is a fully managed NoSQL database which provides seamless scalability
at a low cost for structured and unstructured data when compared to operating a scalable
database in house. Tables can be created to store any amount of data and server high levels of
traffic. Photoshop are able to easily create a DynamoDB table to store the photos they store
which will enable faster searching. [20]

Route 53: ​This is a DNS web service which is scalable and highly available, it's used to connect
users requests to PhotoBox AWS infrastructure. It allows PhotoBox to manage traffic based on
criteria such as location, latency and the health of the endpoints. Route 53 works with S3
Buckets and AWS CloudFront. [21]

AWS CloudFront: Is a content delivery network, it is used to get data from S3 buckets to
distribute this data to its edge locations globally closest to the user location that requested the
data to provide low latency and fast data access. [22]

A bucket on AWS S3 is where files objects are stored, which contain a logical nest of folders
and sub-folders. Per account you can have a max of 100 buckets. There is no restriction to the
size and any size object can be stored in a bucket. It can store PDF, TEXT, Audio, video, JPEG
basically any file type. The objects are then managed by REST style HTTP and SOAP
interfaces and the objects can be downloaded by the Bit Torrent protocol or the HTTP GET
Interface which can be used by any web framework. A key is assigned to an object when the
object is created, this key is then used for object retrieval. [23]

Previously AWS recommended randomizing prefix names with the hash characters for better
performance for objects that were going to be accessed frequently but now the recommendation
is to use sequential data based naming for prefixes. AWS reiterates not to think of S3 like
traditional storage with a single network endpoint and recommends to spread requests across
multiple connections to horizontally scale performance by using multiple concurrent requests to
S3 as there are no limits to the connections you can make to a bucket. It’s also recommended
to access S3 buckets from EC2 instances within the same region where possible to keep data
transfer costs low and to reduce latency on the network. [24]

PhotoBox wanted improved security at a granular level which they did not have with their
on-prem solution. AWS user’s policies IAM (identity access management) policies and ACL’s
(Access control lists) to grant the permission such as read, write, full control to groups of users
and users individually, this allows PhotoBox to have full control over their customers.
Access Analyzer is used to monitor access policies applied to the buckets, with built in features
warning of the policies you have created on a bucket would make the bucket publicly accessible
and quickly fix any access issues. [25] [26]

AWS Macie is used to identify PhotoBox customer’s objects in buckets for classification by
streaming the objects content for analysis into memory or for deeper analysis downloads the full
copy of the object and after classification deletes the object just keeping the metadata. This
metadata is used to secure the objects from security threats by monitoring both data and user’s
accounts, if a threat is seen then it will trigger a password reset. [26]

PhotoBox customers typically after first use of the photo rarely accessed the photo again, which
led to the problem that it was difficult to spot corrupted objects. PhotoBox wanted the migration
strategy to immediately fix any inconsistencies. AWS achieves this by using an entity tag ETag
which is the hash an object which reflects changes to the object that always matches the MD5
hash (message digest algorithm) of an object. [26]

AWS S3 also uses cyclic redundancy checks and content MD5 checksums to detect any data
corruption on data at rest and uses redundant data to repair any corruption. Also data packets
are checked for corruption using checksums on data for all network traffic during the storing and
retrieval process. The integrity of the object as a whole can be validated using the MD5 hash.
Although PhotoBox said that this is not always the case, this can happen if the object was
created by part copy or multipart upload operation then MD5 will not have the Etag. [27]

PhotoBox also wanted to query the object from AWS S3 inventory and compare to what has
been corrupted at the very beginning of the process. This is achieved by utilizing S3 inventory
on objects and their corresponding metadata, the inventory can be automated to run reports
weekly or daily on buckets or a subset of objects which have a shared prefix. The inventory can
also be used to verify replication status and encryption status on objects [26]

PhotoBox wants to help customers select and organize their photos that they want to put in their
photo book and improve their customer experience, AWS Rekognition could help then to
achieve this goal by making it easier to identify objects such as people in photos using facial
analysis searching on customers photos stored on S3 buckets. [28]

PhotoBox customers typically access their photos heavily in the first 90 days so they wanted
storage classes that would help them achieve both the performance and cost targets so chose a
mix of S3 standard and Infrequent access.

S3 standard storage is used for the frequently accessed data so customer’s photos will stay
here for 90 days. Objects are then moved to S3 Infrequent access which has a lower GB
retrieval and storage price compared to S3 Standard. Both storage classes offer high
throughput and low latency with 99.9% durability across multiple availability zones. In the event
of an availability zone not being available then both classes of storage will still be available in
another availability zone. Both storage classes can be configured at both object level and for a
single bucket across all classes of storage. Meaning that you do not have to change the object
storage bucket when you move to a different storage class. [29]

AWS also offers intelligent storage classes to meet varying storage requirements such as,
S3 ZIA: For re-creatable less access data, with a retrieval fee per GB, minimum storage
duration and minimum object size. [29]
S3 Glacier: This is geared towards archive data, can select by minutes or by hours, with a
retrieval fee per GB, minimum storage duration and minimum object size.
S3 Glacier Deep Dive: This is for archive data that is accessed less frequently, with a retrieval
fee per GB, minimum storage duration and minimum object size.
Objects stored on AWS intelligent tiering can use lifecycle policies to ensure that objects are
stored in each tier for the specific amount of time you want. PhotoBox can set a policy to move
objects after 90 days to S3 IA. It has to be noted that there is a charge for each lifecycle
transition request.
Versioning control will allow PhotoBox to preserve, retrieve and restore any version of an object
that is stored in a bucket providing an extra layer of protection in case an object is accidentally
overwritten or has expired. Versioning can be enabled on buckets and managed by lifecycle
rules where you can archive to Glacier storage class or remove. It has to be noted that there is
an additional cost to this service. [29]

Advantages to AWS S3
• S3 is very good at storing static content and delivering it back to end users directly via URL
without the need to proxy via applications. Route 53 can be used for DNS web services which
are highly scalable and highly available. AWS cloudfront is used to get data from S3 buckets
and deliver that data to AWS edge locations. AWS Lambda allows customers to run code
without having to manage servers on a pay as you consume basis. AWS DynamoDB is a fully
managed NoSQL database which provides seamless scalability at a low cost for structured and
unstructured data when compared to operating a scalable database in house.

• One of the biggest benefits S3 has is that it’s fully managed and can scale to customers’
requirements eliminating the necessity for companies to worry about the underlying
infrastructure. This also allows companies to focus just on the just the accessing and uploading
of objects, then managing the lifecycle of these objects allowing S3 to take care of the storage
that the objects sits on. S3 has many different storage classes to choose from to meet varying
business needs and requirements and provides great features such as versioning control and
lifecycle management to manage content stored on S3.

• S3 provides improved security at a granular level that would not be possible for most
company’s on-prem solution by using IAM and ACCs and using the access analyzer to monitor
access to ensure that the right access is granted to an object or bucket which ensure that users
only access the data that they should be accessing. Customers can also use AWS Maice to
analyse objects to provide metadata, this metadata is then used to provide threat protection on
both the objects and users accounts and automatically triggers a password reset if a threat is
triggered. For businesses like PhotoBox this is of utmost importance as it instills trust by the
users of their solution that their data is secure and cannot be accessed by anyone else. If
someone was to access another person’s photos this would be detrimental to their business.

• S3 provides solutions to help customers find and repair corrupted objects by using cyclic
redundancy checks and content MD5 checksums to detect data corruption on data at rest and
uses redundant data to repair the corruption. Also data packets are checked during the storing
and retrieval process using checksums to find any inconsistencies. Such services would be
extremely challenging for customers when using an on-prem solution as they would have to first
figure out how this can be achieved and then find a technology solution that would work with
their on-prem infrastructure to achieve this.

• AWS provides many platforms artificial Intelligence that can be integrated with objects
stored on S3 such as AWS Rekognition which is very suitable for companies like Photobox
which analyzes video and images, AWS Lex for voice recognition, AWS personalize for
marketing, AWS Polly for automated text to voice, AWS Comprehend for language processing,
AWS Texttract for the extraction of fields and forms, AWS transcribe for speech recognition and
AWS translate for language translation. For companies who use on-prem solutions such service
would be more difficult to implement whereas with AWS these solutions are readily available to
use which allows business to be innovative and provide better service to their users and grow
their business.

Disadvantages of AWS S3

• If you use versioning, there is a charge for the storage of each of the versions stored on S3,
this will drastically increase cost especially if you are rendering images of large sizes.
 • Companies will need to ensure that they fully understand the S3 environment as cost scales
as files accumulate in buckets, need to plan ahead with lifecycle rules so that you only store
files you need and remove everything else automatically. Costs will also rise as many of the
services there is an additional charge such as using the lifecycle rules.

• Complicated storage classes with different storage classes, can be difficult to understand
which storage classes fits for every task. Storage classes also have a minimum storage
duration, meaning you only pay for what you used but if you decide to move or delete the files
before the duration expires means that you will still be paying for that period of storage even if
you are not using it. This could also mean that you only use one or two of the storage classes
which could limit the value that S3 offers.

• Minimal default visibility into the buckets content, need to create a tagging convention from
the start and enforce across all applications that write to S3.

• Unlimited scalability can also be a challenge leading to poor architectural decisions as there
is no end to the amount of storage you can use. This is useful as businesses grow but can spiral
so periodic architectural reviews on the system and also on S3 costs to ensure that the right
decisions are in line with business objectives.
References

[1]
R. Spruijt, "Storage design and sizing guidelines for VDI," Brian Madden, 11 07 2013. [Online].
Available: https://www.brianmadden.com/opinion/Storage-design-and-sizing-guidelines-for-VDI.
[Accessed 05 12 2019].
[2]
VMware, "Setting Up Virtual Desktops in Horizon 7," 07 2019. [Online]. Available:
https://docs.vmware.com/en/VMware-Horizon-7/7.9/horizon-virtual-desktops.pdf. [Accessed 12
12 2019].
[3]
VMware, "Configure Horizon Storage Accelerator for vCenter Server," 23 08 2019. [Online].
Available:
https://docs.vmware.com/en/VMware-Horizon-7/7.10/horizon-console-administration/GUID-FE0
46913-1555-4957-9EBD-B0FF86AC6525.html. [Accessed 12 12 2019].
[4]
VMware, "Antivirus Considerations in a VMware Horizon 7 Environment," [Online]. Available:
https://techzone.vmware.com/resource/antivirus-considerations-vmware-horizon-7-environment.
[Accessed 05 12 2019].
[5]
VMware, "Setting Up Virtual Desktops in Horizon," [Online]. Available:
https://docs.vmware.com/en/VMware-Horizon-7/7.1/view-71-setting-up-virtual-desktops.pdf.
[Accessed 01 12 2019].
[6]
VMware,
"file:///C:/Users/Administrator/Downloads/docu48704_White-Paper_-VNX-FAST-VP-VNX5200,-
VNX5400,-VNX5600,-VNX5800,-VNX7600,-and-VNX8000---A-Detailed-Review%20(3).pdf,"
[Online]. Available:
file:///C:/Users/Administrator/Downloads/docu48704_White-Paper_-VNX-FAST-VP-VNX5200,-V
NX5400,-VNX5600,-VNX5800,-VNX7600,-and-VNX8000---A-Detailed-Review%20(3).pdf.
[Accessed 14 12 2019].
[7]
Dell, "DELL EMC UNITY: BEST PRACTICES GUIDE," 01 2017. [Online]. Available:
https://www.dellemc.com/resources/en-us/asset/white-papers/products/storage/h15093-dell_em
c_unity-best_practices_guide.pdf. [Accessed 05 12 2019].
[8]
Trainertest.com, "vSAN vs Storage Array," [Online]. Available:
https://www.lynda.com/vSphere-tutorials/vSAN-vs-storage-array/5007870/5032072-4.html.
[Accessed 05 12 2019].
[9]
Dell, "EMC Host Connectivity Guide for VMWare ESX ... - Dell EMC," [Online]. Available:
https://www.dellemc.com/en-us/collaterals/unauth/technical-guides-support-information/products
/storage-2/docu5265.pdf. [Accessed 08 12 2019].
[10]
VMwrae, "Managing Disk Groups and Devices," 31 05 2019. [Online]. Available:
https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.virtualsan.doc/GUID-88
790522-2FA0-476F-B874-6FE4E8E5B908.html. [Accessed 15 12 2019].
[11]
VMwarearena, "Difference between Hybrid vSAN and All-Flash vSAN," 21 02 2018. [Online].
Available: http://www.vmwarearena.com/difference-between-hybrid-vsan-and-all-flash-vsan/.
[Accessed 12 12 2019].
[12]
VMware, "VMware Virtual SAN Design and Sizing Guide for Horizon Desktop Infrastructures,"
2014, 07. [Online]. Available:
https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/whitepaper/products/vsa
n/vmw-tmd-virt-san-dsn-szing-guid-horizon-view-white-paper.pdf. [Accessed 05 12 2019].
[13]
VMware, "VMware Virtual SAN Design and Sizing Guide for horizon View Virtual Desktop
Infrastructure," 07 2014. [Online]. Available:
https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/whitepaper/products/vsa
n/vmw-tmd-virt-san-dsn-szing-guid-horizon-view-white-paper.pdf. [Accessed 12 12 2019].
[14]
R. RADHAKRISHNAN, "VSAN Erasure Coding – RAID 5 And RAID 6," VMarena, 12 05 2019.
[Online]. Available: https://vmarena.com/vsan-erasure-coding-raid-5-and-raid-6/. [Accessed 17
12 2019].
[15]
VMware, "Using Deduplication and Compression," 18 04 2019. [Online]. Available:
https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.virtualsan.doc/GUID-3
D2D80CC-444E-454E-9B8B-25C3F620EFED.html. [Accessed 08 12 2019].
[16]
C. Coghan, "VSAN 6.2 Part 1 – Deduplication and Compression," 12 02 2016. [Online].
Available:
https://cormachogan.com/2016/02/12/vsan-6-2-part-1-deduplication-and-compression/.
[Accessed 11 12 2019].
[17]
VMware, "Understanding vSAN Architecture: Disk Groups," 15 04 2019. [Online]. Available:
https://blogs.vmware.com/virtualblocks/2019/04/18/vsan-disk-groups/. [Accessed 18 12 2019].
[18]
VMware, "Essential Virtual SAN," [Online]. Available:
https://book.akij.net/eBooks/2018/March/5ab76ee97f532/SAnet.cd.EssentialVirtualSAN(VSAN).
pdf. [Accessed 17 12 2019].
[19]
AWS, [Online]. Available: https://aws.amazon.com/lambda/. [Accessed 10 12 2019].
[20]
AWS, "Amazon DynamoDB FAQs," [Online]. Available:
https://aws.amazon.com/dynamodb/faqs/?sc_channel=PS&sc_campaign=acquisition_UK&sc_p
ublisher=google&sc_medium=dynamodb_b&sc_content=sitelink&sc_detail=dynamodb%20aws
&sc_category=dynamodb&sc_segment=faqs&sc_matchtype=e&sc_country=UK&s_kwcid=AL!4
422!3!620424. [Accessed 10 12 2019].
[21]
Intellipaat, "What is AWS Route 53 in Amazon?," 21 09 2019. [Online]. Available:
https://intellipaat.com/blog/what-is-aws-route53/. [Accessed 11 12 2019].
[22]
AWS, "What Is Amazon CloudFront?," [Online]. Available:
https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Introduction.html.
[Accessed 12 12 2019].
[23]
AWS, "Working with Amazon S3 Buckets," [Online]. Available:
https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html. [Accessed 11 12 2019].
[24]
AWS, "Performance Guidelines for Amazon S3," [Online]. Available: Performance Guidelines for
Amazon S3. [Accessed 12 12 2019].
[25]
AWS, "How can I secure the files in my Amazon S3 bucket?,"
https://aws.amazon.com/premiumsupport/knowledge-center/secure-s3-resources/, 17 05 219.
[Online]. Available: How can I secure the files in my Amazon S3 bucket?. [Accessed 11 12
2019].
[26]
AWS, "General S3 FAQs," [Online]. Available: https://aws.amazon.com/s3/faqs/. [Accessed 12
12 2019].
[27]
AWS, "Common Response Headers," [Online]. Available:
https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html.
[Accessed 12 12 2019].
[28]
S. Banerjee, "AWS AI/ML Services Explore Amazon Webservices suite of AI/ML services,"
Medium, 19 11 2019. [Online]. Available:
https://medium.com/explore-artificial-intelligence/aws-ai-ml-services-8510ef1064af. [Accessed
12 12 2019].
[29]
AWS, "Amazon S3 Storage Classes," [Online]. Available:
https://aws.amazon.com/s3/storage-classes/. [Accessed 11 12 2019].