You are on page 1of 94

EXPERIENCE SMART

MULTIPURPOSE SHARED
GLOBAL STORAGE WITH ECS

Experience Smart Multipurpose Shared Global Storage with ECS


Table of Contents
Introduction.................................................................................................................................................. 4
About this lab ..................................................................................................................................................5

Lab Overview ................................................................................................................................................ 6


Lab Environment .............................................................................................................................................7

Lab Credentials ................................................................................................................................................8

Lab Scenario ..................................................................................................................................................10

Labs...............................................................................................................................................................11

Lab 1 - Configuring A New Tenant In ECS ..................................................................................................... 13


Tenant Configuration .....................................................................................................................................14

Lab 2 - Ingesting Data Using An S3 Application ........................................................................................... 35


Data Ingest into the ECS Cloud Storage Platform ............................................................................................36

Lab 3 - Performing Metadata Search Using An S3 Client Application ........................................................... 40


ECS Metadata Search Media Reviewer............................................................................................................41

Lab 4 - Querying Object Metadata with Apache Spark SQL for use by HDFS based applications................... 51
Querying object metadata with Apache Spark SQL for analytics......................................................................52

Lab 5 - Accessing Data through NFS ........................................................................................................... 61


Working with ECS Data via NFS and S3 ...........................................................................................................62

Troubleshooting ......................................................................................................................................... 82
Troubleshooting & Tips ..................................................................................................................................83

Conclusion ................................................................................................................................................. 87
Conclusion.....................................................................................................................................................88

Additional Solution Information.................................................................................................................. 89


Business Challenge........................................................................................................................................90

Solution .........................................................................................................................................................91

ECS ................................................................................................................................................................92

Other Client Applications ...............................................................................................................................94


Introduction

Experience Smart Multipurpose Shared Global Storage with ECS Page 4


About this lab
Elastic Cloud Storage (ECS) is a modern, multi-purpose, shared global storage that scales into Exabytes to store both
small and large files with strong global consistency.

This lab offers first hand experience of how ECS smart storage simplifies multi-tenant data access and management
with instant meta-data search and multi-protocol access to the same data set using NFS or S3.

Experience Smart Multipurpose Shared Global Storage with ECS Page 5


Lab Overview

Experience Smart Multipurpose Shared Global Storage with ECS Page 6


Lab Environment
The environment for this lab includes the following:

• 1 x Windows 2012 Domain Controller. This server provides DNS and AD services to the environment.
• 1 x Windows 2012 Launchpad. This server is the host you are currently logged into and provides access into
the other components of this lab.
• Two sites with three ECS nodes each. Each node is a VM running ECS 2.2 software and has two disks: one for
OS install and another for object storage. Real world ECS installs require a minimum four-node setup; this
three-node install is for demonstration purposes only.
• 1 x Linux VM that hosts Spark Notebook

Lab Architecture

Experience Smart Multipurpose Shared Global Storage with ECS Page 7


Lab Credentials
The following set of credentials are used in the labs:

ECS system administrator

• Username: ecsadmin01@vlab.local
• Password: Password123!

Namespace administrator for media unit

• Username: rainpoleadmin@vlab.local
• Password: Password123!

Object User for media team

• Username: mediauser@vlab.local
• Password: Password123!

Alternatively, if you chose to skip the configuration steps in Lab1 and Lab2, following set of credentials have been pre-
configured for use in Labs 3, 4, and 5:

Namespace administrator for media unit

• Username: rainpoleadmin2@vlab.local
• Password: Password123!

Object User for media team

• Username: mediauser2@vlab.local
• Password: Password123!

Experience Smart Multipurpose Shared Global Storage with ECS Page 8


Environment Credentials

The following is a list of all lab servers and appliance and their corresponding IP addresses and credentials.

Note: This table is for reference purposes, you may not access everything when completing the use cases in the lab
guide.

Experience Smart Multipurpose Shared Global Storage with ECS Page 9


Lab Scenario
In these labs, you will see how the storage administrator extends the ECS Cloud Storage platform to the new tenant,
which is the media division. Then you will see how each of the three teams in the division achieve the use cases listed
below:

The content generation team needs to ingest image content and its associated metadata.

Two different teams would use the metadata associated with the media images:

• The media review team has a custom application (Media Reviewer) that needs to search for images based on
the metadata to review the content.
• The analytics team uses a Zeppelin notebook to view object metadata and run Spark SQL queries to extract
data for analysis.

A remote advertising team needs to access archived media content from their windows platform as a mounted file
system, for reference and reuse.

Experience Smart Multipurpose Shared Global Storage with ECS Page 10


Labs
In the exercises, you will experience how ECS smart storage simplifies data access and management with instant
meta-data search, NFS access, analytics enabled by HDFS and built-in optimizations for speed and storage efficiency,
and access data using multiple protocols simultaneously to appreciate how ECS is built from the ground up to support
multiple protocols for unstructured (Object and File) workloads on a single cloud-scale storage platform, natively
speeding up storage for traditional archive services as well as modern web, mobile, Internet of Things (IoT), and
Hadoop applications. You can store, access or consume data without any translation gateways.

You have the opportunity to perform tenant configuration, object data ingest using S3 protocol, metadata search using
SPARK SQL and S3 client applications, and NFS access from a Windows environment.

Note: You can optionally start with Lab 3, which allows you to skip the tenant configuration and object data ingest
modules. You will use a pre-configured tenant and pre-populated data for the metadata search and NFS access Labs.

Contents

Lab 1: Configuring A New Tenant In ECS

• Perform the role of a system administrator to create new namespace and configure a namespace admin
• Perform the role of a namespace admin to enable access for users and applications within the media unit

Lab 2: Ingesting Data Using An S3 Application

• Connect an S3 client application to a namespace and object user


• Observe ingest of image data

Lab 3: Performing Metadata Search Using an S3 Client Application

• Observe how S3 client applications can leverage the ECS 2.2 metadata search capability to efficiently
discover and access data sets

Lab 4: Querying Object Metadata with Apache Spark SQL

• Experience the ability to use an SQL interface to the metadata search capability in ECS
• Write SQL queries to access data based on metdata search criteria

Lab 5: Accessing Data through NFS and S3

• Walk through the use case of ingesting content via S3 and then accessing that content via NFS mount in
Windows.

Experience Smart Multipurpose Shared Global Storage with ECS Page 11


• Configure an NFS export in ECS to allow the media unit to access bucket content via NFS from external sources
sharing data via S3. In other words, experience cross head access of a bucket in ECS using two different
protocols (NFS and S3).
• Access and view the shared content in Windows using NFS Client to mount the bucket and MS Paint to view
the object.

Experience Smart Multipurpose Shared Global Storage with ECS Page 12


Lab 1 - Configuring A New Tenant In
ECS

Experience Smart Multipurpose Shared Global Storage with ECS Page 13


Tenant Configuration
In this lab, you will experience how a storage administrator for RainPole can configure the media department as a
tenant in ECS. Prior to performing this task, an administration account was created that provides the storage
administrator with access to provision these tenants in ECS. ECS supports access by multiple tenants, where each
tenant is defined by a namespace and the namespace has a set of configured users who can store and access objects
within the namespace.

Note: You may skip this lab (Lab 1) and Lab 2 if you prefer to explore the features in ECS with preconfigured tenants
and data.

Open Site 1

The lab is backed by two virtual instances of ECS software, simulating Boston and New York data centers in a geo-
federated environment. The lab experience is identical to provisioning a true ECS appliance.

On the desktop, double click ECS Site1 (Boston)

Experience Smart Multipurpose Shared Global Storage with ECS Page 14


Browser Exception

Click Advanced
Advanced.

Experience Smart Multipurpose Shared Global Storage with ECS Page 15


Add Exception

Click Proceed to 192.168.1.11 (unsafe)


(unsafe).

Experience Smart Multipurpose Shared Global Storage with ECS Page 16


Login to ECS Portal at Site 1 (Boston)

Login using the following credentials:

• User Name: ecsadmin01@vlab.local


• Password: Password123!

Click Login

Open the Guide

To ensure all configuration steps are performed in the right order, the storage administrator can use the configuration
guide, which is displayed on their first login to ECS.

To display the guide on subsequent logins, click the Open Guide icon.

Experience Smart Multipurpose Shared Global Storage with ECS Page 17


Open the Checklist

Click Browse the checklist to access the configuration checklist.

Pre-configuration

Before a new tenant/namespace can be provisioned, a storage pool, virtual data center (VDC) and replication group
need to be configured.

Experience Smart Multipurpose Shared Global Storage with ECS Page 18


In the RainPole configuration there are two VDCs available, each representing a different site - Boston (BosVDC
BosVDC) and
New York (NycVDC
NycVDC). A replication group between these sites called BosNycRG has been created. The storage
administrator needs to start by creating a namespace for the tenant.

Note: All steps appear complete in green because a tenant has been preconfigured for lab users who prefer to skip
this lab (Lab 1) and Lab 2.

You can close the checklist.

Experience Smart Multipurpose Shared Global Storage with ECS Page 19


Create a Namespace

To create a new namespace, click Manage, and then click Namespace


Namespace.

Click New Namespace


Namespace.

Enter Namespace Details

In the New Namespace area, specify the following:

• In the Name field, type media

Experience Smart Multipurpose Shared Global Storage with ECS Page 20


• In the User Admin field, type rainpoleadmin@vlab.local
rainpoleadmin@vlab.local. This is the namespace administrator, a valid domain
user.
• In the Replication Group list box, accept the default, BosNycRG

Scroll down, and then click Save


Save.

Experience Smart Multipurpose Shared Global Storage with ECS Page 21


Namespace Created

Your new Namespace has been created.

Logout

You will now log out of the ECS portal as the storage administrator and continue configuration of the tenant by acting
as the Namespace administrator, rainpoleadmin@vlab.local.

Click logout
logout.

Access the ECS Portal as Namespace Admin

You will now access the ECS portal using the credentials of the Namespace admin domain user. You will create an
object user and provision a bucket for object storage within the media Namespace.

Login using the following credentials:

• User Name: rainpoleadmin@vlab.local


• Password: Password123!

Experience Smart Multipurpose Shared Global Storage with ECS Page 22


Click Login

Error message

An error message will be displayed. This will not affect lab functionality and can be ignored.

Close dialog

Close the dialog.

New Object User

To provide access to the ECS object store, the Namespace administrator must first create an object user with the
appropriate credentials.

Click Manage, and then click Users


Users.

Experience Smart Multipurpose Shared Global Storage with ECS Page 23


Click New Object User
User.

An object user account is utilized by a user or an application to consume the object storage platform via REST API like
S3, Swift or Centera CAS.

Specify a Username and Namespace

• In the Name field, type mediauser@vlab.local


• In the Namespace list box,, accept the default, media

Experience Smart Multipurpose Shared Global Storage with ECS Page 24


Click Next to Add Passwords
Passwords.

Generate a S3 Secret Key (S3 Password)

Object users need a S3 Secret Key to be able to connect to the HTTP endpoint and read/write objects. You will now
create one for the user, mediauser@vlab.local. This S3 Secret Key is distributed out of band. You will save this access
key for use in later labs.

Click Generate and Add Password to create an S3 Secret Key for mediauser@vlab.local.

Experience Smart Multipurpose Shared Global Storage with ECS Page 25


Note: Your access key will differ from the one in the screen shot.

Copy Access Key

Click in the key control field. A blue outline should appear around the field to signify it is active. Triple-click in the field
to select all of the key text. Right-click on the selection and select Copy to copy the access key to clipboard.

Create a Blank Text Document to Store the Access Key

Minimize the ECS Portal. Right-click on the desktop, and select New
New, Text Document
Document. Name the file
mediauser@vlab.local.txt
mediauser@vlab.local.txt.

Experience Smart Multipurpose Shared Global Storage with ECS Page 26


Store the Access Key In the Text Document

Double click to open the text file. Right-click, and paste the access key into the file. Click File and select Save
Save. Finally,
close the text document.

Experience Smart Multipurpose Shared Global Storage with ECS Page 27


Provision a Bucket

Now that you have provisioned an object user and secret key, you will now provision a bucket for the object user to
consume.

Click Buckets
Buckets, and then click New Bucket
Bucket.

Enter Bucket Details

In the New Bucket area, specify the following:

Experience Smart Multipurpose Shared Global Storage with ECS Page 28


• In the Name field, type images
• In the Namespace list box, select media
• In the Replication Group list box, accept the default, BosNycRG
• In the Bucket Owner field, type mediauser@vlab.local. The owner will also act as the owner of the file system
when accessing the bucket via NFS export.

Proceed to the next step to enable metadata search on the bucket and configure indexes.

Enable Metadata Search

The media unit has both a media review team and an analytics team that needs to be able to search for objects within
the bucket using metadata search criteria.

The configuration to enable metadata search has to be done by the Namespace admin using the portal. This can also
be done by a bucket owner or an admin user using the ECS management API or the S3 API to configure metadata
search and indexes at the time of bucket creation.

Experience Smart Multipurpose Shared Global Storage with ECS Page 29


Scroll down to Metadata Search
Search. Click Enabled
Enabled.

Experience Smart Multipurpose Shared Global Storage with ECS Page 30


Configure Metdata Search Keys

The Namespace admin must know the metadata attributes that are required to be searchable. While system metadata
attributes are available to be selected, user metadata keys need to be manually entered.

The teams in the media unit need to be able to search for data based on the following user metadata attributes:

• image-width (integer)
• image-height (integer)
• image-viewcount (integer)
• gps-latitude (decimal)
• gps-longitude (decimal)

To configure these as search keys:

• Click Add
• In the Metadata Key Type list box, select User
• In the Key Name field, type x-amz-meta-image-width
x-amz-meta-image-width. The name is already prefixed, complete the rest.
• In the Data Type list box, select integer
• Click Add

Experience Smart Multipurpose Shared Global Storage with ECS Page 31


Experience Smart Multipurpose Shared Global Storage with ECS Page 32
Configure Additional Search keys

Repeat the previous step for the remaining four metadata search keys:

• image-height (integer)
• image-viewcount (integer)
• gps-latitude (decimal)
• gps-longitude (decimal)

When the five keys are complete, scroll down, and then click Save
Save.

Experience Smart Multipurpose Shared Global Storage with ECS Page 33


Success!

You have successfully operated as a Namespace administrator to create an object user and provision a brand new
bucket for the teams in the media unit. They will now use the object user to ingest and access data.

Proceed to the next lab to start executing the media team use cases of ingest, metadata search and NFS based access.

Experience Smart Multipurpose Shared Global Storage with ECS Page 34


Lab 2 - Ingesting Data Using An S3
Application

Experience Smart Multipurpose Shared Global Storage with ECS Page 35


Data Ingest into the ECS Cloud Storage Platform
The ECS platform supports ingest of data via multiple protocols such as Object (S3, Swift, CAS, Atmos), File (NFS V3),
and HDFS. This lab will allow you to experience the simplicity and efficiency of ingesting data using an S3 client
application.

The content generation team in the media wing uses an S3-based client application called Media Loader to copy
images to the ECS platform. Its configuration process is covered in this lab and is similar to that of any S3 client.

You may skip Lab 1 and Lab 2 if you prefer to explore metadata search and NFS access capabilities with a pre-
configured tenant and pre-populated data.

Launch The Media Loader Client Application

Minimize the ECS portal window, and on the desktop double-click Media Loader
Loader.

Experience Smart Multipurpose Shared Global Storage with ECS Page 36


Enter details to point the client to the Media tenant's bucket

In the Media Loader, specify the following:

• The IP and Port for the Boston site are pre-populated


• In the Username field, type mediauser@vlab.local
• In the Bucket Name field, type images
• Pictures
Pictures: The number of images to be loaded, accept the default, 100

For the secret key, follow the instructions detailed in the next steps.

Secret Key (S3 Password)

Object users need a secret key to be able to connect to the HTTP endpoint and read/write objects to this bucket. You
created this in the previous lab and copied into a file on the desktop called mediauser@vlab.local.txt
mediauser@vlab.local.txt.

Experience Smart Multipurpose Shared Global Storage with ECS Page 37


Open the file, highlight the secret key text, right-click and select Copy
Copy.

Paste Key

Apply the copied key into the Secret Key area in the Media Loader application.

Right-click, and select Paste


Paste.

Experience Smart Multipurpose Shared Global Storage with ECS Page 38


Upload Images into the ECS cloud storage platform

Click Upload to start loading the image files using the S3 client application.

Experience how quickly image contents are copied. When the process is complete you will see the success status and
duration of the copy.

Success!

You have successfully performed object I/O to ECS Cloud Storage.

Proceed to experience how the uploaded data can be accessed through S3 based metadata search and NFS protocols.

Experience Smart Multipurpose Shared Global Storage with ECS Page 39


Lab 3 - Performing Metadata Search
Using An S3 Client Application

Experience Smart Multipurpose Shared Global Storage with ECS Page 40


ECS Metadata Search Media Reviewer
After completing this lab, you will have a better understanding of how to leverage the ability to search for objects
residing in ECS using metadata search.

The media review team of the media unit has an S3 client application to review the content. This lab will take you
through the type of metadata queries that can be performed efficiently by using the ECS metadata search API.

Close Media Loader

Close Media Loader as it is no longer required.

Copy Secret Key

Open the mediauser2@vlab.local.txt file located on the desktop and copy the secret key.

Open Media Reviewer

On the desktop, there are two icons to open the Media Reviewer.

Experience Smart Multipurpose Shared Global Storage with ECS Page 41


Double-click Media Reviewer - Media2.bat to use a preconfigured bucket that has already been loaded with many
images.

Log into Media Reviewer application

In the Media Reviewer, specify the following:

• In the Username field, type mediauser2@vlab.local


• In the Secret Key field, right-click, and select Paste

Click Login

Experience Smart Multipurpose Shared Global Storage with ECS Page 42


Ensure the correct bucket is selected for meta data search

Ensure the images2 bucket is selected.

Experience Smart Multipurpose Shared Global Storage with ECS Page 43


Search Example 1: Identify geographic of pictures with a high view count

In the Minimum number of views field, type 5000

Click Search
Search.

All images in the bucket with a meta data attribute named viewcount that is greater that 5000 will be returned.

Experience Smart Multipurpose Shared Global Storage with ECS Page 44


View image

Click on any of the red pushpins on the map to view more details about the selected image. Observe Number of views
is above 5000
5000.

Click Close
Close.

Experience Smart Multipurpose Shared Global Storage with ECS Page 45


Search Example 2: Search within a specific geographic boundary

Next, you will not be specifying a Minimum number of views but instead will search within a selected geographic
boundary.

Delete 5000 from the Minimum number of views field, and then select Use selected area
area.

Experience Smart Multipurpose Shared Global Storage with ECS Page 46


Search

On the map, click and drag to move the gray marquee down to South America. You can use the circular touch points to
cover the entire landmass of South America.

Click Search
Search.

Based on the geographic boundaries you provided on the map, four meta data search parameters were passed in the
query to ECS. Two parameters indicated that the latitude has to be between the north and south boundaries. The
other two parameters indicated that the longitude had to be between the east and west boundaries.

Click Pictures to view thumbnails of the search results.

Experience Smart Multipurpose Shared Global Storage with ECS Page 47


Show Picture

On any one of the pictures, click Show


Show.

Experience Smart Multipurpose Shared Global Storage with ECS Page 48


Latitude and Longitude

At the bottom of the window, observe the latitude and longitude are within the boundaries of the search you
performed.

Click Close
Close.

Experience Smart Multipurpose Shared Global Storage with ECS Page 49


Logout

Click Logout
Logout, and then close the Media Reviewer.

Conclusion

This lab demonstrates how an application can query objects using meta data indexes specified on the S3 bucket. The
application in this lab used different variations of meta data search queries to find and group results.

You began with a simple search that located all images with a meta data attribute viewcount that was larger than
5000. The second search was more complex, utilizing a geographic range in which longitude and latitude needed to
be within the plot you specified.

Experience Smart Multipurpose Shared Global Storage with ECS Page 50


Lab 4 - Querying Object Metadata
with Apache Spark SQL for use by
HDFS based applications

Experience Smart Multipurpose Shared Global Storage with ECS Page 51


Querying object metadata with Apache Spark SQL for analytics
Introduction

The analytics wing of the media unit uses Zeppelin Notebook with SPARK SQL libraries so that they can run SQL
queries on large data sets. Apache Spark is a fast general engine for large-scale data processing.

In this lab, SPARK SQL queries are translated to ECS metadata search queries so that data can be efficiently accessed
and queries can be serviced by use of ECS metadata indexes.

Launch Zeppelin

Apache Zeppelin is a web-based notebook for data analytics. It provides an easy front end for interacting with Apache
Spark data sets.

Minimize any open windows, and then double-click Zeppelin


Zeppelin.

Experience Smart Multipurpose Shared Global Storage with ECS Page 52


Navigate to Notebook

Click Bucket Metadata Search Tutorial


Tutorial.

Notebook Overview

Inside the notebook, you will see a list of paragraphs. Each paragraph can be run individually by clicking run on the
step or by clicking the run all paragraphs at the top of the notebook. In this lab, you will be running them one at a
time.

Experience Smart Multipurpose Shared Global Storage with ECS Page 53


Load the ECS Search Extension

First, you will need to load the ECS search extension into Zeppelin.

Click the run icon in the paragraph with the z.load() statement. This step may take up to a minute to complete the first
time it is run.

Result

When the extension is loaded, you will see a result object res0 or res1
res1.

Configure the Connection to ECS

Next, a connection to ECS is configured. There are four parameters already provided:

• bucketName = images2
• endpoint = http://192.168.1.11:9020/
• secretKey = the shared secret key for the user, mediauser2@vlab.local.
• user = mediauser2@vlab.local

Experience Smart Multipurpose Shared Global Storage with ECS Page 54


Register bucket

The last two lines in this script will register the bucket as a temporary table for use with SparkSQL. This does not load
any ECS objects into memory; it simply queries ECS for the indexed attributes on the bucket for later querying.

Run

Click the run icon in the paragraph to connect to ECS.

When complete, you will now have a DataFrame object that has columns defined for the indexed attributes in ECS.

Use Spark SQL to Select Objects

Next, you can use Spark SQL to query the objects in your bucket. In the query, you will select all of the objects in the
bucket that have at least 5000 views and up to 10000 views.

Click the run icon in the paragraph to execute the query.

Experience Smart Multipurpose Shared Global Storage with ECS Page 55


Results

The query will run and display the results. Feel free to modify the SQL and try your own examples. You can quickly
execute the SQL statement by pressing Shift-Enter
Shift-Enter.

While it is outside the scope of this lab, such query results can easily be saved on ECS as an HDFS file and made
available to analytics applications.

Compute the Average View Count

The next paragraph will compute the average number of views for the images in the bucket. It uses the avg function to
compute the averages of all the results in the column. Click the run icon to execute the query.

Experience Smart Multipurpose Shared Global Storage with ECS Page 56


Results

The notebook will be populated with the results of your query:

View Count Histogram

Next, you will create a histogram of the view counts. This query uses Hive operators and grouping to break the objects
down into buckets of 1000 views. Click the run icon to execute the query.

Experience Smart Multipurpose Shared Global Storage with ECS Page 57


Graph the Results

A table is displayed with the number of images per each bucket of thousands of views. To view this visually, click the
bar graph icon:

Experience Smart Multipurpose Shared Global Storage with ECS Page 58


Bar Graph

You can now see the same data set rendered as a bar graph. Clicking on the other icons can change the view style of
the result set.

Explore - Optional

SparkSQL is a powerful engine that can be used to analyze your data in many ways. Feel free to experiment. To create
a new paragraph, hover under the previous paragraph and click +

The results of such analysis can also be made available to Hadoop applications directly from the Zeppelin notebook or
by persisting the results as an HDFS file on ECS

Experience Smart Multipurpose Shared Global Storage with ECS Page 59


%sql

To use SparkSQL in your paragraph, start the first line with %sql then begin your SELECT statement on the following
line. Press Shift-Enter to execute your query.

Close Zeppelin

When ready, close Zeppelin


Zeppelin, and then proceed to the next lab.

For More Information

• You can get more information about SparkSQL on the Apache Spark website.
• The Zeppelin notebook application is available from Apache.
• The ECS driver for Spark is available from GitHub under emcvipr/spark-ecs-s3.
• More information on HDFS capabilities in ECS is available in http://www.emc.com/campaign/ecs-hadoop/
index.htm
• This ECS demo is also available as a Docker container from DockerHub under emccorp/spark-ecs-s3.

Experience Smart Multipurpose Shared Global Storage with ECS Page 60


Lab 5 - Accessing Data through NFS

Experience Smart Multipurpose Shared Global Storage with ECS Page 61


Working with ECS Data via NFS and S3
After completing these steps, you will have a better understanding of how to work with data using multiple protocols
(S3 and NFS). You will first upload data via S3 browser into a bucket and then retrieve that data via NFS mount in
Windows Explorer.

This lab details the steps for how an outside marketing firm could sharing advertising media via S3 to RainPole's
advertising team whom them will access this data via NFS mount in Windows.

Logout

If you are still logged in from a previous step, click logout


logout.

Experience Smart Multipurpose Shared Global Storage with ECS Page 62


Log into ECS portal as the Namespace admin

Login using the following credentials:

• User Name: rainpoleadmin2@vlab.local


• Password: Password123!

Click Login

Error message

An error message will be displayed. This will not affect lab functionality and can be ignored.

Close dialog

Close the dialog.

Experience Smart Multipurpose Shared Global Storage with ECS Page 63


Create file system enabled bucket

1. Click Manage
2. Click Buckets
3. In the Namespace list box, ensure media2 is selected
4. Click New Bucket
Bucket.

Bucket details

1. In the Name field, type rainpole-media

Experience Smart Multipurpose Shared Global Storage with ECS Page 64


2. In the Namespace list box, select media2
3. In the Replication Group list box, ensure BosNycRG is selected
4. In the Bucket Owner field, type mediauser2@vlab.local

Scroll down to File System


System.

Experience Smart Multipurpose Shared Global Storage with ECS Page 65


Bucket details contd.

1. Under File System


System, click Enabled
2. In the Default Bucket Group field, type DefaultBucketGroup1
3. Under Group FIle Permissions
Permissions, click Read and Write
4. Under Group Directory Permissions
Permissions, click Read and Write

Scroll down to the end of the page.

Save

Click Save to create the bucket.

Experience Smart Multipurpose Shared Global Storage with ECS Page 66


Confirm creation of new bucket

Your new bucket has been successfully created.

Experience Smart Multipurpose Shared Global Storage with ECS Page 67


Configure the NFS export

1. Click File
2. In the Namespace list box, ensure media2 is selected
3. Click New Export
Export.

Export details

1. In the Namespace list box, ensure media2 is selected


2. In the Bucket list box, select rainpole-media

Experience Smart Multipurpose Shared Global Storage with ECS Page 68


3. Click Add

Add Export Host

1. In the Export Host field, type * (this will allow access from all NFS clients)

Experience Smart Multipurpose Shared Global Storage with ECS Page 69


2. Under Permissions
Permissions, click Read/Write
3. Under Authentication
Authentication, click Sys
4. In the AnonUser field, type mediauser2@vlab.local
5. In the AnonGroup field, type DefaultBucketGroup1
6. In the RootSquash field, type mediauser2@vlab.local
7. Click Add

Experience Smart Multipurpose Shared Global Storage with ECS Page 70


Confirm export

The new export host is displayed in the grid.

Click Save to complete the export configuration.

Verify configuration

Verify your NFS export configuration was added for the rainpole-media bucket.

Experience Smart Multipurpose Shared Global Storage with ECS Page 71


Open S3 Browser

You will now configure access the the file system enabled bucket via S3 Browser. This will allow you to upload your
source image that you want to access and read back via NFS.

On the desktop, double-click S3 Browser


Browser.

Experience Smart Multipurpose Shared Global Storage with ECS Page 72


Configure S3 browser to view the edited image

You are now prompted to Add New Account. Specify the following:

1. In the Account Name field, type RainPole Media


2. In the Storage Type list box, select S3 Compatible Storage
3. In the REST Endpoint field, type 192.168.1.11:9020
4. In the Access KEY ID field, type mediauser2@vlab.local
5. In the Secret Access Key field, open the file mediauser2@vlab.local.txt on the desktop. Right-click, and then
copy/paste the Secret Access Key.
6. Click Add new account

Experience Smart Multipurpose Shared Global Storage with ECS Page 73


Preview the edited image in S3 browser

After the account is added, select rainpole-media


rainpole-media. There are currently no objects in the bucket.

Upload

Click Upload
Upload, and then select Upload file(s)

Experience Smart Multipurpose Shared Global Storage with ECS Page 74


Select file

In the left navigation pane, click Desktop


Desktop.

In the right navigation pane, select ECS.jpg and then click Open
Open.

Experience Smart Multipurpose Shared Global Storage with ECS Page 75


Confirm upload

Confirm ECS.jpg has been successfully uploaded into the rainpole-media bucket.

Mount the NFS export in Windows

You will now use a Windows client for NFS command to mount the export. This command will allow access to the
export as if it were another drive in the Windows environment.

On the taskbar, click Command Prompt


Prompt.

Experience Smart Multipurpose Shared Global Storage with ECS Page 76


Mount command

Type the following command:

mount -o mtype=soft -o nolock -o sec=sys -o fileaccess=777 192.168.1.11:/media2/rainpole-media e:

Press Enter.

Type:

mount

Press Enter
Enter.

Verify the mount is listed and ready.

Open and edit a file via Windows Explorer and MS Paint

On the taskbar, click Windows Explorer


Explorer.

Experience Smart Multipurpose Shared Global Storage with ECS Page 77


Select E:

Using Windows Explorer, you gain a more visual representation of the export you mounted in ECS.

In the left navigation pane, the NFS drive is mounted and named rainpole-media (\\192.168.1.11\media2) (E:)
(E:).

Click to display the contents.

Experience Smart Multipurpose Shared Global Storage with ECS Page 78


Select image

You can think of this as a listing of the objects in the rainpole-media bucket.

Right-click ECS.jpg
ECS.jpg, and then select Edit
Edit.

Experience Smart Multipurpose Shared Global Storage with ECS Page 79


View image

Verify the image displays correctly.

Close Paint
Paint.

Experience Smart Multipurpose Shared Global Storage with ECS Page 80


Conclusion

You have now experienced the capabilities of using NFS within ECS. Furthermore, you have seen first-hand how cross-
head access can be leveraged.

You first created a new file system enabled bucket in ECS which was then referenced via NFS export configuration.
Then, you uploaded content to the bucket via the AWS S3 API using S3 Browser. To experience cross-head access, you
viewed the uploaded image by creating a mount in Windows (using the Windows NFS Client) and opening/previewing
the image.

Experience Smart Multipurpose Shared Global Storage with ECS Page 81


Troubleshooting

Experience Smart Multipurpose Shared Global Storage with ECS Page 82


Troubleshooting & Tips
If you encounter issues during your vLab experience, there are a couple of things to check.

Check ECS Health

If you have trouble logging in to the ECS GUI (hangs, reports bad password), you should make sure that ECS has fully
initialized. Sometimes if the vLab is busy, this can take a little longer than normal.

Launch DT Status Checker

On your desktop, double-click the icon that says ECS DT Status


Status:

Wait for ECS to Become 100% Ready

Inside the GUI, you'll see the initialization status for each Virtual Data Center (VDC). If both VDCs do not indicate
100% readiness, wait for initialization to complete and then return to your task.

Experience Smart Multipurpose Shared Global Storage with ECS Page 83


Once all the DTs (directory tables) are initialized, you should see 100% in both VDCs. Note that it's normal for the
VDCs to have a different number of total DTs.

Restart Zeppelin Interpreter

If you get any strange errors with Zeppelin, you can try restarting its interpreter. First, click on the Interpreter tab at the
top:

Then click the restart button to restart the interpreter. You can then go back to your notebook and restart your
execution from the top.

Experience Smart Multipurpose Shared Global Storage with ECS Page 84


Recreate NFS mount

If you experience any issues (like the message seem below) attempting to access/modify images via NFS, you can try
to recreate the NFS mount by following the steps below.

Experience Smart Multipurpose Shared Global Storage with ECS Page 85


From the command prompt, unmount the NFS export by typing:

umount e:

Enter. If you're warned and asked to continue this operation, type y and press Enter
Press Enter Enter.

Type the following command to recreate the mount:

mount -o mtype=soft -o nolock -o sec=sys -o fileaccess=777 192.168.1.11:/media2/images2 e:

Press Enter
Enter. You will see a success message.

Type the following command and press Enter to verify your mount properties.

mount

Verify the mount is listed and ready. Note that the drive letter is e:

Experience Smart Multipurpose Shared Global Storage with ECS Page 86


Conclusion

Experience Smart Multipurpose Shared Global Storage with ECS Page 87


Conclusion
Congratulations!

RainPole's IT team has successfully extended the ECS cloud storage platform to its media and communications wing.

All of the following needs have been successfully met and the IT team is convinced that ECS is a smart and flexible
multipurpose cloud storage platform that addresses varied needs:

Ingest of image content and associated metadata by the content generation team.

Use of the metadata associated with the media images by two different teams:

• The media review team used a custom application (Media Reviewer) to search for images based on the
metadata to review the content
• The analytics team used a Zeppelin notebook to view object metadata and run Spark SQL queries to extract
data for analysis

A remote advertising team could access archived media content from their windows platform as a mounted file
system, for reference and reuse.

Experience Smart Multipurpose Shared Global Storage with ECS Page 88


Additional Solution Information

Experience Smart Multipurpose Shared Global Storage with ECS Page 89


Business Challenge
RainPole

RainPole’s enterprise IT offers a competitive cloud storage platform to all business units using EMC Elastic Cloud
Storage. They have achieved Terabyte to Exabyte multisite scale-out architecture, giving customers flexibility and
control to scale out according to business needs.

RainPole's Media Business

RainPole now wants to extend their cloud infrastructure to the media and communications wing. The media wing is
faced with an explosive growth of unstructured data and the need to serve modern cloud applications.

The following use cases are to be supported for the media wing:

The content generation team needs to ingest image content and its associated metadata.

Two different teams would use the metadata associated with the media images:

• The media review team has a custom application (Media Reviewer) that needs to search for images based on
the metadata to review the content.
• The analytics team wants to use Zeppelin notebook to view object metadata and run Spark SQL queries to
extract data for analysis.

A remote advertising team needs to access archived media content from their Windows platform as a mounted file
system for reference and reuse.

Experience Smart Multipurpose Shared Global Storage with ECS Page 90


Solution
RainPole’s IT team have evaluated EMC ECS 2.2 and determined that its metadata search functionality and NFS V3
support can help address the media unit’s needs. Metadata search can assist in the discovery of objects based on
metadata, as ECS 2.2 has the ability to index object metadata and make it available for search. Support for NFS
enables access to object data as a file system by legacy applications requiring file access.

The ECS infrastructure gives RainPole’s IT team the ability to offer a wide range of data ingest and access options such
as Object protocols (S3, Swift), NFS, and Hadoop file system access to achieve a TCO that is 30% lower than that of
public cloud storage. ECS also provides the flexibility of being deployed as software on commodity hardware or in a
turnkey appliance form factor.

Experience Smart Multipurpose Shared Global Storage with ECS Page 91


ECS
Elastic Cloud Storage (ECS) is a multi-purpose shared global storage that scales to store exabytes of both small and
large files with strong global consistency.

ECS is built from the ground up to support multiple protocols for unstructured (Object and File) workloads on a single
cloud-scale storage platform, natively speeding up storage for traditional archive services as well as modern web,
mobile, Internet of Things (IoT), and Hadoop applications. You can store, access or consume data without any
translation gateways.

Experience Smart Multipurpose Shared Global Storage with ECS Page 92


Elastic Cloud Storage Benefits

ECS delivers simple storage management of globally-distributed infrastructure under a single global namespace with
anywhere access to content. It simplifies data governance and management with instant metadata search, analytics
enabled by HDFS, and built-in optimizations for speed and storage efficiency.

Building on the EMC experience for enterprise-grade features for protection, availability, encryption, authentication,
and access controls, ECS delivers a long list of ISV application and technology support. ECS provides you more control
of your data assets with enterprise-class object, file, and HDFS storage in a secure and compliant system. With the
new management, monitoring, and scripting capabilities, you can offer storage as a service within your enterprises.

ECS features a flexible software-defined architecture that is layered in such a way to promote limitless scalability. Each
layer is completely abstracted and independently scalable with high availability and no single point of failure.

ECS brings public cloud capabilities within your own datacenter and behind the corporate firewall. Leveraging COTS
(commodity off the shelf) hardware, high storage efficiency, smaller datacenter footprint, and simple management,
enterprises can benefit from public-cloud economics within your own datacenter.

Experience Smart Multipurpose Shared Global Storage with ECS Page 93


Other Client Applications
Media Reviewer

A sample custom client application that runs on Windows platforms and uses the S3 API interface to extract data from
ECS.

Media Loader

A sample s3 client application that ingests image data into the ECS cloud storage platform using the S3 object
protocol.

Apache Zeppelin Spark Notepad

A sample general-purpose client application. It is a web-based notebook that enables interactive data analytics and
use of SPARK SQL on top of ECS metadata search APIs.

Experience Smart Multipurpose Shared Global Storage with ECS Page 94