NAME :

YEAR :
BRANCH :
REGISTER NUMBER :

Certified to be the bonafide record of work done by the above

student in VIIth SEMESTER in CS8711 –CLOUD COMPUTING LABORATORY
during the year 2022 – 2023

Signature of the staff in-charge Signature of the Head of the Department

Submitted for the practical examination held on .

Internal Examiner External Examiner

 INDEX
SL. DATE TITLE PAGE MARK SIGNATURE
NO NO
Virtual box/VMware Workstation
1 02/09/2022 with different flavours of Linux or
windows OS

Installation a C compiler and

2 09/09/2022 execution of simple C program in
Virtual machines.

Installation of Google App Engine.

3 16/09/2022
Create hello world Application

Use GAE(Google App Engine)

4 23/09/2022 launcher to launch the web
applications

Execution of Scheduling Algorithms

5 07/10/2022
Using Cloud Simulator

Transfer the files from one virtual

6 14/10/2022 machine to another Virtual
machine/host operating System

Install Hadoop single node cluster and

7 28/10/2022 execute Word Count Program using
Map reduce Concept

Open Stack Online Demo For

8 04/11/2022
Virtualization
CS8711 CLOUD COMPUTING LABORATORY LTPC 0032

OBJECTIVES:

The student should be made to:

• Be exposed to tool kits for grid and cloud environment.
• Be familiar with developing web services/Applications in grid framework
• Learn to run virtual machines of different configuration.
• Learn to use Hadoop

LIST OF EXPERIMENTS:

Course Objective:
• To develop web applications in cloud
• To learn the design and development process involved in creating a cloud based application
• To learn to implement and use parallel programming using Hadoop
Exercises:
1. Install Virtualbox/VMware Workstation with different flavours of linux or windows OS on top
of windows7 or 8.
2. Install a C compiler in the virtual machine created using virtual box and execute Simple
Programs
3. Install Google App Engine. Create hello world app and other simple web applications using
python/java.
4. Use GAE launcher to launch the web applications.
5. Simulate a cloud scenario using CloudSim and run a scheduling algorithm that is not present in
CloudSim.
6. Find a procedure to transfer the files from one virtual machine to another virtual machine.
7. Install Hadoop single node cluster and run simple applications like wordcount.
8. Find a procedure to launch virtual machine using trystack (Online Openstack Demo Version)
Course Outcome:

On completion of this course, the students will be able to:

• Configure various virtualization tools such as Virtual Box, VMware workstation.

• Design and deploy a web application in a PaaS environment.
• Learn how to simulate a cloud environment to implement new schedulers.
• Install and use a generic cloud environment that can be used as a private cloud.
• Manipulate large data sets in a parallel environment.

TOTAL: 45 PERIODS
Ex No.1A CLOUD COMPUTING

EX.No:1 VMware Workstation Installation

Aim:
Find procedure to Install Virtualbox/VMware Workstation with different flavours of linux or
windows OS on top of windows7 or 8.

PROCEDURE TO INSTALL
Step 1- Download Link
Link for downloading the software is https://www.vmware.com/products/workstation-pro/workstation-
pro-evaluation.html. Download the software for windows. Good thing is that there is no signup process.
Click and download begins. Software is around 541 MB.

Step 2- Download the installer file

It should probably be in the download folder by default, if you have not changed the settings in your
browser. File name should be something like VMware-workstation-full-15.5.1-15018445.exe. This file
name can change depending on the version of the software currently available for download. But for
now, till the next version is available, they will all be VMware Workstation 15 Pro.

Step 3- Locate the downloaded installer file

For demonstration purpose, I have placed the downloaded installer on my desktop. Find the installer on
your system and double click to launch the application.
 VMware workstation 15 pro for windows 10 installer file screenshot.
Step 4- User Access Control (UAC) Warning
Now you should see User Access Control (UAC) dialog box. Click yes to continue.

VMware Workstation 12 Pro installer windows 10 UAC screenshot

Initial Splash screen will appear. Wait for the process to complete.

04 2
VMware Workstation 15 Installation Splash Screen
Step 5- VMware Workstation Setup wizard
Now you will see VMware Workstation setup wizard dialog box. Click next to continue.
 VMware Workstation 15 Installation – Setup Wizard
Step 6- End User Licence Agreement
This time you should see End User Licence Agreement dialog box. Check “I accept the terms in the
Licence Agreement” box and press next to continue.

06 4
VMware Workstation 15 Installation – End User Licence Agreement
Step 7- Custom Setup options
Select the folder in which you would like to install the application. There is no harm in leaving the
defaults as it is. Also select Enhanced Keyboard Driver check box.
VMware Workstation 15 Pro installation – select installation folder
Step 8- User Experience Settings
Next you are asked to select “Check for Updates” and “Help improve VMware Workstation Pro”. Do as
you wish. I normally leave it to defaults that is unchecked.
VMware Workstation 15 Installation – User Experience Settings
Step 9- Application Shortcuts preference
Next step is to select the place you want the shortcut icons to be placed on your system to launch the
application. Please select both the options, desktop and start menu and click next.
VMware workstation 15 pro installation shortcut selection checkbox screenshot.
Step 10- Installation begins
Now you see the begin installation dialog box. Click install to start the installation process.

Screenshot for VMware Workstation 15 pro installation begin confirmation dialog box on windows 10.
Below screenshot shows Installation in progress. Wait for this to complete.
Screenshot for VMware Workstation 15 pro installation process.
At the end you will see installation complete dialog box. Click finish and you are done with the
installation process. You may be asked to restart your computer. Click on Yes to restart.
VMware Workstation 15 Installation – Installation Complete
Step 11- Launch VMware Workstation
After the installation completes, you should see VMware Workstation icon on the desktop. Double click
on it to launch the application.
Screenshot for VMware Workstation 15 Pro icon on windows 10 desktop.
Step 12- Licence Key
If you see the dialog box asking for licence key, click on trial or enter the licence key. Then what you
have is the VMware Workstation 15 Pro running on your windows 10 desktop. If don’t have the licence
key, you will have 30 days trial.

VMware Workstation 15 Pro home screen

Step 13- At some point if you decide to buy
At some point of time if you decide to buy the Licence key, you can enter the Licence key by going
to Help->Enter a Licence Key
You can enter the 25 character licence key in the dialog box shown below and click OK. Now you have
the licence version of the software.
Ex No.1B Install VirtualBox/VMware Workstation with different flavours of Linux or
windows OS on top of windows 7 or 8.

Introduction

❖ Virtualization:
• Virtualization is the creation of virtual servers, infrastructures, devices and
computing resources.
• Virtualization changes the hardware-software relations and is one of the
foundational elements of cloud computing technology that helps utilize the
capabilities of cloud computing to the full.
• Virtualization techniques allow companies to turn virtual their networks,
storage, servers, data, desktops and applications.

❖ Hypervisor or Virtual Machine Monitor (VMM)

A hypervisor or virtual machine monitor (VMM) is a piece of computer
software, firmware or hardware that creates and runs virtual machines. A
computer on which a hypervisor is running one or more virtual machines is
defined as a host machine. Each virtual machine is called a guest machine. The
hypervisor presents the guest operating systems with a virtual operating platform
and manages the execution of the guest operating systems. Multiple instances of
a variety of operating systems may share the virtualized hardware resources.

❖ Types of Virtualization
• Operating-system-level virtualization - is a server-virtualization method
where the kernel of an operating system allows for multiple isolated user-
space instances, instead of just one. Such instances (sometimes called
containers, software containers,[1] virtualization engines (VE), virtual
private servers (VPS), or jails) may look and feel like a real server from the
point of view of its owners and users
• Platform / Hardware virtualization -Hardware virtualization or platform
virtualization refers to the creation of a virtual machine that acts like a real
computer with an operating system. Software executed on these virtual
machines is separated from the underlying hardware resources. For example,
a computer that is running Microsoft Windows may host a virtual machine
that looks like a computer with the Ubuntu Linux operating system; Ubuntu-
based software can be run on the virtual machine.
• In hardware virtualization, the host machine is the actual machine on which
the virtualization takes place, and the guest machine is the virtual machine.
The words host and guest are used to distinguish the software that runs on the
physical machine from the software that runs on the virtual machine. Different
types of hardware virtualization include:
 IT8711 FOSS and Cloud Computing Lab

o Full virtualization: Almost complete simulation of the actual

hardware to allow software, which typically consists of a guest
operating system, to run unmodified.
o Partial virtualization: Some but not all of the target environment
is simulated. Some guest programs, therefore, may need
modifications to run in this virtual environment.
o Para virtualization: A hardware environment is not simulated;
however, the guest programs are executed in their own isolated
domains, as if they are running on a separate system.
• Application virtualization is software technology that encapsulates computer
programs from the underlying operating system on which it is executed. A fully
virtualized application is not installed in the traditional sense, although it is still
executed as if it were.

❖ Oracle Virtualbox

o VirtualBox is a general-purpose full virtualizer for x86 hardware, targeted at

server, desktop and embedded use.Each virtual machine can execute its own
operating system, including versions of Microsoft Windows, Linux, BSD, and
MS-DOS. VMware Workstation is developed and sold by VMware, Inc., a
division of EMCCorporation
❖ Ubuntu

o Ubuntu is an operating system like any other and it is free & open source. It
means that we can download it freely and install on as many computers as we
like. By the term open source it means that we can actually see its code. To
provide a more secure environment, the ―SUDO‖ tool is used to assign
temporary privileges for performing administrative tasks. Ubuntu comes
installed with a wide range of Software that includes Libre Office ,Firefox,
Thunderbird

Steps in Installing Oracle Virtualbox with CentOS 7

Step 1:

• Download and Install Oracle Virtual Box latest version & Extension
package
o https://virtualbox.org/wiki/downloads

Step 2:

• Download CentOS 7 OVA(Open Virtual Appliance) from

o Link : https://linuxvmimages.com/images/centos-7
Step 3: The files are downloaded in your Local machine and Click the Oracle VM
VirtualBox 6.0.8 Setup Wizard

Oracle VM VirtualBox 6.0.8 Setup Wizard is open and follows the steps to install
Oracle VM VirtualBox 6.0.8 installation is completed
Step 4: Import the Oracle VM Virtual Extension pack into the Oracle Virtual Box
 Oracle VMVirtual Extension pack is imported into the Oracle Virtual Box
successfully.
Step 5: Click import Appliance and select CentOS-7.8.2003.com OVA (Open Virtual
Appliance) file from download directory
Installation process started
Step 6: CentOS-7.8.2003.com is installed successfully and click start button to launch the
virtual machine

Step 7: Login into CentOS 7

• Login Details
o User name : centos
o Password : centos
 Install a C compiler in the virtual machine created using virtual box
Ex No.2 and execute Simple Programs

Steps in Installing C or C++ Compiler in Virtual machine and executing simple

programs

Step 1 : Install the C or C++ compiler on Ubuntu-14.04 Virtual Machine by

$ sudo apt install g++

Step 2: Create a file for writing C program.

$ sudogedit add.c

Source Code:

Sum of two numbers

#include<stdio.h>
int main()
{
int a,b,c;
printf("Enter two nos:");
scanf("%d%d",&a,&b);
c=0;
c=a+b;
printf("Sum of two nos is: %d",c);
return 0;
}
Step 3: Compile the Program
$sudo g++ add.c

Step 4: Run the Program

$ ./a.out

Expected Output:
Enter two nos : 2 3
Sum of two nos is: 5
Output:
Ex No.3 Install Google App Engine. Create hello world app and other simple web
applications using python/java. Use GAE launcher to launch the web
applications
Introduction

➢ Google Cloud Platform (GCP)

o Google Cloud Platform (GCP), offered by Google, is a suite of cloud

computing services that runs on the same infrastructure that Google uses
internally for its end-user products, such as Google Search, Gmail, file storage,
and YouTube.
o Alongside a set of management tools, it provides a series of modular cloud
services including computing, data storage, data analytics and machine learning.
o Google Cloud Platform provides infrastructure as a service, platform as a
service, and serverless computing environments.

➢ Platform as a Service (PaaS)

o Cloud computing service which provides a computing platform and a solution
stack as a service.
o Consumer creates the software using tools and/or libraries from the provider.
o Provider provides the networks, servers, storage, etc.

➢ Google App Engine:

o Google App Engine was first released as a beta version in April 2008.
o It is a is a Platform as a Service (PaaS) cloud computing platform for
developing and hosting web applications in Google-managed data centers.
 o Google‘s App Engine opens Google‘s production to any person in the world at
no charge.
o Google App Engine is software that facilitates the user to run his web
applications on Google infrastructure.
o It is more reliable because failure of any server will not affect either the
performance of the end user or the service of the Google.
o It virtualizes applications across multiple servers and data centers.
▪ Other cloud-based platforms include offerings such as Amazon Web
Services and Microsoft's Azure Services Platform.
➢ Introduction of Google App Engine
• Google App Engine lets you run your web applications on Google's
infrastructure. App Engine applications are easy to build, easy to maintain, and
easy to scale as your traffic and data storage needs grow. With App Engine,
there are no servers to maintain: You just upload your application, and it's ready
to serve your users.
• You can serve your app from your own domain name (such as
https://www.example.com/) using Google Apps. Or, you can serve your app
using a free name on the appspot.com domain. You can share your application
with the world, or limit access to members of your organization.
• Google App Engine supports apps written in several programming languages.
With App Engine's Java runtime environment, you can build your app using
standard Java technologies, including the JVM, Java servlets, and the Java
programming language—or any other language using a JVM-based interpreter
or compiler, such as JavaScript or Ruby. App Engine also features a dedicated
Python runtime environment, which includes a fast Python interpreter and the
Python standard library. The Java and Python runtime environments are built to
ensure that your application runs quickly, securely, and without interference
from other apps on the system.
• With App Engine, you only pay for what you use. There are no set-up costs and
no recurring fees. The resources your application uses, such as storage and
bandwidth, are measured by the gigabyte, and billed at competitive rates. You
control the maximum amounts of resources your app can consume, so it always
stays within your budget. App Engine costs nothing to get started. All
applications can use up to 500 MB of storage and enough CPU and bandwidth
to support an efficient app serving around 5 million page views a month,
 absolutely free. When you enable billing for your application, your free limits
are raised, and you only pay for resources you use above the free levels.
➢ Architecture of Google App Engine

➢ Features of Google App Engine

➢ GAE Application Environment:

• Google App Engine makes it easy to build an application that runs reliably, even
under heavy load and with large amounts of data. App Engine includes the
following features:
• Persistent storage with queries, sorting and transactions
• Automatic scaling and load balancing
• APIs for authenticating users and sending email using Google Accounts
• Task queues for performing work outside of the scope of a web request
• Scheduled tasks for triggering events at specified times and regular intervals
• Dynamic web serving, with full support for common web technologies

➢ Java Runtime Environment

• You can develop your application for the Java runtime environment using
common Java web development tools and API standards. Your app interacts
with the environment using the Java Servlets standard, and can use common
web application technologies such as Java Server Pages
• The Java runtime environment uses Java 6. The App Engine Java SDK supports
developing apps using either Java 5 or 6. The environment includes the Java SE
Runtime Environment (JRE) 6 platform and libraries. The restrictions of the
sandbox environment are implemented in the JVM. An app can use any JVM
byte code or library feature, as long as it does not exceed the sandbox
restrictions. For instance, byte code that attempts to open a socket or write to a
file will throw a runtime exception.
• Your app accesses most App Engine services using Java standard APIs. For the
App Engine data store, the Java SDK includes implementations of the Java Data
Objects (JDO) and Java Persistence API (JPA) interfaces. Your app can use the
JavaMail API to send email messages with the App Engine Mail service. The
java.net HTTP APIs accesses the App Engine URL fetch service.
• App Engine also includes low-level APIs for its services to implement
additional adapters, or to use directly from the application. See the
documentation for the data store, memcache, URL fetch, mail, images and
Google Accounts APIs. Typically, Java developers use the Java programming
language and APIs to implement web applications for the JVM. With the use
 of JVM-compatible compilers or interpreters, you can also use other languages
to develop web applications, such as JavaScript, Ruby.

➢ Workflow of Google App Engine

Step1 : Login to www.cloud.google.com

Step2 : Goto Console

Step 3 : Google Cloud Platform is shown

Step 4 : Click Dashboard in the Google Cloud Plaform

Step 5 : Dashboard in the Google Cloud Plaform

Step 6 : Click New Project and give unique Project Name.

Example : kcet-cloud-project
Step 7 : Google App Engine is initated

Step 8 : Click create Application

Step 9 : Create app and Select Language Python

Step 10 : Python app is created in Google App Engine

Step 11 : Python app Engine application is created

Step 12 : Click Cloud Shell in the Kathir-Cloud-Project

Step 13 : Create a Directory PythonProject using mkdir command

Syntax : mkdir PythonProject

Step 14 : Click Editor to create Python application

Step 15 : Click New File in the PythonProject Folder (Python file)

Step 16 : Create main.py file

main.py file

import &nbsplogging

from &nbspflask import &nbspFlask

app =&nbspFlask( name )

@app.route('/')
def hello():
return 'Hello&nbspWorld'

if name == ' main ':

&nbspapp.run(host='127.0.0.1',port=8080, debug=True)

Step 17 : Create app.yaml file

app.yaml

runtime: python
env: flex
entrypoint: gunicorn -b :$PORT&nbspmain:app

runtime_config:
python_version: 3
Step 18 : Create requirements.txt file

requirements.txt

Flask==0.11.1
gunicorn==19.6.0

Step 19 : Move to Cloud Shell Environment to run the application

Step 20 : Move to Cloud Shell Environment to run the application

Syntax : gcloud app deploy

Continue the application. It enable service on the given project

It started building the object and fetching the storage object for the created application
It is updating the service

The application is successfully deployed and URL is

https://expanded-curve-289413.uc.r.appspot.com
Step 21 : Run your program in the broswer

Step 22 : Hello World Program is sucessfully run in the browser

Ex.no 3 Create a Simple Hello world application Using GAE

Main points for 3rd experiment in cloud lab

1.Email sign in

2. install google cloud sdk tools(installing time will depend on internet speed of ur computer)

3. create 1 folder and create 2 files inside the folder( like folder name exp3(files 1.app.yaml
2.index.py)

File1

app.yaml code
runtime: python27
api_version: 1
threadsafe: false

handlers:
- url: /
script: index.py

file2
index.py(python file)
print "welcome to python gae";

step4. command prompt commands(open cloud sdk shell command prompt)

C:\Users\cse\AppData\Local\Google\Cloud SDK> google-cloud-sdk\bin\dev_appserver.py

"C:\Users\cse\Downloads\Documents\exp3"

Step 5. Output type in chrome browser

http://localhost:8080/

welcome to google app engine

Ex.No 4 Create a Simple Web application Using GAE

Main points for 4rd experiment in cloud lab

1.Email sign in
2. install google cloud sdk tools(installing time will depend on internet speed of ur computer)
3. create 1 folder for images and create 5 files for web application execution( like folder name
exp3(app.yaml,index.html,main.css,todo.py)

Create one folder -under that folder create 4files and 1 images folder (img
add

step4. command prompt commands(open cloud sdk shell command prompt)

C:\Users\cse\AppData\Local\Google\Cloud SDK> google-cloud-sdk\bin\dev_appserver.py

"C:\Users\cse\Downloads\Documents\exp4"

Step 5. Output

type in chrome browser http://localhost:8080/

Create new Todo?

Login with your Google Account to create and review todos. Login
Just do default login details and add one todo list.
 Ex No.5 Simulate a cloud scenario using CloudSim

Simulate a cloud scenario using CloudSim and run a

scheduling algorithm that is not present in CloudSim.
Introduction:
❖ CloudSim
• A Framework for modeling and simulation of Cloud Computing
Infrastructures and services
• Originally built at the Cloud Computing Distributed Systems (CLOUDS)
Laboratory, The University of Melbourne, Australia
• It is completely written in JAVA
❖ Main Features of CloudSiM
o Modeling and simulation
o Data centre network topologies and message-passing applications
o Dynamic insertion of simulation elements
o Stop and resume of simulation
o Policies for allocation of hosts and virtual machines
❖ Cloudsim – Essentials
• JDK 1.6 or above http://tinyurl.com/JNU-JAVA
• Eclipse 4.2 or above http://tinyurl.com/JNU-Eclipse
• Alternatively NetBeanshttps://netbeans.org/downloads
• Up & Running with cloudsim guide: https://goo.gl/TPL7Zh
❖ Cloudsim-Directory structure
• cloudsim/ -- top level CloudSim directory
• docs/ -- CloudSim API Documentation
• examples/ -- CloudSim examples
• jars/ -- CloudSim jar archives
• sources/ -- CloudSim source code
❖ Cloudsim - Layered Architecture
❖ Cloudsim - Component model classes
o CloudInformationService.java
o Datacenter.java,Host.java,Pe.java
o Vm.java,Cloudlet.java
o DatacenterBroker.java
o Storage.java,HarddriveStorage.java, SanStorage.java

❖ Cloudsim - Major blocks/Modules

o org.cloudbus.cloudsim
o org.cloudbus.cloudsim.core
o org.cloudbus.cloudsim.core.predicates
o org.cloudbus.cloudsim.distributions
o org.cloudbus.cloudsim.lists
o org.cloudbus.cloudsim.network
o org.cloudbus.cloudsim.network.datacenter
o org.cloudbus.cloudsim.power
o org.cloudbus.cloudsim.power.lists
o org.cloudbus.cloudsim.power.models
o org.cloudbus.cloudsim.provisioners
o org.cloudbus.cloudsim.util

❖ Cloudsim - key components

o Datacenter
o DataCenterCharacteristics
o Host
o DatacenterBroker
o RamProvisioner
o BwProvisioner
o Storage
o Vm
o VMAllocationpolicy
o VmScheduler
o Cloudlet
o CloudletScheduler
o CloudInformationService
o CloudSim
o CloudSimTags
o SimEvent
o SimEntity
o CloudsimShutdown
o FutureQueue
o DefferedQueue
o Predicate and associative classes.
CloudSim Elements/Components
SCHEDULING ALGORTIHMS USING CLOUD SIM
How to use CloudSim in Eclipse
CloudSim is written in Java. The knowledge you need to use CloudSim is basic Java
programming and some basics about cloud computing. Knowledge of programming IDEs such
as Eclipse or NetBeans is also helpful. It is a library and, hence, CloudSim does not have to be
installed. Normally, you can unpack the downloaded package in any directory, add it to the Java
classpath and it is ready to be used. Please verify whether Java is available on your system.
To use CloudSim in Eclipse:
1. Download CloudSim installable files
from https://code.google.com/p/cloudsim/downloads/list and unzip
2. Open Eclipse
3. Create a new Java Project: File -> New
4. Import an unpacked CloudSim project into the new Java Project
5. The first step is to initialise the CloudSim package by initialising the CloudSim library, as
follows:

CloudSim.init(num_user, calendar, trace_flag)

6. Data centres are the resource providers in CloudSim; hence, creation of data centres is a
second step. To create Datacenter, you need the DatacenterCharacteristics object that stores the
properties of a data centre such as architecture, OS, list of machines, allocation policy that covers
the time or spaceshared, the time zone and its price:

Datacenter datacenter9883 = new Datacenter(name, characteristics, new VmAllocationPolicySimple(hostList), s

7. The third step is to create a broker:

DatacenterBroker broker = createBroker();

8. The fourth step is to create one virtual machine unique ID of the VM, userId ID of the VM’s
owner, mips, number Of Pes amount of CPUs, amount of RAM, amount of bandwidth, amount
of storage, virtual machine monitor, and cloudletScheduler policy for cloudlets:

Vm vm = new Vm(vmid, brokerId, mips, pesNumber, ram, bw, size, vmm, new
CloudletSchedulerTimeShared())
9. Submit the VM list to the broker:

broker.submitVmList(vmlist)
10. Create a cloudlet with length, file size, output size, and utilisation model:

Cloudlet cloudlet = new Cloudlet(id, length, pesNumber, fileSize, outputSize, utilizationModel, utilizationMode
11. Submit the cloudlet list to the broker:

broker.submitCloudletList(cloudletList)
12. Start the simulation:

CloudSim.startSimulation()
Sample Output from the Existing Example:
Starting CloudSimExample1...
Initialising...
Starting CloudSim version 3.0
Datacenter_0 is starting...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>null
Broker is starting...
Entities started.
0.0: Broker: Cloud Resource List received with 1 resource(s)
0.0: Broker: Trying to Create VM #0 in Datacenter_0
0.1: Broker: VM #0 has been created in Datacenter #2, Host #0
0.1: Broker: Sending cloudlet 0 to VM #0
400.1: Broker: Cloudlet 0 received
400.1: Broker: All Cloudlets executed. Finishing...
400.1: Broker: Destroying VM #0
Broker is shutting down...
Simulation: No more future events
CloudInformationService: Notify all CloudSim entities for shutting down.
Datacenter_0 is shutting down...
Broker is shutting down...
Simulation completed.
Simulation completed.
========== OUTPUT ==========
Cloudlet ID STATUS Data center ID VM ID Time Start Time Finish Time
0 SUCCESS 2 0 400 0.1 400.1
*****Datacenter: Datacenter_0*****
User id Debt
3 35.6

CloudSimExample1 finished!
 1. You can copy few (or more) lines with copy & paste mechanism.
For this you need to share clipboard between host OS and guest OS, installing Guest
Addition on both the virtual machines (probably setting bidirectional and restarting them).
You copy from guest OS in the clipboard that is shared with the host OS.
Then you paste from the host OS to the second guest OS.
2. You can enable drag and drop too with the same method (Click on the machine, settings,
general, advanced, drag and drop: set to bidirectional )
3. You can have common Shared Folders on both virtual machines and use one of the
directory shared as buffer to copy.
Installing Guest Additions you have the possibility to set Shared Folders too. As you put a
file in a shared folder from host OS or from guest OS, is immediately visible to the other.
(Keep in mind that can arise some problems for date/time of the files when there are
different clock settings on the different virtual machines).
If you use the same folder shared on more machines you can exchange files directly copying
them in this folder.
4. You can use usual method to copy files between 2 different computer with client-server
application. (e.g. scp with sshd active for linux, winscp... you can get some info about SSH
servers e.g. here)
You need an active server (sshd) on the receiving machine and a client on the sending
machine. Of course you need to have the authorization setted (via password or, better, via
an automatic authentication method).
Note: many Linux/Ubuntu distribution install sshd by default: you can see if it is running
with pgrep sshd from a shell. You can install with sudo apt-get install openssh-server.
5. You can mount part of the file system of a virtual machine via NFS or SSHFS on the
other, or you can share file and directory with Samba.
You may find interesting the article Sharing files between guest and host without
VirtualBox shared folders with detailed step by step instructions.
You should remember that you are dialling with a little network of machines with different
operative systems, and in particular:

• Each virtual machine has its own operative system running on and acts as a physical
machine.
• Each virtual machine is an instance of a program owned by an user in the hosting operative
system and should undergo the restrictions of the user in the hosting OS.
E.g Let we say that Hastur and Meow are users of the hosting machine, but they did not
allow each other to see their directories (no read/write/execute authorization). When each of
them run a virtual machine, for the hosting OS those virtual machine are two normal
programs owned by Hastur and Meow and cannot see the private directory of the other user.
This is a restriction due to the hosting OS. It's easy to overcame it: it's enough to give
authorization to read/write/execute to a directory or to chose a different directory in which
both users can read/write/execute.
• Windows likes mouse and Linux fingers. :-)
I mean I suggest you to enable Drag & drop to be cosy with the Windows machines and
the Shared folders or to be cosy with Linux.
When you will need to be fast with Linux you will feel the need of ssh-keygen and
to Generate once SSH Keys to copy files on/from a remote machine without writing
password anymore. In this way it functions bash auto-completion remotely too!
Ex No. Simulate a cloud scenario using CloudSim and running a scheduling
algorithm

Procedure to import Eclipse, running scheduling algorithms in your system

Step 1: Link to download Eclipse and download Eclipse for Windows 64bit into your Local
machine

https://www.eclipse.org/downloads/packages/release/kepler/sr1/eclipse-ide-
java-developers

Windows x86_64

Step 2: Download scheduling source code cloudsim-code-master from git hub repository in
your local machine

https://github.com/shiro873/Cloudsim-Code

Src-scheduling source
files
Step 3: Download commons-maths3-3.6.1 from git hub repository in your local machine

https://commons.apache.org/proper/commons-math/download_math.cgi

Commons-maths3-
3.6.1-bin.zip

Step 4: Downloaded Eclipse, cloudsim-3.0.3 and Apache Commons Math 3.6.1 in your local
machine and extract cloudsim-3.0.3 and Apache Commons Math 3.6.1

Downloaded Files
Step 5: First of all, navigate to the folder where you have unzipped the eclipse folder and
open Eclipse.exe

Step 6: Now within Eclipse window navigate the menu: File -> New -> Project, to open the
new project wizard
Step 7: A ‗New Project‗ wizard should open. There are a number of options displayed and
you have to find & select the ‗Java Project‗ option, once done click ‘Next‗

Step 8: Now a detailed new project window will open, here you will provide the project name
and the path of CloudSim-master-code project source code, which will be done as follows:

Project Name: CloudSim

Step 9: Unselect the ‘Use default location’ option and then click on ‘Browse’ to open the path
where you have unzipped the Cloudsim-code-master project and finally click Next to set project
settings.

Step 10: Make sure you navigate the path till you can see the bin, docs, examplesetc folder in
the navigation plane.
Step 11: Once done finally, click ‗Next‘ to go to the next step i.e. setting up of project
settings

Step 12: Once the project is configured you can open the ‗Project Explorer‗ and start
exploring the Cloudsim project. Also for the first time eclipse automatically start building the
workspace for newly configured Cloudsim project, which may take some time depending on
the configuration of the computer system.

Following is the final screen which you will see after Cloudsim is configured.
Step 13: Now just to check you within the ‗Project Explorer‗, you should navigate to the
‗src‗ folder, then expand the package ‗default package‗ and double click to open the
‗RoundRobin.java‗.

Step 14: Now navigate to the Eclipse menu ‗Run ->Run‗ or directly use a keyboard shortcut
‘Ctrl + F11’ to execute the ‘RoundRobin.java‘. If it is successfully executed it should be
displaying the following type to output in the console window of the Eclipse IDE.
Execution of Scheduling Algorithms Using Cloud Simulator

Scheduling Algorithms (FirstCome First Service) using Cloud Sim

Step1. Extract cloud sim software and eclipse ide for java developers. (all software will be

available in your local drives)

Step2. Create a new project in eclipse using default settings.

Step 3. Add external jar files of cloud sim software.

Step 4. Create package name (com.sjfs)

Step 5. Create 5files under src folder

Example

After created the above file right click sfj_scheduler.java file and choose run as java
application.

Output
Ex.No 6 Transfer the files from one virtual machine to another
Virtual machine/host operating System

Study link : https://www.edureka.co/blog/windows-to-cloudera-demo/ (Transfer files from

Windows to Cloudera Demo VM)

Step1: Download and Install FileZilla

Step 2:
Ex.no 7a Install Hadoop single node cluster and execute Word
Count Program using Map reduce Concept

Study link: http://tdongsi.github.io/blog/2015/11/20/wordcount-sample-in-

cloudera-quickstart-vm/

File Creation

[cloudera@quickstart temp]$ echo “This is a hadoop tutorial test" > wordcount.txt

File Move To Hdfs

[cloudera@quickstart temp]$ hdfs dfs -mkdir /user/cloudera/input

[cloudera@quickstart temp]$ hdfs dfs -put /home/cloudera/temp/wordcount.txt

/user/cloudera/input

[cloudera@quickstart temp]$ hdfs dfs -ls /user/cloudera/input

Found 1 items
-rw-r--r-- 1 cloudera cloudera 31 2015-01-15 18:04
/user/cloudera/input/wordcount.txt

List The Files In Hdfs

[cloudera@quickstart temp]$ ls /user

ls: cannot access /user: No such file or directory

[cloudera@quickstart temp]$ hdfs dfs -ls /user

Found 5 items
drwxr-xr-x - cloudera cloudera
drwxr-xr-x - mapred hadoop
drwxrwxrwx - hive hive
drwxrwxrwx - oozie oozie
drwxr-xr-x - spark spark
0 2014-12-18 07:08 /user/cloudera
0 2014-12-18 07:08 /user/history
0 2014-12-18 07:08 /user/hive
0 2014-12-18 07:09 /user/oozie
0 2014-12-18 07:09 /user/spark

To see the content of a file on HDFS, use cat subcommand:

[cloudera@quickstart temp]$ hdfs dfs -cat /user/cloudera/input/wordcount.txt

this is a hadoop tutorial test
Running the WordCount example

[cloudera@quickstart temp]$ ls -ltr /usr/lib/hadoop-mapreduce/

lrwxrwxrwx 1 root root 44 Dec 18 07:01 hadoop-mapreduce-examples.jar ->
hadoop-mapreduce-examples-2.5.0-cdh5.3.0.jar

[cloudera@quickstart temp]$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-

mapreduce-examples.jar
Valid program names are:

To run the WordCount example using the input file that we just moved to
HDFS, use the following command:

[cloudera@quickstart temp]$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-

mapreduce-examples.jar wordcount
/user/cloudera/input/wordcount.txt /user/cloudera/output

output of WordCount example in the output folder.

[cloudera@quickstart temp]$ hdfs dfs -ls /user/cloudera/output

Found 2 items
-rw-r--r-- 1 cloudera cloudera 0 2015-11-15 18:15
/user/cloudera/output/_SUCCESS
-rw-r--r-- 1 cloudera cloudera 43 2015-11-15 18:15
/user/cloudera/output/part-r-00000

[cloudera@quickstart temp]$ hdfs dfs -cat /user/cloudera/output/part-r-00000

a 1
hadoop 1
is 1
test 1
this 1
tutorial 1
Simple execution method of wordcount program
create wordcount file
[cloudera@quickstart ~]$ echo "This is hadoop tutorial test" > wordcount.txt

Create input folder

[cloudera@quickstart ~]$ hdfs dfs -mkdir /user/cloudera/input

Move wordcount file to input folder

[cloudera@quickstart ~]$ hdfs dfs -put /home/cloudera/wordcount.txt
/user/cloudera/input

List the input folder files

[cloudera@quickstart ~]$ hdfs dfs -ls /user/cloudera/input
Found 1 items
-rw-r--r-- 1 cloudera cloudera 29 2021-03-04 22:56
/user/cloudera/input/wordcount.txt

List users folder

[cloudera@quickstart ~]$ hdfs dfs -ls /user
Found 8 items
drwxr-xr-x - cloudera cloudera 0 2021-03-04 22:55 /user/cloudera
drwxr-xr-x - mapred hadoop 0 2017-10-23 10:29 /user/history
drwxrwxrwx - hive supergroup 0 2017-10-23 10:31 /user/hive
drwxrwxrwx - hue supergroup 0 2017-10-23 10:30 /user/hue
drwxrwxrwx - jenkins supergroup 0 2017-10-23 10:30 /user/jenkins
drwxrwxrwx - oozie supergroup 0 2017-10-23 10:30 /user/oozie
drwxrwxrwx - root supergroup 0 2017-10-23 10:30 /user/root
drwxr-xr-x - hdfs supergroup 0 2017-10-23 10:31 /user/spark

file contents check

[cloudera@quickstart ~]$ hdfs dfs -cat /user/cloudera/input/wordcount.txt
This is hadoop tutorial test

Run the word count

[cloudera@quickstart ~]$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-
mapreduce-examples.jar wordcount /user/cloudera/input/wordcount.txt
/user/cloudera/output

Bytes Written=39

Output folder check

[cloudera@quickstart ~]$ hdfs dfs -ls /user/cloudera/output
Found 2 items
-rw-r--r-- 1 cloudera cloudera 0 2021-03-04 23:00
/user/cloudera/output/_SUCCESS
-rw-crl-o-u
r-d-era1 cloudera 39 2021-03-04 23:00
/user/cloudera/output/part-r-00000
Verify output
[cloudera@quickstart ~]$ hdfs dfs -cat /user/cloudera/output/part-r-00000
This 1
hadoop 1
is 1
test 1
tutorial 1
[cloudera@quickstart ~]$
 HADOOP BASIC COMMANDS

fsck
HDFS Command to check the health of the Hadoop file system.

Command: hdfs fsck /

//list the files under hadoop directory

[cloudera@quickstart ~]$ hdfs dfs -ls /

//make folder under hadoop directory

[cloudera@quickstart ~]$ hdfs dfs -mkdir /testfolder1

[cloudera@quickstart ~]$ hdfs dfs -ls /

Found 8 items
drwxrwxrwx - hdfs supergroup 0 2017-10-23 10:29 /benchmarks
drwxr-xr-x - hbase supergroup 0 2019-10-03 21:44 /hbase
drwxr-xr-x - solr solr 0 2017-10-23 10:32 /solr
drwxr-xr-x - cloudera supergroup 0 2019-10-03 22:03 /tamilfolder
drwxr-xr-x - cloudera supergroup 0 2019-10-03 22:09 /testfolder1
drwxrwxrwt - hdfs supergroup 0 2019-10-03 21:44 /tmp
drwxr-xr-x - hdfs supergroup 0 2017-10-23 10:31 /user
drwxr-xr-x - hdfs supergroup 0 2017-10-23 10:31 /var

//copy a file from local file system to hadoop file system

//before this command create a file called sample1.txt under
home/cloudera directory.

[cloudera@quickstart ~]$ hdfs dfs -copyFromLocal /home/cloudera/sample1.txt

/tamilfolder/
[cloudera@quickstart ~]$ hdfs dfs -ls /

//using put command move local files to hadoop file system

//befor these commands create students.txt files under /home/cloudera
folder.
[cloudera@quickstart ~]$ hdfs dfs -put /home/cloudera/students.txt /testfolder1/

[cloudera@quickstart ~]$ hdfs dfs -ls /testfolder1

Found 1 items
-rw-r--r-- 1 cloudera supergroup 56 2019-10-03 22:16
/testfolder1/students.txt

//copy a file from hdfs to localfile system

[cloudera@quickstart ~]$ hdfs dfs -get /testfolder1/students.txt
/home/cloudera/Downloads
[cloudera@quickstart ~]$ ls

cloudera-manager Downloads kerberos Pictures sample1

Templates
cm_api.py eclipse lib Public sample1~ Videos
Desktop enterprise-deployment.json Music queries.txt
studentsdetails.txt~ workspace
Documents express-deployment.json parcels queries.txt~ students.txt

//check local file system files

[cloudera@quickstart ~]$ ls Downloads
students.txt

//count the no of files under hadoop directory

[cloudera@quickstart ~]$ hdfs dfs -count /user
28 921 861282820 /user

//count no of files under testfolder1 directory

[cloudera@quickstart ~]$ hdfs dfs -count /testfolder1
1 1 56 /testfolder1

//remove the file from specific location

[cloudera@quickstart ~]$ hdfs dfs -rm /testfolder1/students.txt
Deleted /testfolder1/students.txt

//remove testfolder1 and its contents also

[cloudera@quickstart ~]$ hdfs dfs -rm -r /testfolder1
Deleted /testfolder1

[cloudera@quickstart ~]$ hdfs dfs -ls /

Found 7 items
drwxrwxrwx - hdfs supergroup 0 2017-10-23 10:29 /benchmarks
drwxr-xr-x - hbase supergroup 0 2019-10-03 21:44 /hbase
drwxr-xr-x - solr solr 0 2017-10-23 10:32 /solr
drwxr-xr-x - cloudera supergroup 0 2019-10-03 22:03 /tamilfolder
drwxrwxrwt - hdfs supergroup 0 2019-10-03 21:44 /tmp
drwxr-xr-x - hdfs supergroup 0 2017-10-23 10:31 /user
drwxr-xr-x - hdfs supergroup 0 2017-10-23 10:31 /var

//copy command in hdfs

//create twofolder testfolder1 and tamilfolder

//create one files in each directory
// finally copy a file from testfolder to tamilfolder
[cloudera@quickstart ~]$ hdfs dfs -mkdir /testfolder1
[cloudera@quickstart ~]$ hdfs dfs -ls /
Found 8 items
drwxrwxrwx - hdfs supergroup 0 2017-10-23 10:29 /benchmarks
drwxr-xr-x - hbase supergroup 0 2019-10-03 21:44 /hbase
drwxr-xr-x - solr solr 0 2017-10-23 10:32 /solr
drwxr-xr-x - cloudera supergroup 0 2019-10-03 22:03 /tamilfolder
drwxr-xr-x - cloudera supergroup 0 2019-10-03 22:53 /testfolder1
drwxrwxrwt - hdfs supergroup 0 2019-10-03 21:44 /tmp
drwxr-xr-x - hdfs supergroup 0 2017-10-23 10:31 /user
drwxr-xr-x - hdfs supergroup 0 2017-10-23 10:31 /var

[cloudera@quickstart ~]$ hdfs dfs -put /home/cloudera/students.txt /testfolder1/

[cloudera@quickstart ~]$ hdfs dfs -ls /

Found 8 items
drwxrwxrwx - hdfs supergroup 0 2017-10-23 10:29 /benchmarks
drwxr-xr-x - hbase supergroup 0 2019-10-03 21:44 /hbase
drwxr-xr-x - solr solr 0 2017-10-23 10:32 /solr
drwxr-xr-x - cloudera supergroup 0 2019-10-03 22:03 /tamilfolder
drwxr-xr-x - cloudera supergroup 0 2019-10-03 22:53 /testfolder1
drwxrwxrwt - hdfs supergroup 0 2019-10-03 21:44 /tmp
drwxr-xr-x - hdfs supergroup 0 2017-10-23 10:31 /user
drwxr-xr-x - hdfs supergroup 0 2017-10-23 10:31 /var

[cloudera@quickstart ~]$ hdfs dfs -ls /tamilfolder

Found 1 items
-rw-r--r-- 1 cloudera supergroup 35 2019-10-03 22:03 /tamilfolder/sample1

[cloudera@quickstart ~]$ hdfs dfs -ls /testfolder1

Found 1 items
-rw-r--r-- 1 cloudera supergroup 56 2019-10-03 22:53
/testfolder1/students.txt

[cloudera@quickstart ~]$ hdfs dfs -cp /testfolder1/students.txt /tamilfolder/

[cloudera@quickstart ~]$ hdfs dfs -ls /tamilfolder

Found 2 items
-rw-r--r-- 1 cloudera supergroup 35 2019-10-03 22:03 /tamilfolder/sample1
-rw-r--r-- 1 cloudera supergroup 56 2019-10-03 22:55
/tamilfolder/students.txt

//remove folder using rmdir methods

[cloudera@quickstart ~]$ hdfs dfs -mkdir /test1

[cloudera@quickstart ~]$ hdfs dfs -ls /

Found 9 items
drwxrwxrwx - hdfs supergroup 0 2017-10-23 10:29 /benchmarks
drwxr-xr-x - hbase supergroup 0 2019-10-03 21:44 /hbase
drwxr-xr-x - solr solr 0 2017-10-23 10:32 /solr
drwxr-xr-x - cloudera supergroup 0 2019-10-03 22:55 /tamilfolder
drwxr-xr-x - cloudera supergroup 0 2019-10-03 22:58 /test1
drwxr-xr-x - cloudera supergroup 0 2019-10-03 22:53 /testfolder1
drwxrwxrwt - hdfs supergroup 0 2019-10-03 21:44 /tmp
drwxr-xr-x - hdfs supergroup 0 2017-10-23 10:31 /user
drwxr-xr-x - hdfs supergroup 0 2017-10-23 10:31 /var

[cloudera@quickstart ~]$ hdfs dfs -rmdir /test1

[cloudera@quickstart ~]$ hdfs dfs -ls /

Found 8 items
drwxrwxrwx - hdfs supergroup
drwxr-xr-x - hbase supergroup
drwxr-xr-x - solr solr
drwxr-xr-x - cloudera supergroup
drwxr-xr-x - cloudera supergroup
drwxrwxrwt - hdfs supergroup
drwxr-xr-x - hdfs supergroup
drwxr-xr-x - hdfs supergroup
0 2017-10-23 10:29 /benchmarks
0 2019-10-03 21:44 /hbase
0 2017-10-23 10:32 /solr
0 2019-10-03 22:55 /tamilfolder
0 2019-10-03 22:53 /testfolder1
0 2019-10-03 21:44 /tmp
0 2017-10-23 10:31 /user
0 2017-10-23 10:31 /var

//help command for particular command

[cloudera@quickstart ~]$ hdfs dfs -usage mkdir
Usage: hadoop fs [generic options] -mkdir [-p] <path> ...
EX.NO 7b BASIC HADOOP INSTALLATION AND COMMANDS

Install Hadoop
Step 1: Click here to download the Java 8 Package. Save this file in your
home directory.

Step 2: Extract the Java Tar File.

Command: tar -xvf jdk-8u101-linux-i586.tar.gz

Fig: Hadoop Installation – Extracting Java Files

Step 3: Download the Hadoop 2.7.3 Package.

Command: wget https://archive.apache.org/dist/hadoop/core/hadoop-2.7.3/hadoop-

2.7.3.tar.gz

Fig: Hadoop Installation – Downloading Hadoop

Step 4: Extract the Hadoop tar File.

Command: tar -xvf hadoop-2.7.3.tar.gz

Fig: Hadoop Installation – Extracting Hadoop Files

Step 5: Add the Hadoop and Java paths in the bash file (.bashrc).

Open. bashrc file. Now, add Hadoop and Java Path as shown below.

Command: vi .bashrc
 Fig: Hadoop Installation – Setting Environment Variable

Then, save the bash file and close it.

For applying all these changes to the current Terminal, execute the source command.

Command: source .bashrc

Fig: Hadoop Installation – Refreshing environment variables

To make sure that Java and Hadoop have been properly installed on your system
and can be accessed through the Terminal, execute the java -version and hadoop
version commands.

Command: java -version

 Fig: Hadoop Installation – Checking Java Version

Command: hadoop version

Fig: Hadoop Installation – Checking Hadoop Version

Step 6: Edit the Hadoop Configuration files.

Command: cd hadoop-2.7.3/etc/hadoop/

Command: ls

All the Hadoop configuration files are located in hadoop-2.7.3/etc/hadoop directory as

you can see in the snapshot below:
 Fig: Hadoop Installation – Hadoop Configuration Files

Step 7: Open core-site.xml and edit the property mentioned below inside
configuration tag:

core-site.xml informs Hadoop daemon where NameNode runs in the cluster. It contains
configuration settings of Hadoop core such as I/O settings that are common to HDFS &
MapReduce.

Command: vi core-site.xml

Fig: Hadoop Installation – Configuring core-site.xml

1
<?xml version="1.0" encoding="UTF-8"?>
2 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
3 <configuration>
4 <property>
5 <name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
6
</property>
7 </configuration>
8
Step 8: Edit hdfs-site.xml and edit the property mentioned below inside
configuration tag:
hdfs-site.xml contains configuration settings of HDFS daemons (i.e. NameNode,
DataNode, Secondary NameNode). It also includes the replication factor and block size
of HDFS.

Command: vi hdfs-site.xml

Fig: Hadoop Installation – Configuring hdfs-site.xml

1
2 <?xml version="1.0" encoding="UTF-8"?>
3 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
4 <property>
5 <name>dfs.replication</name>
6 <value>1</value>
7 </property>
<property>
8 <name>dfs.permission</name>
9 <value>false</value>
10 </property>
11 </configuration>
12
Step 9: Edit the mapred-site.xml file and edit the property mentioned below
inside configuration tag:

mapred-site.xml contains configuration settings of MapReduce application like number

of JVM that can run in parallel, the size of the mapper and the reducer process, CPU
cores available for a process, etc.

In some cases, mapred-site.xml file is not available. So, we have to create the mapred-
site.xml file using mapred-site.xml template.

Command: cp mapred-site.xml.template mapred-site.xml

Command: vi mapred-site.xml.

Fig: Hadoop Installation – Configuring mapred-site.xml

1
<?xml version="1.0" encoding="UTF-8"?>
2 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
3 <configuration>
4 <property>
5 <name>mapreduce.framework.name</name>
<value>yarn</value>
6
</property>
7 </configuration>
8

Step 10: Edit yarn-site.xml and edit the property mentioned below inside
configuration tag:

yarn-site.xml contains configuration settings of ResourceManager and NodeManager

like application memory management size, the operation needed on program &
algorithm, etc.

Command: vi yarn-site.xml
 Fig: Hadoop Installation – Configuring yarn-site.xml

1
2 <?xml version="1.0">
3 <configuration>
4 <property>
5 <name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
6
</property>
7 <property>
8 <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</
9 name>
1 <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
0 </configuration>
1
1
Step 11: Edit hadoop-env.sh and add the Java Path as mentioned below:

hadoop-env.sh contains the environment variables that are used in the script to run
Hadoop like Java home path, etc.

Command: vi hadoop–env.sh

Fig: Hadoop Installation – Configuring hadoop-env.sh

Step 12: Go to Hadoop home directory and format the NameNode.

Command: cd
Command: cd hadoop-2.7.3

Command: bin/hadoop namenode -format

Fig: Hadoop Installation – Formatting NameNode

This formats the HDFS via NameNode. This command is only executed for the first
time. Formatting the file system means initializing the directory specified by the
dfs.name.dir variable.

Never format, up and running Hadoop filesystem. You will lose all your data stored in
the HDFS.

Step 13: Once the NameNode is formatted, go to hadoop-2.7.3/sbin directory

and start all the daemons.

Command: cd hadoop-2.7.3/sbin

Either you can start all daemons with a single command or do it individually.

Command: ./start-all.sh

The above command is a combination of start-dfs.sh, start-yarn.sh & mr-jobhistory-

daemon.sh

Or you can run all the services individually as below:

Start NameNode:

The NameNode is the centerpiece of an HDFS file system. It keeps the directory tree of
all files stored in the HDFS and tracks all the file stored across the cluster.

Command: ./hadoop-daemon.sh start namenode

 Fig: Hadoop Installation – Starting NameNode

Start DataNode:

On startup, a DataNode connects to the Namenode and it responds to the requests

from the Namenode for different operations.

Command: ./hadoop-daemon.sh start datanode

Fig: Hadoop Installation – Starting DataNode

Start ResourceManager:

ResourceManager is the master that arbitrates all the available cluster resources and
thus helps in managing the distributed applications running on the YARN system.
Its work is to manage each NodeManagers and the each application’s
ApplicationMaster.

Command: ./yarn-daemon.sh start resourcemanager

//Hadoop - wordcount program in java
import java.io.IOException;

import java.util.*;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.conf.*;

import org.apache.hadoop.io.*;

import org.apache.hadoop.mapreduce.*;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

public class WordCount {

public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> {

private final static IntWritable one = new IntWritable(1);

private Text word = new Text();

public void map(LongWritable key, Text value, Context context) throws IOException,
InterruptedException {

String line = value.toString();

StringTokenizer tokenizer = new StringTokenizer(line);

while (tokenizer.hasMoreTokens()) {

word.set(tokenizer.nextToken());

context.write(word, one);

} } }

public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> {

public void reduce(Text key, Iterable<IntWritable> values, Context context)

throws IOException, InterruptedException {

int sum = 0;

for (IntWritable val : values) {

sum += val.get();

context.write(key, new IntWritable(sum));

public static void main(String[] args) throws Exception {

Configuration conf = new Configuration();

//@SuppressWarnings(“deprecation”)

Job job = new Job(conf,"wordcount");

job.setJarByClass(WordCount.class);

job.setOutputKeyClass(Text.class);

job.setOutputValueClass(IntWritable.class);

job.setMapperClass(Map.class);

job.setReducerClass(Reduce.class);

job.setInputFormatClass(TextInputFormat.class);

job.setOutputFormatClass(TextOutputFormat.class);

FileInputFormat.addInputPath(job, new Path(args[0]));

FileOutputFormat.setOutputPath(job, new Path(args[1]));

job.waitForCompletion(true);
}
}
Ex No.8 Find a procedure to launch virtual machine using Openstack
Introduction:
❖ OpenStack was introduced by Rackspace and NASA in July 2010.
❖ OpenStack is an Infrastructure as a Service known as Cloud Operating System, that
take resources such as Compute, Storage, Network and Virtualization Technologies and
control those resources at a data center level
❖ The project is building an open source community - to share resources and technologies
with the goal of creating a massively scalable and secure cloud infrastructure.
❖ The software is open source and limited to just open source APIs such as Amazon.

The following figure shows the OpenStack architecture

OpenStack architecture
• It is modular architecture
• Designed to easily scale out
• Based on (growing) set of core services
The major components are
1. Keystone
2. Nova
3. Glance
4. Swift
5. Quantum
6. Cinder
 I

• KEYSTONE :
o Identity service
o Common authorization framework
o Manage users, tenants and roles
o Pluggable backends (SQL,PAM,LDAP, IDM etc)

• NOVA
o Core compute service comprised of
▪ Compute Nodes – hypervisors that run virtual machines
• Supports multiple hypervisors KVM,Xen,LXC,Hyper-V
and ESX
▪ Distributed controllers that handle scheduling, API calls, etc
• Native OpenStack API and Amazon EC2 compatible
API
• GLANCE
o Image service
o Stores and retrieves disk images (Virtual machine templates)
o Supports RAW,QCOW,VHD,ISO,OVF & AMI/AKI
o Backend Storage : File System, Swift, Gluster, Amazon S3
• SWIFT
o Object Storage service
o Modeled after Amazon‘s Service
o Provides simple service for storing and retrieving arbitrary data
o Native API and S3 compatible API
• NEUTRON
o Network service
o Provides framework for Software Defined Network
o Plugin architecture
▪ Allows intergration of hardware and software based network
solutions
• Open vSwitch, Cisco UCS,Standard Linux
Bridge,NiCira NVP
 • CINDER
o Block Storage (Volume) service
o Provides block storage for Virtual machines(persistent disks)
o Similar to Amazon EBS service
o Plugin architecture for vendor extensions
▪ NetApp driver for cinder
• HORIZON
o Dashboard
o Provides simple self service UI for end-users
o Basic cloud administrator functions
▪ Define users, tenants and quotas
▪ No infrastructure management
• HEAT OpenStack Orchestration
o Provides template driven cloud application orchestration
o Modeled after AWS Cloud Formation
o Targeted to provide advanced functionality such as high availability
and auto scaling
o Introduced by Redhat
• CEILOMETER – OpenStack Monitoring and Metering
o Goal: To Provide a single infrastructure to collect measurements from
an entire OpenStack Infrastructure; Eliminate need for multiple agents
attaching to multiple OpenStack Projects
o Primary targets metering and monitoring: Provided extensibility

❖ Steps in Installing Openstack

Step 1:
• Download and Install Oracle Virtual Box latest version & Extension
package
o https://virtualbox.org/wiki/downloads

Step 2:

• Download CentOS 7 OVA(Open Virtual Appliance) from

o Link : https://linuxvmimages.com/images/centos-7
• Import CentOS 7 OVA(Open Virtual Appliance) into Oracle Virtual Box
EX.NO 8 OPEN STACK ONLINE DEMO FOR VIRTUALIZATION

OpenStack is an open-source software cloud computing platform. OpenStack is primarily used

for deploying an infrastructure as a service (IaaS) solution like Amazon Web Service (AWS). In
other words, you can make your own AWS by using OpenStack. If you want to try out
OpenStack, TryStack is the easiest and free way to do it.
In order to try OpenStack in TryStack, you must register yourself by joining TryStack Facebook
Group. The acceptance of group needs a couple days because it’s approved manually. After you
have been accepted in the TryStack Group, you can log in TryStack.

TryStack.org Homepage
I assume that you already join to the Facebook Group and login to the dashboard. After you log
in to the TryStack, you will see the Compute Dashboard like:
OpenStack Compute Dashboard

Overview: What we will do?

In this post, I will show you how to run an OpenStack instance. The instance will be accessible
through the internet (have a public IP address). The final topology will like:
 Network topology
As you see from the image above, the instance will be connected to a local network and the local
network will be connected to internet.

Step 1: Create Network

Network? Yes, the network in here is our own local network. So, your instances will be not
mixed up with the others. You can imagine this as your own LAN (Local Area Network) in the
cloud.

1. Go to Network > Networks and then click Create Network.

2. In Network tab, fill Network Name for example internal and then click Next.
3. In Subnet tab,
1. Fill Network Address with appropriate CIDR, for example 192.168.1.0/24. Use private
network CIDR block as the best practice.
2. Select IP Version with appropriate IP version, in this case IPv4.
3. Click Next.
4. In Subnet Details tab, fill DNS Name Servers with 8.8.8.8 (Google DNS) and then
click Create.

Step 2: Create Instance

 Now, we will create an instance. The instance is a virtual machine in the cloud, like AWS EC2.
You need the instance to connect to the network that we just created in the previous step.

1. Go to Compute > Instances and then click Launch Instance.

2. In Details tab,
1. Fill Instance Name, for example Ubuntu 1.
2. Select Flavor, for example m1.medium.
3. Fill Instance Count with 1.
4. Select Instance Boot Source with Boot from Image.
5. Select Image Name with Ubuntu 14.04 amd64 (243.7 MB) if you want install Ubuntu
14.04 in your virtual machine.
3. In Access & Security tab,
1. Click [+] button of Key Pair to import key pair. This key pair is a public and private key that
we will use to connect to the instance from our machine.
2. In Import Key Pair dialog,
1. Fill Key Pair Name with your machine name (for example Edward-Key).
2. Fill Public Key with your SSH public key (usually is in ~/.ssh/id_rsa.pub). See
description in Import Key Pair dialog box for more information. If you are using
Windows, you can use Puttygen to generate key pair.
3. Click Import key pair.
3. In Security Groups, mark/check default.
4. In Networking tab,
1. In Selected Networks, select network that have been created in Step 1, for example internal.
5. Click Launch.
6. If you want to create multiple instances, you can repeat step 1-5. I created one more instance
with instance name Ubuntu 2.

Step 3: Create Router

I guess you already know what router is. In the step 1, we created our network, but it is isolated.
It doesn’t connect to the internet. To make our network has an internet connection, we need a
router that running as the gateway to the internet.

1. Go to Network > Routers and then click Create Router.

2. Fill Router Name for example router1 and then click Create router.
3. Click on your router name link, for example router1, Router Details page.
4. Click Set Gateway button in upper right:
1. Select External networks with external.
2. Then OK.
5. Click Add Interface button.
1. Select Subnet with the network that you have been created in Step 1.
2. Click Add interface.
6. Go to Network > Network Topology. You will see the network topology. In the example, there
are two network, i.e. external and internal, those are bridged by a router. There are
instances those are joined to internal network.

Step 4: Configure Floating IP Address

 Floating IP address is public IP address. It makes your instance is accessible from the internet.
When you launch your instance, the instance will have a private network IP, but no public IP. In
OpenStack, the public IPs is collected in a pool and managed by admin (in our case is TryStack).
You need to request a public (floating) IP address to be assigned to your instance.
1. Go to Compute > Instance.
2. In one of your instances, click More > Associate Floating IP.
3. In IP Address, click Plus [+].
4. Select Pool to external and then click Allocate IP.
5. Click Associate.
6. Now you will get a public IP, e.g. 8.21.28.120, for your instance.

Step 5: Configure Access & Security

OpenStack has a feature like a firewall. It can whitelist/blacklist your in/out connection. It is
called Security Group.
1. Go to Compute > Access & Security and then open Security Groups tab.
2. In default row, click Manage Rules.
3. Click Add Rule, choose ALL ICMP rule to enable ping into your instance, and then click Add.
4. Click Add Rule, choose HTTP rule to open HTTP port (port 80), and then click Add.
5. Click Add Rule, choose SSH rule to open SSH port (port 22), and then click Add.
6. You can open other ports by creating new rules.

Step 6: SSH to Your Instance

Now, you can SSH your instances to the floating IP address that you got in the step 4. If you are
using Ubuntu image, the SSH user will be ubuntu.
Ex.No 6 Transfer Files From Virtual Machine1 To Another Virtual Machine2
Step1: Create two virtual machine using virtual box .

Step2 : click settings icon to change network bridge adapter

Step3: change network settings inot bridged adapter for both virtual machines (vm1,vm2)

Step 4: boot both virtual machines

Step5: open vm2 – terminal window and type ifconfig command to view the ipaddress of vm2

Step6: open vm1 terminal window

Type the following command to transfer a file from one virtual machine1 to virtual machine

scp source file vm2userid@vmipaddress: destination folder name

Example: scp demo.txt ksriet@192.168.0.245:/home/ksriet/Desktop

vm1 source machine filename
Destination vm2 userid ipaddr destinationfolder

Ouput: demo.txt file from vm1 is transferred successfully to vm2

Ex.no3 Python Execution in Google Cloud App Engine(GAE)

Step1 : login into any gmail account and continue further steps for authentication purpose

Open console.cloud.google.com by using browser

Step2: click dash board button

Choose Dash board for Google Cloud sdk

Step 3: Choose APIS and Services menu then choose library sub menu

Step4 :Then search appengine

Step 5: Choose app engine admin API

Step6 :Click enable button

After clicking the cloud shell icon the following screen will open like the below image

The following screen will get open

Step7: Type this command:

git clone https://github.com/GoogleCloudPlatform/python-docs-samples.git in cloud shell

window

Type ls command

Enter into python-docs-samples folder by using the following command : cd python-docs-samples

Enter ls command for the python-docs-samples folder: ls

Enter into appengine folder by using command cd appengine : cd appengine

Enter ls command to view the files under appengine folder : ls

Enter into standard_pyhon3 folder by using command : cd standard_python3

Enter ls command to view files under standard_python3 folder : ls

Enter into hello_world folder by using the command : cd hello_world

Enter ls command to view files under hello _world folder

Run the main.py file using the command : python3 main.py

Output