Professional Documents
Culture Documents
Textbook Heterogeneity High Performance Computing Self Organization and The Cloud Theo Lynn Ebook All Chapter PDF
Textbook Heterogeneity High Performance Computing Self Organization and The Cloud Theo Lynn Ebook All Chapter PDF
https://textbookfull.com/product/biota-grow-2c-gather-2c-cook-
loucas/
https://textbookfull.com/product/measuring-the-business-value-of-
cloud-computing-theo-lynn/
https://textbookfull.com/product/managing-distributed-cloud-
applications-and-infrastructure-a-self-optimising-approach-theo-
lynn/
https://textbookfull.com/product/the-cloud-to-thing-continuum-
opportunities-and-challenges-in-cloud-fog-and-edge-computing-
theo-lynn/
Data Privacy And Trust In Cloud Computing: Building
Trust In The Cloud Through Assurance And Accountability
Theo Lynn
https://textbookfull.com/product/data-privacy-and-trust-in-cloud-
computing-building-trust-in-the-cloud-through-assurance-and-
accountability-theo-lynn/
https://textbookfull.com/product/determinism-and-self-
organization-of-human-perception-and-performance-till-frank/
https://textbookfull.com/product/advances-in-high-performance-
computing-results-of-the-international-conference-on-high-
performance-computing-borovets-bulgaria-2019-ivan-dimov/
https://textbookfull.com/product/high-performance-computing-in-
science-and-engineering-18-transactions-of-the-high-performance-
computing-center-stuttgart-hlrs-2018-wolfgang-e-nagel/
HETEROGENEITY,
HIGH PERFORMANCE
COMPUTING,
SELF-ORGANIZATION
AND THE CLOUD
Edited by
Theo Lynn, John P. Morrison
and David Kenny
Palgrave Studies in Digital Business & Enabling
Technologies
Series Editor
Theo Lynn
Irish Centre for Cloud Computing (IC4)
Dublin City University
Dublin, Ireland
This multi-disciplinary series will provide a comprehensive and coherent
account of cloud computing, social media, mobile, big data, and other
enabling technologies that are transforming how society operates and how
people interact with each other. Each publication in the series will focus on
a discrete but critical topic within business and computer science, covering
existing research alongside cutting edge ideas. Volumes will be written by
field experts on topics such as cloud migration, measuring the business
value of the cloud, trust and data protection, fintech, and the Internet of
Things. Each book has global reach and is relevant to faculty, researchers
and students in digital business and computer science with an interest in
the decisions and enabling technologies shaping society.
Heterogeneity, High
Performance
Computing, Self-
Organization and the
Cloud
Editors
Theo Lynn John P. Morrison
Irish Centre for Cloud Computing Department of Computer Science
(IC4) University College Cork
Dublin City University Cork, Ireland
Dublin, Ireland
David Kenny
Dublin City University
Dublin, Dublin, Ireland
© The Editor(s) (if applicable) and The Author(s) 2018 This book is an open access publication.
Open Access This book is licensed under the terms of the Creative Commons Attribution-
NonCommercial- NoDerivatives 4.0 International License (http://creativecommons.org/
licenses/by-nc-nd/4.0/), which permits any noncommercial use, sharing, distribution and repro-
duction in any medium or format, as long as you give appropriate credit to the original author(s)
and the source, provide a link to the Creative Commons license and indicate if you modified the
licensed material. You do not have permission under this license to share adapted material derived
from this book or parts of it.
The images or other third party material in this book are included in the book’s Creative Commons
license, unless indicated otherwise in a credit line to the material. If material is not included in the
book’s Creative Commons license and your intended use is not permitted by statutory regulation
or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
This work is subject to copyright. All commercial rights are reserved by the author(s), whether the
whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of
illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and
transmission or information storage and retrieval, electronic adaptation, computer software, or by
similar or dissimilar methodology now known or hereafter developed. Regarding these commercial
rights a non-exclusive license has been granted to the publisher.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publication does not imply, even in the absence of a specific statement, that such names are exempt
from the relevant protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authors or the editors give a warranty, express or implied, with respect to the material contained
herein or for any errors or omissions that may have been made. The publisher remains neutral with
regard to jurisdictional claims in published maps and institutional affiliations.
This Palgrave Macmillan imprint is published by the registered company Springer International
Publishing AG part of Springer Nature.
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
Preface
This is the first book in the series, “Advances in Digital Business and
Enabling Technologies”, which aims to contribute to multi-disciplinary
research on digital business and enabling technologies, such as cloud com-
puting, social media, big data analytics, mobile technologies, and the
Internet of Things, in Europe. This first volume focuses on research that
extends conventional thinking on cloud computing architecture design to
greater support High Performance Computing (HPC). Meeting the needs
of HPC users provides significant challenges to cloud service providers,
both technically and culturally, and this book provides a novel approach
and indicates a future direction for cloud computing architecture research
that may address a significant portion of these challenges. Given the sig-
nificant role that HPC plays in scientific advancement and the increasing
dominance of cloud computing as a global enterprise computing para-
digm, this book has value to university educators and researchers, indus-
try, and policy makers.
The content of the book is based on contributions from researchers on
the CloudLightning project, a European Union project funded under
Horizon 2020 (www.cloudlightning.eu). CloudLightning commenced in
2015 and brought together eight project partners from five countries
across Europe to create a new way to provision heterogeneous cloud
resources to deliver services, specified by the user, using a bespoke service
description language. The goal of CloudLightning is to address energy
inefficiencies, particularly in the use of resources, and consequently to
deliver savings to the cloud service provider and the cloud consumer in
v
vi PREFACE
This book was partially funded by the European Union’s Horizon 2020
Research and Innovation Programme through the CloudLightning proj-
ect (http://www.cloudlightning.eu) under Grant Agreement Number
643946, and by the Irish Centre for Cloud Computing and Commerce,
an Enterprise Ireland and IDA funded technology centre.
vii
Contents
ix
x Contents
6 Concluding Remarks 151
Theo Lynn and John P. Morrison
Index 157
Notes on Contributors
xi
xii Notes on Contributors
3D Three dimensional
AMD Advanced Micro Devices
API Application Programming Interface
ARM Advanced Reduced Instruction Set Computing Machine
AWS Amazon Web Services
BDaaS Big Data as as Service
BIOS Basic Input/Output System
CAMP Cloud Application Management for Platforms
CL CloudLightning
CL-SDL CloudLightning Service Description Language
CM Cell Manager
CPU Central Processing Unit
CRM Customer Relationship Management
CSAR Cloud Service Archive
CSP Cloud Service Provider
DDR Double Data Rate
DES Discrete Event Simulators
DFE Data Flow Engine
DNA Deoxyribonucleic Acid
DSP Digital Signal Processor
DUNE Distributed and Unified Numeric Environment
EAD Enterprise Application Developer
EAO Enterprise Application Operator
ERP Enterprise Resource Planning
FLOPs Floating Point Operations per Second
FPGA Field-Programmable Gate Array
GDP Gross Domestic Product
xv
xvi LIST OF ABBREVIATIONS
SI Suitability Index
SLA Service-Level Agreement
SOSM Self-Organisation Self-Management
SPEC Standard Performance Evaluation Corporation
SSL Secure Sockets Layer
ToR Top-of-Rack
TOSCA Topology and Orchestration Specification for Cloud Applications
UI User Interface
VM Virtual Machine
VPN Virtual Private Networks
vRM Virtual Rack Manager
WSC Warehouse Scale Computer
YAML Yet Another Markup Language
List of Figures
xix
xx List of Figures
xxi
CHAPTER 1
Theo Lynn
T. Lynn (*)
Irish Centre for Cloud Computing (IC4), Dublin City University,
Dublin, Ireland
e-mail: theo.lynn@dcu.ie
Three illustrative application scenarios for HPC in the cloud—(i) oil and
gas exploration, (ii) ray tracing, and (iii) genomics—are discussed.
1.1 Introduction
The objective of this book is to introduce readers to CloudLightning, an
architectural innovation in cloud computing based on the concepts of self-
organisation, self-management, and separation of concerns, showing how
it can be used to support high performance computing (HPC) in the
cloud at hyperscale. The remainder of this chapter provides a brief over-
view of cloud computing and HPC, and the challenges of using the cloud
for HPC workloads. This book introduces some of the major design con-
cepts informing the CloudLightning architectural design and discusses
three challenging HPC applications—(i) oil and gas exploration, (ii) ray
tracing, and (iii) genomics.
Table 1.1 Cloud computing essential characteristics, service models, and deploy-
ment models (adapted from Mell and Grance 2011)
Essential characteristics
On-demand Consumers can unilaterally provision computing capabilities as needed
self-service automatically without requiring human interaction with the cloud
provider.
Broad network Capabilities are available over the network and accessed through
access standard mechanisms that promote use by heterogeneous thin or thick
client platforms and interfaces (e.g. devices).
Resource The provider’s computing resources are pooled to serve multiple
pooling consumers using a multi-tenant model, with different physical and
virtual resources dynamically assigned and reassigned according to
consumer demand.
Rapid elasticity Capabilities can be elastically provisioned and released, in some cases
automatically, to scale rapidly outwards and inwards to meet demand.
To the consumer, the capabilities available for provisioning often appear
to be unlimited and can be appropriated in any quantity at any time.
Measured Cloud systems automatically control and optimise resource use by
service leveraging a metering capability at some level of abstraction appropriate
to the type of service. Resource usage can be monitored, controlled,
and reported, providing transparency to the service provider and the
consumer.
Service models
Software as a The capability provided to a consumer to use a provider’s applications
Service running on a cloud infrastructure and accessible by client interface.
Platform as a The capability provided to a consumer to deploy onto the cloud
Service infrastructure consumer-created or acquired applications created using
development technologies provided by the provider.
Infrastructure The capability provided to a consumer to provision computing
as a Service resources to deploy and run arbitrary software such as operating systems
and applications.
Deployment models
Private Cloud The cloud infrastructure is provisioned for exclusive use by a single
organisation comprising multiple consumers. Ownership, management,
and operation of the infrastructure may be done by one or more of the
organisations in the community, by a third party, or a combination of
both, and it may exist on or off premise.
Community The cloud infrastructure is provisioned for exclusive use by a specific
Cloud community of consumers from organisations that have shared concerns.
Ownership, management, and operation of the infrastructure may be
done by one or more of the organisations in the community, by a third
party, or a combination of both, and it may exist on or off premise.
(continued)
4 T. LYNN
Public Cloud The cloud infrastructure is provisioned for open use by the general
public. It may be owned, managed, and operated by a business,
academic, or government organisation, or some combination of them.
It exists on the premises of the cloud provider.
Hybrid Cloud The cloud infrastructure is a composition of two or more distinct cloud
infrastructures (private, community, or public) that remain unique
entities, but are bound together by standardised or proprietary
technology that enables data and application portability.
Despite the benefits of HPC, widespread use of HPC has been ham-
pered by the significant upfront investment and indirect operational
expenditure associated with running and maintaining HPC infrastruc-
tures. The larger supercomputer installations require an investment of up
to US$1 billion to operate and maintain. As discussed, performance is the
overriding concern for HPC users. HPC machines consume a substantial
amount of energy directly and indirectly to cool the processors.
Unsurprisingly, heat density and energy efficiency remain a major issue
and has a direct dependence on processor type. Increasingly, the HPC
community is focusing beyond mere performance to performance per
watt. This is particularly evident in the Green500 ranking of supercom-
puters.1 Cursory analysis of the most energy efficient supercomputers sug-
gests that the use of new technologies such as Graphical Processing Units
(GPUs) results in significant energy efficiencies (Feldman 2016). Other
barriers to greater HPC use include recruitment and retention of suitably
qualified HPC staff. HPC applications often require configuration and
optimisation to run on specialised infrastructure; thus, staff are required
not only to maintain the infrastructure but to optimise software for a spe-
cific domain area or use case.