Introduction To Egee and Glite Middleware: Enabling Grids For E-Science

Enabling Grids for E-sciencE
Introduction to EGEE and gLite

middleware
Giuseppe Andronico
April 18, 2009 -- Taipei

INFN Sez. CT
www.eu-egee.org
EGEE-II INFSO-RI-031688
Contents
• EGEE status, history and perspectives

• Grid services: gLite 3.1
– Introduction to basic systems
– Main components
– Component interaction
• Sources of further information
EGEE-II INFSO-RI-031688 2
• Quality:
– Monitoring via Nagios - distributed via official releases, configured
through YAIM, integrated with other tools
– Gradual implementation of Service Level Agreement with sites
– Result: 85% of sites are now above the 75% availability threshold
• Geographical expansion:
– now have production sites all across Asia: Australia, China, India,
Japan, Korea, Malaysia, Pakistan, Taiwan, Thailand
 In certification: Indonesia, Philippines, Vietnam
~280 sites
45 countries
>80,000 CPUs
>20 PetaBytes
>14,000 users
>250,000 jobs/day
Production & Volunteer grids
Applications database
• 61 registered applications so far

• Joint initiative between EGEE and SEE-GRID-SCI
• http://appdb.eu-egee.org/ & http://grid.ct.infn.it/egee_applications/
Readiness status:
7 HandledByRS
6 In production
5 Ready for deployment
4 Ready for validation
3 Ready for portal interface
2 Ready for gLite middleware
1 Ready for standalone use / running in a local cluster
EGEE→EGI transition planning
Grid Security
Users in many locations and organisations
Access services (User Interface) :

logon, upload credentials, run m/w commands
Grid Services
Built on Grid Security Infrastructure (GSI)
Encryption and Data Integrity, Authentication and Authorization
Gate Keeping: Identify users and give permissions
Data High Level

Software Computation
Storage Service
Resources in many locations and organisations
Introductive concepts
• Computational Resources: machines where an user can

execute a program (“job”), store or retrieve files
• Job: is a computational task (an executable, a script) that
the user wants to be executed in the Grid, having finally
the results back on it's machine
• VO: people owning at different institutions but working at
similar problems sharing resources, applications, data, …
Grid Concepts
User User
Interface Interface
Grid services
Extendable Infrastructure
gLite decomposition
CLI API
Access
Authorization Information & Job

Monitoring Monitoring
Auditing
Authentication
Information &
Security Services Monitoring Services
Metadata File & Replica Job Package

Catalog Catalog Provenance Manager
Accounting
Storage Data Computing Workload
Element Movement Element Management
Data Services Job Mgmt. Services
Overview paper http://doc.cern.ch//archive/electronic/egee/tr/egee-tr-2006-001.pdf
Work Load Management
Users
Tools to:
•Submit jobs to a CE
•Monitor jobs
How do I run a job on a
Compute Element (CE) ? •Get outputs
(CE = batch queue) •Transfer files to CE
•Transfer files between CE
and SE
Resources
Compute elements Network resources Data storage
WMS
Information System (IS)
Users
Resource publishes:
• Configuration
Which resources • Latest status
can I use?
• Access information
Resources
Information System
User applications
Hierarchical information WMS

Monitoring services
BDII (gilda)
VO
Other GIIS GIIS GIIS

(gilda) INFN sez. CT Merida (gilda)
Site Globus
MDS
GRISes GRISes GRISes

Resource
Data Management
Users
Data Management
provides:
How
My canare
Ourdata
data Iare
manage
ininfiles, • Storage Space
and share
and I’ve terabytes
my data? • Data Transfer
• Replication
Resources
LCG File Catalog (LFC)
Catalogs
LFN (Logical file name)
GUID (Grid unique identifier)
SimLinks
SURL (Site URL)
TURL (Transfer URL)
Data Movement
•GridFTP: basic transfer protocol

•Data Scheduler (DS)
•File Transfer/Placement Service (FTS/FPS)
•Transfer Queue (Table)
•Transfer Agent (Network)
Basic gLite use case:
Enabling Grids for E-sciencE Job submission
User Interface Information System
Submit job
(executable + small inputs) Resource Broker
query
Retrieve
status &
create (small) output files
proxy
query
publish
Submit job
state
Retrieve
File and Replica Catalog output
Job
Logging
status
Site X
Register file Computing Element Storage Element
Input file(s)
Job
status
VO Management process
Service Output file(s)
(DB of VO users)
Logging and
bookkeeping
Main components
User Interface (UI): The place where users logon to the Grid
Resource Broker (RB) (Workload Management System (WMS):

Matches the user requirements with the available resources on the Grid
Information System: Characteristics and status of CE and SE
File and replica catalog: Location of grid files and grid file replicas
Logging and Bookkeeping (LB): Log information of jobs
Computing Element (CE): A batch queue on a site’s computers where

the user’s job is executed
Storage Element (SE): provides (large-scale) storage for files
Main components
User Interface (UI): The place where users logon to the Grid
Resource Broker (RB) (Workload Management System (WMS):

Matches the user requirements with the available resources on the Grid
All built upon

Information System: Characteristics and status of CE and SE
authorisation,
File and replica catalog: Location of grid files and grid file replicas
authentication,
Logging and Bookkeeping (LB): Log information of jobs
security
Computing Element (CE): A batch queue on a site’s computers where
the user’s job is executed
Storage Element (SE): provides (large-scale) storage for files
Who provides the resources?!
Service Provider Note

User interface User / institute / VO Computer with client SW
Resource Broker VOs - EGEE does
(WMS) not fund RBs
Information System Grid operations -
EGEE funded effort
File and replica catalog VOs - EGEE does
not fund catalogs
Logging and VOs - EGEE does
Bookkeeping not fund LB servers
Computing Element VOs - EGEE does VOs provide resources to
(CE) not fund CEs match average need
Storage Element VOs - EGEE does VOs provide resources to
(SE) not fund SEs match average need
External services User / institute / VO To extend the capabilities of
the core infrastructure
Further information
• EGEE digital library: http://egee.lib.ed.ac.uk/

• EGEE www.eu-egee.org
• gLite http://www.glite.org/
• GILDA https://gilda.ct.infn.it /
• Open Grid Forum http://www.gridforum.org/

• Globus Alliance http://www.globus.org/
• VDT http://www.cs.wisc.edu/vdt/
Questions

Introduction To Egee and Glite Middleware: Enabling Grids For E-Science

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Introduction To Egee and Glite Middleware: Enabling Grids For E-Science

Uploaded by

Copyright:

Available Formats

Enabling Grids for E-sciencE

Introduction to EGEE and gLite

April 18, 2009 -- Taipei

• EGEE status, history and perspectives

• Sources of further information

• 61 registered applications so far

Users in many locations and organisations

Access services (User Interface) :

Gate Keeping: Identify users and give permissions

Data High Level

Resources in many locations and organisations

• Computational Resources: machines where an user can

Authorization Information & Job

Metadata File & Replica Job Package

Hierarchical information WMS

Other GIIS GIIS GIIS

GRISes GRISes GRISes

GUID (Grid unique identifier)

SURL (Site URL)

TURL (Transfer URL)

•GridFTP: basic transfer protocol

Resource Broker (RB) (Workload Management System (WMS):

Information System: Characteristics and status of CE and SE

Logging and Bookkeeping (LB): Log information of jobs

Computing Element (CE): A batch queue on a site’s computers where

Storage Element (SE): provides (large-scale) storage for files

Resource Broker (RB) (Workload Management System (WMS):

All built upon

Storage Element (SE): provides (large-scale) storage for files

Service Provider Note

• EGEE digital library: http://egee.lib.ed.ac.uk/

• Open Grid Forum http://www.gridforum.org/

You might also like