Automating operational

procedures with
Daniel Fernández Rodríguez - danielfr@cern.ch (IT-CM-RPS)
Akos Hencz - ahencz@cern.ch (IT-CM-LCS)
CERN IT-CM Group

21/04/2016 HEPiX Spring 2016 Workshop 2

Outline
• What is Rundeck?
• What does Rundeck provide us?
• Deployment
• Use cases
• Missing things
• Summary

21/04/2016 HEPiX Spring 2016 Workshop 3

What is Rundeck?
 Automate routine operational procedures
 Company -
 Language Java
 Apache Software License - Code on
 Actively developed

21/04/2016 HEPiX Spring 2016 Workshop 4

Concepts
• Projects: used to separate management activity. All Rundeck
activities occur within the context of a project
• Jobs: a sequence of steps, job options, and the nodes where the
steps are executed
• Executions: An execution is a representation of the activity of a
running or completed job
• Plugins: extend and expand Rundeck’s functionality
• http://rundeck.org/plugins/index.html

21/04/2016 HEPiX Spring 2016 Workshop 5

What does Rundeck provide us?
• Common place for implemented workflows and jobs
• Clean and easy-to-use Web interface
• Allow for scheduled jobs, cron-style
• Jobs reusability

21/04/2016 HEPiX Spring 2016 Workshop 6

What does Rundeck provide us?
• Fine-grain access control
 Ex: Sysadmins can only execute, Cloud Team modify + execute
• Traceability and auditing
• Delegation internal sensitive tasks to other groups without
exposing credentials or procedures
 Ex: Sysadmins remove node from infrastructure w/o having rights
• Complete log of every executed job

21/04/2016 HEPiX Spring 2016 Workshop 7

Deployment
• All puppetized
• https://github.com/voxpupuli/puppet-rundeck
• Currently working on a Puppet module to deploy
Rundeck with some CERN specifics
• Different deployments for different services
• Credentials

21/04/2016 HEPiX Spring 2016 Workshop 8

Deployment
mod_shib

Apache :443

Tomcat :8009

Rundeck :
4440

rundeck-01.cern.ch

21/04/2016 HEPiX Spring 2016 Workshop 9

Use cases
 OpenStack Management  Configuration
 Project Creation  Rename physical host
 Project Quota Updates  Register new Puppet entities
 Various scheduled synchronizations
 Reporting on the Cloud  Host Management
 Usage and status
 Enable/disable Compute Host
 Health reports
 HW Interventions
 Notifications  Remove physical host
 Hyper-V Patching Campaigns  Internal
 VM owners  Auto backups
 Checks and verifications  Reload nodes
 Consistency checks
 ….
 GNI Tickets
21/04/2016 HEPiX Spring 2016 Workshop 10
HW Intervention on a Compute Node
• Disable nova-service
• Switch Alarms OFF
Disable • Update associated Service-Now ticket
compute node

• Send e-mail to owners of affected VMs

Notifications

Save intervention
Add remote AT job
details
Other tasks

Send calendar Post new

invitation message broker

21/04/2016 HEPiX Spring 2016 Workshop 11

HW Intervention on a Compute Node

Kerberos granting ticket for svcrdeck obtained successfully

[INFO] Changed OS_AUTH_URL to https://keystone.cern.ch:5001/v2.0/
[INFO] Changed SERVICE_ENDPOINT https://keystone.cern.ch:35358/v2.0/

[INFO] Changed OS_REGION_NAME to cell12

[INFO] Appending cern.ch to the host name...
[INFO] Trying to disable nova-compute on P05614911S20073.cern.ch...
+-------------------------+--------------+----------+---------------------------------------------------------+
| Host | Binary | Status | Disabled Reason |
+-------------------------+--------------+----------+---------------------------------------------------------+
| P05614911S20073.cern.ch | nova-compute | disabled | [INC0982341] Disabled by rsantalu using Rundeck (11823) |
+-------------------------+--------------+----------+---------------------------------------------------------+
[INFO] nova-compute sucessfully disabled on P05614911S20073.cern.ch.

TOTAL HOSTS SUCCESS ERROR

[PERFORM] SUMMARY NOVA DISABLE 1 1 0

[INFO] Using roger to disable alarms on the following hosts: P05614911S20073

[INFO] Trying to disable roger alarms on P05614911S20073...
[INFO] Roger alarms sucessfully disabled on P05614911S20073.
[
{
"app_alarmed": false,
"appstate": "production",
"expires": "",
"hostname": "p05614911s20073.cern.ch",
"hw_alarmed": false,
"message": "[INC0982341] Disabled by rsantalu using Rundeck (11823)",
"nc_alarmed": false,
"os_alarmed": false,

21/04/2016 HEPiX Spring 2016 Workshop 12

Register new Puppet entities
• Create new puppet modules & hostgroups
• JIRA Configuration Change ticket
 Ticket status APPROVED
 JIRA Web-Hook executes job via Rundeck REST API

21/04/2016 HEPiX Spring 2016 Workshop 13

Register new Puppet entities
• Parse JIRA ticket with the given ID
 Create project in Gitlab, set ACLs, user groups, LDAP links
 Create puppet skeleton in project & QA branch
 Register project in Foreman and other systems
• Close ticket

21/04/2016 HEPiX Spring 2016 Workshop 14

GNI Alarm Tickets •
•
Opened to Cloud Team
GNI
1 • Not assigned

3 2

Mark ticket as resolved

or
Assign to person in ROTA

21/04/2016 HEPiX Spring 2016 Workshop 15

Missing things…
• Pass data between jobs (https://trello.com/c/oU8OjZoH)
• Share jobs between Projects (https://trello.com/c/jtFcbflm)
• Resume job from where error occurred (
https://trello.com/c/o999XAhW)
• Configure scheduled jobs to be run once (AT jobs) (
https://trello.com/c/cmqu4fdW)
• Allow user interaction (https://trello.com/c/LPnsZ019)

21/04/2016 HEPiX Spring 2016 Workshop 16

Summary
• Rundeck is a great platform, easy to use and deploy
• Active development and increasing user community
• Adapt its behaviour to your preferences via custom plugins
• Valid for simple jobs but also for more complicated workflows

21/04/2016 HEPiX Spring 2016 Workshop 17

Questions?
• Thank you for your attention!

21/04/2016 HEPiX Spring 2016 Workshop 18

I like it. Where can start from?
• Web page: http://rundeck.org
• Oficial docs: http://rundeck.org/docs/index.html
• Rundeck Trello Board for future ideas:
https://trello.com/b/sn3g9nOr/rundeck-development
• Mailing list: http://
groups.google.com/group/rundeck-discuss
• Puppet module:
https://github.com/voxpupuli/puppet-rundeck

21/04/2016 HEPiX Spring 2016 Workshop 20