This action might not be possible to undo. Are you sure you want to continue?
A Shared Central Big Data Repository
This white paper talks about the design considerations for enterprises to run Hadoop as a shared service for multiple departments.
Running an Enterprise Big Data repository requires significant investment in
Learn about the considerations for Enterprises to use Hadoop as a shared service for multiple applications and business units. Read about Identity, Security, Resource Sharing, Monitoring and Operations on the Central Service.
resources. A dedicated cluster for each department is cost-prohibitive, leading to the creation of Big Data silos and underutilization of cluster resources. Enterprises that run Hadoop at scale should allow Hadoop clusters to be shared by different business units. They must also support multiple use cases as well as a checkin/checkout model for an analytic block of works. We cover some design considerations for identity management, security, resource sharing and monitoring that are essential to build a secure, robust, highly available and shared central Big Data repository.
Security is of paramount concern in a shared, multi-tenant environment. Early versions of Hadoop had rudimentary security features, essentially relying on a fair use policy in a trusted environment. Recent versions of Hadoop have added significant identity management features. Let us explore a couple of these in detail. Kerberos Kerberos provides authentication and authorization services. The Kerberos mechanism provides stronger authentication in a more secure fashion than what was available in earlier versions of Hadoop. All clients have to authenticate with a central Kerberos service. Kerberos provides role-based access control and privilege enforcement. Kerberos enforces authentication of data node daemons with the parent services (name node and job tracker). Authentication prevents rogue data nodes from connecting to the parent services and compromising the data stored in the cluster. (Refer to the figure below that demonstrates how Hadoop Kerberos Authentication works.)
Kerberos Key Distribution Center
Request Session Ticket
Session Ticket & Session Key
Data Node Name Node HDFS Layer Task Tracker HDFS Data Job Tracker M/R Layer Data Node Task Tracker HDFS Data HDFS Data
Data Node Task Tracker
Hadoop Kerberos Authentication
Lightweight Directory Access Protocol (LDAP) Integration LDAP can be used to create user accounts in all of the data nodes. This provides fine-grained access control policies and prevents privilege escalation attacks.
Hadoop has several security features as listed below: • Running data node daemons on privileged ports. • Running tasks as the job owner instead of the task tracker daemon user. This prevents other users from changing the job and also viewing the local task data. • Preventing users other than the job owner to look at map outputs. • Restricting a task to only communicate with its parent task tracker to prevent rogue users from inspecting map input data. Data Security Hadoop does not natively integrate with data-at-rest encryption solutions. However, the Intel distribution of Hadoop provides fast encryption using Intel hardware enhancements. Hadoop 2.0 provides SSL transport between Hadoop daemons and during the shuffle phase.
Allocating shared resources to different users and groups in a fair and efficient manner poses some unique challenges in Hadoop. Hadoop does not provide policies and SLAs that are typical of shared systems. Hadoop presents the storage layer (HDFS) as a single shared resource but the computational layer (MapReduce) requires some fine-tuning for optimal results. Nevertheless, here are some recommendations on running a user-friendly shared Hadoop cluster. Resource Usage Limits • HDFS Quotas: HDFS provides name quotas and file quotas. Both are very useful to enforce sensible limits on HDFS usage. Designing a sensible shared directory structure is important, since quotas are set at either file or directory level. It is a good practice to have a common directory that is shared across groups and separate quota-limited directories for each group in a shared cluster. • Task Slots: Task slots are configured on a per node basis. They take into account the total capacity of the cluster. Individual jobs are then monitored to determine the number of mappers. A multiple of the number of map slots is the recommended practice.
Hadoop provides different schedulers as plug-ins. That said, not all schedulers are created equal. The FIFO scheduler should not be used as it can lead to significant resource underutilization and job starving. The fair scheduler is a good option for a dedicated cluster but may lead to resource contention in a shared environment. The capacity scheduler is the optimal choice for a shared cluster. The capacity scheduler provides multi-tenancy controls that prevent a user or a group of users from overwhelming the cluster. It also provides capacity guarantees through soft limits and enforceable hard limits. The capacity scheduler additionally improves security by providing ACLs for job queues.
Hadoop provides good monitoring options. We recommend using Ganglia or similar monitoring for production clusters. JMX monitoring should also be enabled. Recent versions of Hadoop ship with the more flexible metrics2 framework for metrics collection. Using metrics2 in the Ganglia context provides valuable insight into cluster usage. Oozie workflows also enables SLA tracking, which is important for a shared cluster.
We have discussed several operational considerations such as security, optimal resource sharing and monitoring. In addition to these, the operations team needs to build a proactive ‘service’ approach that addresses the full range of service components present in a Hadoop environment. Each of these components is a potential point of failure. Operations needs to shift from passive monitoring to actively meeting SLAs in a new distributed environment. This shift in focus necessitates a new organizational culture in addition to operational excellence. Operational Excellence Operational excellence for a shared cluster is not just about cluster health and uptime. Service metrics such as job completion rate, resource sharing and meeting SLAs is also significant. It is important to operationalize the aspects of identity, security, resource sharing and monitoring discussed above. To accomplish these, Hadoop operations need to perform regular audits, fire drills and ensure well documented processes and procedures. A runbook-based troubleshooting guide and well formulated support levels (Level 1, Level 2, and Level 3) with an easy escalation procedure are also required. If SLAs mandate limited service interruption, then the runbooks should have maximum resolution times and mandatory escalation based on severity and time-sensitive resolution. Operational excellence is a function of all of the above.
Design considerations for multi-tenant, multi-use Hadoop clusters are: • Design for security as part of the initial cluster planning. • Implement user-friendly resource sharing while meeting SLAs. • Use the Capacity Scheduler. • Monitor service metrics in addition to cluster metrics. • Institutionalize operational excellence through streamlined procedures and by cultivating a service mindset. As Hadoop becomes more mainstream and indispensable to enterprises, it is imperative that they build, operate and scale shared Hadoop clusters. The design considerations discussed in this paper will help enterprises accomplish the essential mission of running multi-tenant, multi-use Hadoop clusters at scale.
© 2013 Impetus Technologies, Inc. All rights reserved. Product and company names mentioned herein may be trademarks of their respective companies. Oct 2013 #52991
Impetus Technologies is a leading provider of Big Data solutions for the Fortune 500®. We help customers effectively manage the “3-Vs” of Big Data and create new business insights across their enterprises. Visit http://bigdata.impetus.com or write to us at firstname.lastname@example.org