This blog provides instructions on setting up the ResourceManager (RM) High Availability (HA) feature in a HDFS cluster. The Active and Standby ResourceManagers embed the Zookeeper-based ActiveStandbyElector to determine which ResourceManager should be active.
Note: This guide assumes that an existing HDP cluster has been manually installed and deployed. It provides instructions on how to manually enable ResourceManager HA on top of the existing cluster.
The ResourceManager was a single point of failure (SPOF) in an HDFS cluster. Each cluster had a single ResourceManager, and if that machine or process became unavailable, the entire cluster would be unavailable until the ResourceManager was either restarted or started on a separate machine. This situation impacted the total availability of the HDFS cluster in two major ways:
• In the case of an unplanned event such as a machine crash, the cluster would be unavailable until an operator restarted the ResourceManager.
• Planned maintenance events such as software or hardware upgrades on the ResourceManager machine would result in windows of cluster downtime.
The ResourceManager HA feature addresses these problems. This feature enables you to run redundant ResourceManagers in the same cluster in an Active/Passive configuration with a hot standby. This mechanism thus facilitates either a fast failover to the standby ResourceManager during machine crash or a graceful administrator-initiated failover during planned maintenance.
Hardware Resources
Ensure that you prepare the following hardware resources:
• ResourceManager machines: The machines where you run Active and Standby ResourceManagers should have exactly the same hardware. For recommended hardware for ResourceManagers, see "Hardware for Master Nodes" in the Cluster Planning Guide.
• Zookeeper machines: For automated failover functionality, there must be an existing Zookeeper cluster available. The Zookeeper service nodes can be co-located with other Hadoop daemons.
Deploy ResourceManager HA Cluster
HA configuration is backwards-compatible and works with your existing single ResourceManager configuration.As described in the following sections, first configure manual or automatic ResourceManager failover. Then deploy the ResourceManager HA cluster.
Configure Manual or Automatic ResourceManager Failover
Complete the following prerequisites:• Make sure that you have a working Zookeeper service. If you had an Ambari deployed HDP cluster with Zookeeper, you can use that Zookeeper service. If not, deploy ZooKeeper using the instructions provided in the Non-Ambari Cluster Installation Guide.
Note: In a typical deployment, ZooKeeper daemons are configured to run on three or five nodes. It is, however, acceptable to co-locate the ZooKeeper nodes on the same hardware as the HDFS NameNode and Standby Node. Many operators choose to deploy the third ZooKeeper process on the same node as the YARN ResourceManager. To achieve performance and improve isolation, Hortonworks recommends configuring the ZooKeeper nodes such that the ZooKeeper data and HDFS metadata is stored on separate disk drives.
• Shut down the cluster using the instructions provided in "Controlling HDP Services Manually," in the HDP Reference Guide.
Set Common ResourceManager HA Properties
The following properties are required for both manual and automatic ResourceManager HA. Add these properties to the etc/hadoop/conf/yarn-site.xml file:
Note: You can also set values for each of the following properties in yarnsite.xml:
yarn.resourcemanager.address.<rm#id>
yarn.resourcemanager.scheduler.address.<rm#id>
yarn.resourcemanager.admin.address.<rm#id>
yarn.resourcemanager.resource#tracker.address.<rm#id>
yarn.resourcemanager.webapp.address.<rm#id>
If these addresses are not explicitly set, each of these properties will use
yarn.resourcemanager.hostname.<rm-id>:default_port
such as DEFAULT_RM_PORT, DEFAULT_RM_SCHEDULER_PORT, etc.
The following is a sample yarn-site.xml file with these common ResourceManager
HA properties configured:
<!-- RM HA Configurations-->
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>${rm1 address}</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>${rm2 address}</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm1</name>
<value>rm1_web_address:port_num</value>
<description>We can set rm1_web_address separately.
If not, it will use
${yarn.resourcemanager.hostname.rm1}:DEFAULT_RM_WEBAPP_PORT
</description>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>rm2_web_address:port_num</value>
</property>
<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.store.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.
ZKRMStateStore</value>
</property>
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>${zk1.address,zk2.address}</value>
</property>
<property>
<name>yarn.client.failover-proxy-provider</name>
<value>org.apache.hadoop.yarn.client.
ConfiguredRMFailoverProxyProvider</value>
</property>
Configure Manual ResourceManager Failover
Automatic ResourceManager failover is enabled by default, so it must be disabled for manual failover.
To configure manual failover for ResourceManager HA, add the yarn.resourcemanager.ha.automatic-failover.enabled configuration property to the etc/hadoop/conf/yarn-site.xml file, and set its value to "false":
<property>
<name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
<value>false</value>
</property>
Configure Automatic ResourceManager Failover
The preceding section described how to configure manual failover. In that mode, the system will not automatically trigger a failover from the active to the standby ResourceManager, even if the active node has failed. This section describes how to configure automatic failover.
1. Add the following configuration options to the yarn-site.xml file:
Example:
<property>
<name>yarn.resourcemanager.ha.automatic-failover.zk-base-path</name>
<value>/yarn-leader-election</value>
<description>Optional setting. The default value is
/yarn-leader-election</description>
</property>
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>yarn-cluster</value>
</property>
2. Automatic ResourceManager failover is enabled by default.
If you previously configured manual ResourceManager failover by setting the value of
yarn.resourcemanager.ha.automatic-failover.enabled to "false", you must
delete this property to return automatic failover to its default enabled state.
Deploy the ResourceManager HA Cluster
1. Copy the etc/hadoop/conf/yarn-site.xml file from the primary ResourceManager host to the standby ResourceManager host.
2. Make sure that the clientPort value set in etc/zookeeper/conf/zoo.cfg matches the port set in the following yarn-site.xml property:
<property>
<name>yarn.resourcemanager.zk-state-store.address</name>
<value>localhost:2181</value>
</property>
3. Start ZooKeeper. Execute this command on the ZooKeeper host machine(s):
su - zookeeper -c "export ZOOCFGDIR=/usr/hdp/current/zookeeper-server/
conf ; export ZOOCFG=zoo.cfg; source /usr/hdp/current/zookeeper-server/conf/
zookeeper-env.sh ; /usr/hdp/current/zookeeper-server/bin/zkServer.sh start"
4. Start HDFS using the instructions provided in "Controlling HDP Services Manually".
5. Start YARN using the instructions provided in "Controlling HDP Services Manually".
6. Set the active ResourceManager:
MANUAL FAILOVER ONLY: If you configured manual ResourceManager failover, you
must transition one of the ResourceManagers to Active mode. Execute the following CLI
command to transition ResourceManager "rm1" to Active:
yarn rmadmin -transitionToActive rm1
You can use the following CLI command to transition ResourceManager "rm1" to Standby mode:
yarn rmadmin -transitionToStandby rm1
AUTOMATIC FAILOVER: If you configured automatic ResourceManager failover, no
action is required -- the Active ResourceManager will be chosen automatically.
7. Start all remaining unstarted cluster services using the instructions provided in
"Controlling HDP Services Manually," in the HDP Reference Guide.
Minimum Settings for Automatic ResourceManager
HA Configuration
The minimum yarn-site.xml configuration settings for ResourceManager HA with automatic failover are as follows:
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>192.168.1.9</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>192.168.1.10</value>
</property>
<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.store.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.
ZKRMStateStore</value>
</property>
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>192.168.1.9:2181,192.168.1.10:2181</value>
<description>For multiple zk services, separate them with comma</description>
</property>
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>yarn-cluster</value>
</property>
Testing ResourceManager HA on a Single Node
If you would like to test ResourceManager HA on a single node (launch more than one
ResourceManager on a single node), you need to add the following settings in yarnsite.
xml.
To enable ResourceManager "rm1" to launch:
<property>
<name>yarn.resourcemanager.ha.id</name>
<value>rm1</value>
<description>If we want to launch more than one RM in single node, we need
this configuration</description>
</property>
To enable ResourceManager rm2 to launch:
<property>
<name>yarn.resourcemanager.ha.id</name>
<value>rm2</value>
<description>If we want to launch more than one RM in single node, we need
this configuration</description>
</property>
You should also explicitly set values specific to each ResourceManager for the following properties separately in yarn-site.xml:
• yarn.resourcemanager.address.<rm-id>
• yarn.resourcemanager.scheduler.address.<rm-id>
• yarn.resourcemanager.admin.address.<rm-id>
• yarn.resourcemanager.resource#tracker.address.<rm-id>
• yarn.resourcemanager.webapp.address.<rm-id>
For example:
<!-- RM1 Configs -->
<property>
<name>yarn.resourcemanager.address.rm1</name>
<value>localhost:23140</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address.rm1</name>
<value>localhost:23130</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm1</name>
<value>localhost:23188</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address.rm1</name>
<value>localhost:23125</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address.rm1</name>
<value>localhost:23141</value>
</property>
<!-- RM2 configs -->
<property>
<name>yarn.resourcemanager.address.rm2</name>
<value>localhost:33140</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address.rm2</name>
<value>localhost:33130</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>localhost:33188</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address.rm2</name>
<value>localhost:33125</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address.rm2</name>
<value>localhost:33141</value>
</property>
No comments:
Post a Comment