Make sure to start/stop the Hadoop services in the prescribed order.
• Ranger
• Knox
• ZooKeeper
• HDFS
• YARN
• HBase
• Hive Metastore
• HiveServer2
• WebHCat
• Oozie
• Hue
• Storm
• Kafka
• Atlas
I. Starting HDP services
1. Start Ranger. Execute the following commands on the Ranger host machine:
sudo service ranger-admin start
sudo service ranger-usersync start
2. Start Knox. When starting the gateway with the script below, the process runs in the background. The log output is written to /var/log/knox and a PID (process ID) is written to /var/run/knox. Execute this command on the Knox host machine.
su -l knox -c “/usr/hdp/current/knox-server/bin/gateway.sh start”
Note
If Knox has been stopped without using gateway.sh stop, you must start the service using gateway.sh clean. The clean option removes all log files in /var/log/knox.
3. Start ZooKeeper. Execute this command on the ZooKeeper host machine(s):
su – zookeeper -c “export ZOOCFGDIR=/usr/hdp/current/zookeeper-server/conf ; export ZOOCFG=zoo.cfg; source /usr/hdp/current/zookeeper-server/conf/zookeeper-env.sh ; /usr/hdp/current/zookeeper-server/bin/zkServer.sh start”
4. Start HDFS
o If you are running NameNode HA (High Availability), start the JournalNodes by executing these commands on the JournalNode host machines:
su -l hdfs -c “/usr/hdp/current/hadoop-hdfs-journalnode/../hadoop/sbin/hadoop-daemon.sh start journalnode”
where $HDFS_USER is the HDFS user. For example, hdfs.
o Execute this command on the NameNode host machine(s):
su -l hdfs -c “/usr/hdp/current/hadoop-hdfs-namenode/../hadoop/sbin/hadoop-daemon.sh start namenode”
o If you are running NameNode HA, start the ZooKeeper Failover Controller (ZKFC) by executing the following command on all NameNode machines. The starting sequence of the ZKFCs determines which NameNode will become Active.
su -l hdfs -c “/usr/hdp/current/hadoop-hdfs-namenode/../hadoop/sbin/hadoop-daemon.sh start zkfc”
o If you are not running NameNode HA, execute the following command on the Secondary NameNode host machine. If you are running NameNode HA, the Standby NameNode takes on the role of the Secondary NameNode.
su -l hdfs -c “/usr/hdp/current/hadoop-hdfs-namenode/../hadoop/sbin/hadoop-daemon.sh start secondarynamenode”
o Execute these commands on all DataNodes:
su -l hdfs -c “/usr/hdp/current/hadoop-hdfs-datanode/../hadoop/sbin/hadoop-daemon.sh start datanode”
5. Start YARN
o Execute this command on the ResourceManager host machine(s):
su -l yarn -c “/usr/hdp/current/hadoop-yarn-resourcemanager/sbin/yarn-daemon.sh start resourcemanager”
o Execute this command on the History Server host machine:
su -l mapred -c “/usr/hdp/current/hadoop-mapreduce-historyserver/sbin/mr-jobhistory-daemon.sh start historyserver”
o Execute this command on the timeline server:
su -l yarn -c “/usr/hdp/current/hadoop-yarn-timelineserver/sbin/yarn-daemon.sh start timelineserver”
o Execute this command on all NodeManagers:
su -l yarn -c “/usr/hdp/current/hadoop-yarn-nodemanager/sbin/yarn-daemon.sh start nodemanager”
6. Start HBase
o Execute this command on the HBase Master host machine:
su -l hbase -c “/usr/hdp/current/hbase-master/bin/hbase-daemon.sh start master; sleep 25”
o Execute this command on all RegionServers:
su -l hbase -c “/usr/hdp/current/hbase-regionserver/bin/hbase-daemon.sh start regionserver”
7. Start the Hive Metastore. On the Hive Metastore host machine, execute the following commands:
su $HIVE_USER
nohup /usr/hdp/current/hive-metastore/bin/hive –service metastore>/var/log/hive/hive.out 2>/var/log/hive/hive.log &
Where $HIVE_USER is the Hive user. For example,hive.
8. Start HiveServer2. On the Hive Server2 host machine, execute the following commands:
su $HIVE_USER
nohup /usr/hdp/current/hive-server2/bin/hiveserver2 -hiveconf hive.metastore.uris=/tmp/hiveserver2HD.out 2 /tmp/hiveserver2HD.log
Where $HIVE_USER is the Hive user. For example,hive.
9. Start Oozie. Execute the following command on the Oozie host machine:
su -l oozie -c “/usr/hdp/current/oozie-server/bin/oozied.sh start”
10. As a root user, execute the following command on the Hue Server:
/etc/init.d/hue start
This command starts several subprocesses corresponding to the different Hue components. Even though the root user is the one calls the init.d script, the actual process runs with the Hue user.
11. Start Storm services using a process controller, such as supervisord. For example, to start the storm-nimbus service:
sudo /usr/bin/supervisorctl
storm-drpc RUNNING pid 9801, uptime 0:05:05
storm-nimbus STOPPED Dec 01 06:18 PM
storm-ui RUNNING pid 9800, uptime 0:05:05
supervisor> start storm-nimbus
storm-nimbus: started
where $STORM_USER is the operating system user that installed Storm. For example, storm.
12. Start Kafka with the following commands:
su $KAFKA_USER
/usr/hdp/current/kafka-broker/bin/kafka start
where $KAFKA_USER is the operating system user that installed Kafka. For example, kafka.
13. Start the Atlas server with the following commands:
/usr/hdp/<hdp-version>/atlas/bin/atlas_start.py –port 21000
II. Stopping HDP services
2.1. Stop Ranger. Execute the following commands on the Ranger host machine:
sudo service ranger-admin stop
sudo service ranger-usersync stop
2.2. Stop Knox. Execute the following command on the Knox host machine.
su -l knox -c “/usr/hdp/current/knox-server/bin/gateway.sh stop”
2.3. Stop Oozie. Execute the following command on the Oozie host machine.
su -l oozie -c “/usr/hdp/current/oozie-server/bin/oozied.sh stop”
4. Stop WebHCat. On the WebHCat host machine, execute the following command:
su -l hcat -c “/usr/hdp/current/hive-webhcat/sbin/webhcat_server.sh stop”
5. Stop Hive. Execute this command on the Hive Metastore and Hive Server2 host machine.
ps aux | awk ‘{print $1,$2}’ | grep hive | awk ‘{print $2}’ | xargs kill >/dev/null 2>&1
6. Stop HBase
o Execute this command on all RegionServers:
su -l hbase -c “/usr/hdp/current/hbase-regionserver/bin/hbase-daemon.sh stop regionserver”
o Execute this command on the HBase Master host machine:
su -l hbase -c “/usr/hdp/current/hbase-master/bin/hbase-daemon.sh stop master”
7. Stop YARN
o Execute this command on all NodeManagers:
su -l yarn -c “/usr/hdp/current/hadoop-yarn-nodemanager/sbin/yarn-daemon.sh stop nodemanager”
o Execute this command on the History Server host machine:
su -l mapred -c “/usr/hdp/current/hadoop-mapreduce-historyserver/sbin/mr-jobhistory-daemon.sh stop historyserver”
o Execute this command on the timeline server host machine(s):
su -l yarn -c “/usr/hdp/current/hadoop-yarn-timelineserver/sbin/yarn-daemon.sh stop timelineserver”
o Execute this command on the ResourceManager host machine(s):
su -l yarn -c “/usr/hdp/current/hadoop-yarn-resourcemanager/sbin/yarn-daemon.sh stop resourcemanager”
8. Stop HDFS
o Execute this command on all DataNodes:
su -l hdfs -c “/usr/hdp/current/hadoop-hdfs-datanode/../hadoop/sbin/hadoop-daemon.sh stop datanode”
o If you are not running NameNode HA (High Availability), stop the Secondary NameNode by executing this command on the Secondary NameNode host machine:
su -l hdfs -c “/usr/hdp/current/hadoop-hdfs-namenode/../hadoop/sbin/hadoop-daemon.sh stop secondarynamenode”
o Execute this command on the NameNode host machine(s):
su -l hdfs -c “/usr/hdp/current/hadoop-hdfs-namenode/../hadoop/sbin/hadoop-daemon.sh stop namenode”
o If you are running NameNode HA, stop the ZooKeeper Failover Controllers (ZKFC) by executing this command on the NameNode host machines:
su -l hdfs -c “/usr/hdp/current/hadoop-hdfs-namenode/../hadoop/sbin/hadoop-daemon.sh stop zkfc”
o If you are running NameNode HA, stop the JournalNodes by executing these commands on the JournalNode host machines:
su -l hdfs -c “/usr/hdp/current/hadoop-hdfs-journalnode/../hadoop/sbin/hadoop-daemon.sh stop journalnode”
where $HDFS_USER is the HDFS user. For example, hdfs.
9. Stop ZooKeeper. Execute this command on the ZooKeeper host machine(s):
su – zookeeper -c “export ZOOCFGDIR=/usr/hdp/current/zookeeper-server/conf ; export ZOOCFG=zoo.cfg; source /usr/hdp/current/zookeeper-server/conf/zookeeper-env.sh ; /usr/hdp/current/zookeeper-server/bin/zkServer.sh stop”
10. Stop Hue. Execute the following command:
/etc/init.d/hue stop
11. Start Storm services using a process controller, such as supervisor. For example, to stop the storm-nimbus service:
sudo /usr/bin/supervisorctl
storm-drpc RUNNING pid 9801, uptime 0:03:20
storm-nimbus RUNNING pid 9802, uptime 0:03:20
storm-ui RUNNING pid 9800, uptime 0:03:20
supervisor> stop storm-nimbus
storm-nimbus: stopped
where $STORM_USER is the operating system user that installed Storm. For example, storm.
12. Stop Kafka. Execute this command on the Kafka host machine(s):
su $KAFKA_USER
/usr/hdp/current/kafka-broker/bin/kafka stop
where$KAFKA_USER is the operating system user that installed Kafka. For example,kafka.
13. Stop Atlas. Execute the following command.
su $ATLAS_USER
python /usr/hdp/current/atlas-server/bin/atlas_stop.py