How to Find and Kill a running Yarn Application Master in HDInsight with and without SSH access

  • Article

Today we faced a challenge in HDInsight not knowing the SSH user password to terminal into the server, and we needed to kill some running Hive jobs that were too far gone and taking too many resources.

The user knows the user name and password for Hadoop and Ambari, but does not know the SSH user name and password. HDInsight uses separate accounts and passwords for these two purposes.

Without SSH use cURL to talk REST to kill it

You can use the REST endpoints for Resource manager in HDInsight to kill an application using the Hadoop/Ambari password.

1. Download or build your own cURL https://curl.haxx.se/download.html for Windows, I found it here https://www.paehl.com/open_source/?CURL_7.49.1

Make sure to get one WITH SUPPORT SSL (Since HDInsight only works in HTTPS)

 

2. To check the application status, you can get a long list of all the applications

 curl -u admin:SomePassword -H "Content-Type: application/json" -d "https://servername.azurehdinsight.net/ws/v1/cluster/apps/"

Or focus in on your one application-id

 curl -u admin:SomePassword -H "Content-Type: application/json" -d "https://servername.azurehdinsight.net/ws/v1/cluster/apps/application_0000000000_0000/"

 

3. To Kill a job,on the commandline, run cURL to point to your HDInsight server and the application id, and PUT to change the state to KILLED, effectively killing the Yarn job.

 curl -u admin:SomePassword -v -X PUT -H "Content-Type: application/json" -d '{"state": "KILLED"}' "https://servername.azurehdinsight.net/ws/v1/cluster/apps/application_0000000000_0000/state"

 

 

Note: for special characters I found I could use an escape character \ before the special character in the password.

If you get an error about https, try "" double-quotes around the URL instead of '' single-quotes.

More Resource Manager REST options https://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html

 

With SSH use Yarn commands to list and kill applications

I am used to killing Yarn applications like this, over a BASH session connected with the ssh account credentials. From SSH paste in the application name and this command should attempt the kill

 yarn application -kill application_000000000000000_0001

https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnCommands.html

To find the application ID , you can do yarn application -list or use the resource manager UI from the active head node.

From Ambari https://servername.azurehdinsight.net/#/main/dashboard/metrics click Dashbaord, then YARN links tile, click More… > and pick the Active head node, then click the menu for ResourceManagerUI

 

 

It will jump to the website like this URL https://servername.azurehdinsight.net/yarnui/hn/cluster

The first column lists the application ID, which is what Yarn needs to know to kill something.