gRCP Health Check with Powershell

Question

question

FJS asked May 8, '22 FJS commented May 11, '22

gRCP Health Check with Powershell

Hello.
I am checking periodically gRPC health of several microservices, every two seconds, it is working since I got the response they are serving, the output is { "status": "SERVING" }

  while ($true) {grpcurl -d '{ \"service1\": \"name1\" }' -H "authorization: key key"  host1:port grpc.health.v1.Health/Check>> grpcurl -d '{ \"service2\": \"name2\" }' -H "authorization: key key"  host2:port grpc.health.v1.Health/Check>> grpcurl -d '{ \"service3\": \"name3\" }' -H "authorization: key key"  host3:port grpc.health.v1.Health/Check>> grpcurl -d '{ \"service4\": \"name4\" }' -H "authorization: key key"  host4:port grpc.health.v1.Health/Check>> grpcurl -d '{ \"service5\": \"name5\" }' -H "authorization: key key"  host5:port grpc.health.v1.Health/Check>> grpcurl -d '{ \"service6\": \"name6\" }' -H "authorization: key key" host6:port grpc.health.v1.Health/Check>> grpcurl -d '{ \"service7\": \"name7\" }' -H "authorization: key key" host7:port grpc.health.v1.Health/Check;start-sleep 2}

Now I need a notification if the output is not "SERVING" while the code keeps running in a loop, how can I achieve this?

Please note, one or more service can fail but other can be ok so notification should only about the ones are not answering "SERVING", thanks,

windows-server-powershell

· 2

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

RichMatheisen-8856 · May 08 at 06:37 PM

I'm not familiar with grpcurl. It looks like you're running 7 instances of that tool and each (except the last) is appending it's output (STDOUT) to the following instance -- but what does that "following" instance do with that?

Is it only the output from the "host7" instance that you see ('{ status": "SERVING" }'), or do you get back a JSON array of statuses?

Oh, and what sort of notification are you looking for?

0 ·

FJS RichMatheisen-8856 · May 08 at 06:42 PM

Hi,
It is not appending, is running grpcurls in a secuence once per host then 2 seconds after that starts again, the idea is to monitor the health of each 7 services and get a warn once one or more are not responding "SERVING" any notification should work like a beep, popup, etc...

0 ·

Answer 1 · 2022-05-10T14:49:34.68Z

RichMatheisen-8856 answered May 10, '22 FJS commented May 11, '22

There are ways to make this more efficient, and to also handle those times when grpcurl doesn't return anything in STDOUT (e.g. the host cannot be contacted, authentication fails, etc.) but writes an error message to STDERR, but I'll leave that part up to you,

Give this one a try:

 function Serving{
     param(
         $val
     )
     if ($val -isnot [array]){
         return $false
     }
     if ($val.count -ne 3){
         return $false
     }
     Try{
         $p = $val -join "" | ConvertFrom-Json -ErrorAction STOP
         if ($p.status -eq 'serving'){
             $true
         }
         else{
             $false
         }
     }
     Catch{
         Write-Host "'$p' is not JSON"
         $false
     }
 }
 while ($true) {
     $beep = $false
     [array]$response = grpcurl -d '{ \"service1\": \"name1\" }' -H "authorization: key key"  host1:port grpc.health.v1.Health/Check
     if (-Not (Serving $response)){
         Write-Host "Service1 isn't serving" -ForegroundColor Yellow
         $beep = $true
     }
     [array]$response = grpcurl -d '{ \"service2\": \"name2\" }' -H "authorization: key key"  host2:port grpc.health.v1.Health/Check
     if (-Not (Serving $response)){
         Write-Host "Service2 isn't serving" -ForegroundColor Yellow
         $beep = $true
     }
     [array]$response = grpcurl -d '{ \"service3\": \"name3\" }' -H "authorization: key key"  host3:port grpc.health.v1.Health/Check
     if (-Not (Serving $response)){
         Write-Host "Service3 isn't serving" -ForegroundColor Yellow
         $beep = $true
     }
     [array]$response = grpcurl -d '{ \"service4\": \"name4\" }' -H "authorization: key key"  host4:port grpc.health.v1.Health/Check 
     if (-Not (Serving $response)){
         Write-Host "Service4 isn't serving" -ForegroundColor Yellow
         $beep = $true
     }
     [array]$response = grpcurl -d '{ \"service5\": \"name5\" }' -H "authorization: key key"  host5:port grpc.health.v1.Health/Check
     if (-Not (Serving $response)){
         Write-Host "Service5 isn't serving" -ForegroundColor Yellow
         $beep = $true
     }
     [array]$response = grpcurl -d '{ \"service6\": \"name6\" }' -H "authorization: key key"  host6:port grpc.health.v1.Health/Check
     if (-Not (Serving $response)){
         Write-Host "Service6 isn't serving" -ForegroundColor Yellow
         $beep = $true
     }
     [array]$response = grpcurl -d '{ \"service7\": \"name7\" }' -H "authorization: key key"  host7:port grpc.health.v1.Health/Check 
     if (-Not (Serving $response)){
         Write-Host "Service7 isn't serving" -ForegroundColor Yellow
         $beep = $true
     }
     if ($beep){
         [console]::beep(500, 300)
     }
     Start-Sleep 2 
 }

· 1

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

FJS · May 11 at 09:16 PM

Brillant is exactly what I needed, I got some errors when some services are not SERVING also beep. Wonder if I use my email notification instead beeps work as well, I will try, Many thanks!

0 ·

Answer 2 · 2022-05-08T19:36:59.253Z

RichMatheisen-8856 answered May 8, '22

See if this works:

 $responses = @()
 while ($true) {
     $responses += grpcurl -d '{ \"service1\": \"name1\" }' -H "authorization: key key"  host1:port grpc.health.v1.Health/Check
     $responses += grpcurl -d '{ \"service2\": \"name2\" }' -H "authorization: key key"  host2:port grpc.health.v1.Health/Check
     $responses += grpcurl -d '{ \"service3\": \"name3\" }' -H "authorization: key key"  host3:port grpc.health.v1.Health/Check 
     $responses += grpcurl -d '{ \"service4\": \"name4\" }' -H "authorization: key key"  host4:port grpc.health.v1.Health/Check 
     $responses += grpcurl -d '{ \"service5\": \"name5\" }' -H "authorization: key key"  host5:port grpc.health.v1.Health/Check 
     $responses += grpcurl -d '{ \"service6\": \"name6\" }' -H "authorization: key key"  host6:port grpc.health.v1.Health/Check 
     $responses += grpcurl -d '{ \"service7\": \"name7\" }' -H "authorization: key key"  host7:port grpc.health.v1.Health/Check 
     $ctr = 0
     [array]$bad = @()
     ForEach ($response in $responses){
         $ctr++
         if ($response -ne '{ "status": "SERVING" }'){
             $bad += $ctr
         }      
     }
     if ($bad.count -gt 0){
         [console]::beep(500,300)
         $bad |
             ForEach-Object{
                 Write-Host "host$($_): $($responses[$_ - 1])" -ForegroundColor Yellow
             }
    
     }
     $responses = @()
     $bad = @()
     Start-Sleep 2 
 }

I don't know what your host names are so the script will probably need some work to be usable.

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 3 · 2022-05-08T20:43:25.243Z

FJS answered May 8, '22 FJS published May 8, '22

It works, but couple of things all are SERVING now and I get a beep at first just when the script is executed and a beep when the 7 checks are done just before start the loop again, the idea of the beep used as a warning is in case the output is not SERVING also how can I change in the Write-Host "host$ part to have the name of each service?
now I am geting this output...
host1: {
host2: "status": "SERVING"
host3: }
host4: {
host5: "status": "SERVING"
host6: }
host7: {
host8: "status": "SERVING"
host9: }
host10: {
host11: "status": "SERVING"
host12: }
host13: {
host14: "status": "SERVING"
host15: }
host16: {
host17: "status": "SERVING"
host18: }
host19: {
host20: "status": "SERVING"
host21: }
host1: {
host2: "status": "SERVING"
host3: }
host4: {
host5: "status": "SERVING"
host6: }
host7: {
host8: "status": "SERVING"
host9: }
host10: {
host11: "status": "SERVING"
host12: }
host13: {
host14: "status": "SERVING"

thanks!

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 4 · 2022-05-08T23:07:40.037Z

RichMatheisen-8856 answered May 8, '22

I just phonied up a few "responses and commented out the "while" and the execution of the grpcurl tool, and changed the "sleep" time to 5 seconds.

 $responses = @(
     '{ "status": "SERVING" }'
     '{ "status": "SERVING" }'
     '{ "status": "is not SERVING" }'
     '{ "status": "SERVING" }'
     '{ "status": "is not SERVING" }'
     '{ "status": "SERVING" }'
 )
 #while ($true) {
 #     $responses += grpcurl -d '{ \"service1\": \"name1\" }' -H "authorization: key key"  host1:port grpc.health.v1.Health/Check
 #     $responses += grpcurl -d '{ \"service2\": \"name2\" }' -H "authorization: key key"  host2:port grpc.health.v1.Health/Check
 #     $responses += grpcurl -d '{ \"service3\": \"name3\" }' -H "authorization: key key"  host3:port grpc.health.v1.Health/Check 
 #     $responses += grpcurl -d '{ \"service4\": \"name4\" }' -H "authorization: key key"  host4:port grpc.health.v1.Health/Check 
 #     $responses += grpcurl -d '{ \"service5\": \"name5\" }' -H "authorization: key key"  host5:port grpc.health.v1.Health/Check 
 #     $responses += grpcurl -d '{ \"service6\": \"name6\" }' -H "authorization: key key"  host6:port grpc.health.v1.Health/Check 
 #     $responses += grpcurl -d '{ \"service7\": \"name7\" }' -H "authorization: key key"  host7:port grpc.health.v1.Health/Check 
     $ctr = 0
     [array]$bad = @()
     ForEach ($response in $responses){
         $ctr++
         if ($response -ne '{ "status": "SERVING" }'){
             $bad += $ctr
         }      
     }
     if ($bad.count -gt 0){
         [console]::beep(500,300)
         $bad |
             ForEach-Object{
                 Write-Host "host$($_): $($responses[$_ - 1])" -ForegroundColor Yellow
             }
    
     }
     $responses = @()
     $bad = @()
     Start-Sleep -s 5 
 #}

The results were a "beep" and this:

 PS C:\Users\richm> . "c:\Junk\Untitled-22.ps1"
 host3: { "status": "is not SERVING" }
 host5: { "status": "is not SERVING" }
 PS C:\Users\richm>

I don't any way to run grpcurl so I can't tell you what is being written to STDOUT. Perhaps there are multiple lines being written to STDOUT by each execution? Or each line is ended (or preceded) by a CrLf pair?

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 5 · 2022-05-08T23:10:00.26Z

RichMatheisen-8856 answered May 8, '22 RichMatheisen-8856 edited May 8, '22

If there are CrLf in the grpcurl responses they'll screw up the response checking.

Try this version. It removes any whitespace at the begging and end of each response from grpcurl:

 while ($true) {
      $responses += grpcurl -d '{ \"service1\": \"name1\" }' -H "authorization: key key"  host1:port grpc.health.v1.Health/Check
      $responses += grpcurl -d '{ \"service2\": \"name2\" }' -H "authorization: key key"  host2:port grpc.health.v1.Health/Check
      $responses += grpcurl -d '{ \"service3\": \"name3\" }' -H "authorization: key key"  host3:port grpc.health.v1.Health/Check 
      $responses += grpcurl -d '{ \"service4\": \"name4\" }' -H "authorization: key key"  host4:port grpc.health.v1.Health/Check 
      $responses += grpcurl -d '{ \"service5\": \"name5\" }' -H "authorization: key key"  host5:port grpc.health.v1.Health/Check 
      $responses += grpcurl -d '{ \"service6\": \"name6\" }' -H "authorization: key key"  host6:port grpc.health.v1.Health/Check 
      $responses += grpcurl -d '{ \"service7\": \"name7\" }' -H "authorization: key key"  host7:port grpc.health.v1.Health/Check 
     $ctr = 0
     [array]$bad = @()
     ForEach ($response in $responses){
         $r = $response.Trim()
         $ctr++
         if ($r -ne '{ "status": "SERVING" }'){
             $bad += $ctr
         }      
     }
     if ($bad.count -gt 0){
         [console]::beep(500,300)
         $bad |
             ForEach-Object{
                 Write-Host "host$($_): $($responses[$_ - 1].Trim())" -ForegroundColor Yellow
             }
    
     }
     $responses = @()
     $bad = @()
     Start-Sleep 2 
 }

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 6 · 2022-05-08T23:36:47.343Z

FJS answered May 8, '22 RichMatheisen-8856 commented May 9, '22

Still not working this way.
Each grpcurl excution create this output:

host1: {
host2: "status": "SERVING"
host3: }

· 1

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

RichMatheisen-8856 · May 09 at 02:16 AM

Add this line richt below the "ForEach..." on line 11"

 $response | format-hex

I only need to see a couple of the results. Seeing exactly what's in the $response would be enlightening. I'm beginning to suspect that what grpcurl is returning is JSON, and it's on more than one line. It probably looks like this (over 3 lines)

 {
     "status":  "SERVING"
 }

0 ·

Answer 7 · 2022-05-09T00:16:33.257Z

FJS answered May 9, '22

Lets use the last code you shared with curl instead, so we can both test it, goal is if the ouput is not 200 OK then send an email(only for the failing curls) and keep checking in a loop. Also I changed the beep to send email ,which works better for the alarming purpose..

  $responses = @()
 while ($true) {
      $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.marca.com
      $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.marca4.com
      $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.goofle.com
      $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.googrrle.com
      $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.larazon.com
      $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.larazon.es 
      $ctr = 0
      [array]$bad = @()
      ForEach ($response in $responses){
          $r = $response.Trim()
          $ctr++
          if ($r -ne '{ "status": "200" }'){
              $bad += $ctr
          }      
      }
     if ($bad.count -gt 0){
          #we  define the mail object here
 $Message = new-object Net.Mail.MailMessage
 $Message.Subject = "gRPC Health Microservices Sandbox Notification"
 $smtp = new-object Net.Mail.SmtpClient("smtp.gmail.com", 587)
 #we declare our email credentials here, which includes our email and the password to the email
 $smtp.Credentials = New-Object System.Net.NetworkCredential("xxxx@xxx.net", "xxyyxyx");
 $smtp.EnableSsl = $true
 #we specify our email here, whereby it's coming from us 
 $Message.From = "xxxx@xxx.net"
 #we specify who we want to send the message to
 $Message.To.Add("xxxx@xxx.net")
 #we add the body of our message here
 $Message.Body = "response is not SERVING. Check this service!"
 #then we send it with the smtp object
 $smtp.Send($Message)
          $bad |
              ForEach-Object{
                  Write-Host "host$($_): $($responses[$_ - 1].Trim())" -ForegroundColor Yellow
              }
        
      }
      $responses = @()
      $bad = @()
      Start-Sleep 2 
  }

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 8 · 2022-05-09T00:22:51.817Z

FJS answered May 9, '22 FJS edited May 9, '22

I am getting emails for each execution and the idea is get an email only if is not SERVING and what are the service or services which are not SERVING (in the curl code example if is not 200 )

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 9 · 2022-05-09T02:07:05.347Z

RichMatheisen-8856 answered May 9, '22 RichMatheisen-8856 edited May 9, '22

I removed the e-mail from the code. Also, the test on line 14 of your example using curl is incorrect -- curl only returns a numeric status, not a JSON string.

 $responses = @()
 while ($true) {
     $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.marca.com
     $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.marca4.com
     $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.goofle.com
     $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.googrrle.com
     $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.larazon.com
     $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.larazon.es 
     $ctr = 0
     [array]$bad = @()
     ForEach ($response in $responses) {
         $r = $response.Trim()
         $ctr++
         if ($r -ne '200') {
             $bad += $ctr
         }      
     }
     if ($bad.count -gt 0) {
         $bad |
             ForEach-Object {
                 Write-Host "host$($_): $($responses[$_ - 1].Trim())" -ForegroundColor Yellow
             }
           
     }
     $responses = @()
     $bad = @()
     Start-Sleep -s 5 
 }

That code returns this:

 PS C:\Users\richm> . "c:\Junk\Untitled-22.ps1"
 host2: 000
 host4: 000
 host5: 000
 host2: 000
 host4: 000
 host5: 000
 etc.
 etc.
 etc.

Only the non-200 status values are flagged.

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 10 · 2022-05-09T02:28:42.937Z

RichMatheisen-8856 answered May 9, '22

If the response from grpcurl is JSON, then this should get you the correct output:

 while ($true) {
     $responses = @()
     $responses += grpcurl -d '{ \"service1\": \"name1\" }' -H "authorization: key key"  host1:port grpc.health.v1.Health/Check
     $responses += grpcurl -d '{ \"service2\": \"name2\" }' -H "authorization: key key"  host2:port grpc.health.v1.Health/Check
     $responses += grpcurl -d '{ \"service3\": \"name3\" }' -H "authorization: key key"  host3:port grpc.health.v1.Health/Check 
     $responses += grpcurl -d '{ \"service4\": \"name4\" }' -H "authorization: key key"  host4:port grpc.health.v1.Health/Check 
     $responses += grpcurl -d '{ \"service5\": \"name5\" }' -H "authorization: key key"  host5:port grpc.health.v1.Health/Check 
     $responses += grpcurl -d '{ \"service6\": \"name6\" }' -H "authorization: key key"  host6:port grpc.health.v1.Health/Check 
     $responses += grpcurl -d '{ \"service7\": \"name7\" }' -H "authorization: key key"  host7:port grpc.health.v1.Health/Check 
     $ctr = 0
     [array]$bad = @()
     ForEach ($response in $responses) {
         $ctr++
         $pp = $response | ConvertFrom-Json
         if ($pp.status -ne 'SERVING') {
             $bad += $ctr
         }      
     }
     if ($bad.count -gt 0) {
         [console]::beep(500, 300)
         $bad |
             ForEach-Object {
                 Write-Host "host$($_): $($responses[$_ - 1])" -ForegroundColor Yellow
             }
        
     }
     Start-Sleep 2 
 }

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

question