Managed Prometheus cannot fetch an access token from the specified endpoint

Question

I set up scrape targets for Prometheus to automatically discover my Spring Boot services using annotations. These services have metric endpoints protected with token authentication. To retrieve the token, I added the oauth section to the configuration, specifying the token URL and client credentials. Self-managed Keycloak server outside the cluster is used for authentication and authorization.

Initially, the configuration worked fine, and the up{job="kubernetes-service-endpoints"} time series had a value of 1. However, at some point between 2024-04-11 and 2024-04-10, the value dropped to 0. In the Keycloak event log, I found an error message indicating "invalid_client_credentials" and "grant_type=client_credentials".

I'm wondering if any changes were made on the AKS side during that time period that could have caused this issue. Any advice on how to resolve it would be greatly appreciated!

My configuration:

- job_name: 'kubernetes-service-endpoints'
      honor_labels: true
      scrape_interval: 15s
      scrape_timeout: 15s
      kubernetes_sd_configs:
      - role: endpoints
      oauth2:
        client_id: my-monitoring-client
        client_secret: my-monitoring-client-secret
        token_url: https://keycloak-server.my-app.com/realms/MYREALM/protocol/openid-connect/token
      relabel_configs:
      - source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scrape
        regex: true
        action: keep
      - source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scheme
        regex: (https?)
        action: replace
        target_label: __scheme__
      - source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_path
        regex: (.+)
        action: replace
        target_label: __metrics_path__
      - source_labels:
        - __address__
        - __meta_kubernetes_service_annotation_prometheus_io_port
        regex: (.+?)(?::\d+)?;(\d+)
        action: replace
        target_label: __address__
        replacement: $1:$2
      - action: labelmap
        regex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+)
        replacement: __param_$1
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_service_name
        target_label: service
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: node

Answer 1

Hello Andriy S

Welcome to microsoft Q&A, Thankyou for posting your query here.

Ensure that the client_id and client_secret specified in your Prometheus configuration are still valid.

Check the settings in Keycloak for the client to ensure that it is still configured to allow the client_credentials grant type

If you're storing the client_secret in a Kubernetes Secret used by Prometheus, ensure this secret is up-to-date and has been properly propagated to the Prometheus pods.

you should review any changes made to the AKS environment during the period the problem arose like network policies, upgrades to the AKS cluster, or changes to service accounts and RBAC policies.

Make sure the token URL provided in the configuration (https://keycloak-server.my-app.com/realms/MYREALM/protocol/openid-connect/token) is accessible from the Prometheus server

hope this helps you.If an answer has been helpful, please consider accepting the answer to help increase visibility of this question for other members of the Microsoft Q&A community. If not, please let us know what is still needed in the comments so the question can be answered. Thank you for helping to improve Microsoft Q&A!