aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
  • Tools
  • About
aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
  • Tools
  • About
  • Engineering

Introducing Kubernetes Control Plane Metrics In GKE

  • relay
  • September 29, 2022
  • 4 minute read

An essential aspect of operating any application is the ability to observe the health and performance of that application and of the underlying infrastructure to quickly resolve issues as they arise. Google Kubernetes Engine (GKE) already provides audit logs, operational logs, and metrics along with out-of-the-box dashboards and automatic error reporting to facilitate running reliable applications at scale. Using these logs and metrics, Cloud Operations provides the alerts, monitoring dashboards and a Logs Explorer to quickly detect, troubleshoot and resolve issues.

Introducing Kubernetes control plane metrics and why they matter

In addition to these existing sources of telemetry data, we are excited to announce that we are now exposing Kubernetes control plane metrics, which are now Generally Available. With GKE, Google fully manages the Kubernetes control plane; however, when troubleshooting issues it can be helpful to have access to certain metrics emitted by the Kubernetes control plane.

As part of our vision to make Kubernetes easier to use and easier to operate, these control plane metrics are directly integrated with Cloud Monitoring, so you don’t need to manage any metric collection or scrape config.

For example, to understand the health of the API server, you can use metrics like apiserver_request_total and apiserver_request_duration_seconds to track the load that the API server is experiencing, the fraction of API server requests that return errors, and the response latency for requests received by the API server. Also, apiserver_storage_objects can be very useful to monitor the saturation of the API server, especially if you’re using custom controllers. Breakdown this metric by the resource label to find out which Kubernetes custom resource or controller is problematic.

Read More  Speed Up Your Teradata Migration With The BigQuery Permission Mapper Tool

When a pod is created it is initially placed in a “pending” state, indicating it hasn’t yet been scheduled on a node. In a healthy cluster, pending pods are relatively quickly scheduled on a node, providing the workload the resources it needs to run. However, a sustained increase in the number of pending pods may indicate a problem scheduling those pods, which may be caused by insufficient resources or inappropriate configuration. Metrics like scheduler_pending_pods, scheduler_schedule_attempts_total, scheduler_preemption_attempts_total, scheduler_preemption_victims , and scheduler_scheduling_attempt_duration_seconds can alert you to potential scheduling issues, so you can act quickly to ensure sufficient resources are available for your pods. Using these metrics in combination will help you better understand the health of your cluster. For instance, if scheduler_preemption_attempts_total goes up, it means that there are higher priority pods available to be scheduled and the Scheduler is preempting some running pods. However, if the value of scheduler_pending_pods is also increasing, this may indicate that you don’t have enough resources to allocate the higher priority pods.

 

If the Kubernetes scheduler is still unable to find a suitable node for a pod, then the pod will eventually be marked as unschedulable. Kubernetes control plane metrics provide you visibility into pod scheduling errors and unschedulable pods. A spike in either means that the Kubernetes scheduler isn’t able to find an appropriate node on which to run many of your pods, which may ultimately impair the performance of your application. In many cases, a high rate of unschedulable pods will not resolve itself until you take some action to address the underlying cause. A good first place to start troubleshooting the issue is to look for recent FailedScheduling events. (If you have GKE system logs enabled, then all Kubernetes events are available in Cloud Logging.) These FailedScheduling events include a message (for instance, “0/6 nodes are available: 6 Insufficient cpu.”) that very helpfully describes exactly why the pod wasn’t able to be scheduled on any nodes, giving you guidance on how to address the problem.

Read More  HEDIS On FHIR To Improve Quality Of Patient Care

A final example: If you see scheduling jobs is very slow, then one possible cause is that a third-party webhooks might be introducing significant latency, causing the API server to take a long time to schedule a job. Kubernetes control plane metrics such as apiserver_admission_webhook_admission_duration_seconds can expose the admission webhook latency, helping you identify the root cause of slow job scheduling and mitigate the issue.

Displayed in context

Not only are we making these additional Kubernetes control plane metrics available, we’re also excited to announce that all of these metrics are displayed in the Kubernetes Engine section of the Cloud Console, making it easy to identify and investigate issues in-context as you’re managing your GKE clusters.

To view these control plane metrics, go to the Kubernetes clusters section of the Cloud Console, select the “Observability” tab, and select “Control plane”:

 

Since all Kubernetes control plane metrics are ingested into Cloud Monitoring, you can create alerting policies in Cloud Alerting so you’re notified as soon as something needs your attention.

PromQL compatible

When you enable Kubernetes control plane metrics for your GKE clusters, all metrics are collected using Google Cloud Managed Service for Prometheus. This means the metrics are sent to Cloud Monitoring in the same GCP project as your Kubernetes cluster and can be queried using PromQL via the Cloud Monitoring API and Metrics explorer.

For example, you can monitor any spikes in the 99th percentile API server response latency using this PromQL query:

 

sum by (instance, verb) (histogram_quantile(0.99, rate(apiserver_request_duration_seconds_bucket{cluster="cluster-name"}[5m])))

 

Third-party support

If you monitor your GKE cluster using popular third party observability tools, any third party observability tool can ingest these Kubernetes control plane metrics using the Cloud Monitoring API.

Read More  Cloud Scheduler: Now Available Across 23 GCP Regions

For example, if you’re a Datadog customer and you’ve enabled Kubernetes control plane metrics for your GKE cluster, then Datadog provides enhanced visualizations that include Kubernetes control plane metrics from the API server, scheduler, and controller manager.

Pricing

All Kubernetes control plane metrics are charged at the standard price for metrics ingested from Google Cloud Managed Service for Prometheus.

Get started

GKE clusters running control plane version 1.23.6 or later can now access metrics from the Kubernetes API server, Scheduler, and Controller Manager. Kubernetes control plane metrics are not available for GKE Autopilot clusters.

The following gcloud command will update a cluster to enable the collection of metrics from the API server, scheduler, and controller manager:

 

gcloud container clusters update [CLUSTER_ID] \
  --zone=[ZONE] \
  --project=[PROJECT_ID] \
  --monitoring=SYSTEM,API_SERVER,SCHEDULER,CONTROLLER_MANAGER

 

Kubernetes control plane metrics can also be configured using Terraform.

Learn more about configuring the collection of control plane metrics.

 

 

By: Nathan Beach (Group Product Manager, Google Kubernetes Engine)
Source: Google Cloud Blog

relay

Related Topics
  • Containers
  • GKE
  • Google Cloud
  • Kubernetes
  • Metrics
  • PromQL
  • Tutorials
You May Also Like
View Post
  • Engineering
  • Software Engineering

My First Pull Request At Age 14

  • March 24, 2023
View Post
  • DevOps
  • Engineering

Verify POST Endpoint Availability With Uptime Checks

  • March 24, 2023
View Post
  • Engineering
  • Tools

Monitor Kubernetes Cloud Costs With Open Source Tools

  • March 20, 2023
View Post
  • Engineering
  • People
  • Technology

Linux Foundation Training & Certification & Cloud Native Computing Foundation Partner With Corise To Prepare 50,000 Professionals For The Certified Kubernetes Administrator Exam

  • March 16, 2023
View Post
  • Engineering
  • People
  • Software Engineering

A Guide To Managing Your Agile Engineering Team

  • March 15, 2023
View Post
  • Engineering
  • People

10 Ways Wikimedia Does Developer Advocacy

  • March 15, 2023
View Post
  • Architecture
  • Computing
  • Design
  • Engineering
  • Multi-Cloud

3 Ways To Screw Up A Multicloud Deployment

  • March 14, 2023
View Post
  • Data
  • Engineering

Sentiment Analysis With BigQuery ML

  • March 13, 2023

Stay Connected!
LATEST
  • 1
    How AI Can Improve Digital Security
    • March 27, 2023
  • 2
    Docker’s Bad Week
    • March 27, 2023
  • 3
    My First Pull Request At Age 14
    • March 24, 2023
  • 4
    AWS Chatbot Now Integrated Into Microsoft Teams
    • March 24, 2023
  • 5
    Verify POST Endpoint Availability With Uptime Checks
    • March 24, 2023
  • 6
    Sovereign Clouds Are Becoming A Big Deal Again
    • March 23, 2023
  • 7
    Ditching Google: The 3 Search Engines That Use AI To Give Results That Are Meaningful
    • March 23, 2023
  • 8
    Pythonic Techniques For Handling Sequences
    • March 21, 2023
  • 9
    Oracle Cloud Infrastructure to Increase the Reliability, Efficiency, and Simplicity of Large-Scale Kubernetes Environments at Reduced Costs
    • March 20, 2023
  • 10
    Monitor Kubernetes Cloud Costs With Open Source Tools
    • March 20, 2023
about
Hello World!

We are aster.cloud. We’re created by programmers for programmers.

Our site aims to provide guides, programming tips, reviews, and interesting materials for tech people and those who want to learn in general.

We would like to hear from you.

If you have any feedback, enquiries, or sponsorship request, kindly reach out to us at:

[email protected]
Most Popular
  • 1
    Cloudflare Takes On Online Fraud Detection Market
    • March 15, 2023
  • 2
    Linux Foundation Training & Certification & Cloud Native Computing Foundation Partner With Corise To Prepare 50,000 Professionals For The Certified Kubernetes Administrator Exam
    • March 16, 2023
  • 3
    Cloudflare Democratizes Post-Quantum Cryptography By Delivering It For Free, By Default
    • March 16, 2023
  • 4
    Daily QR “Scan Scams” Phishing Users On Their Mobile Devices
    • March 16, 2023
  • 5
    Lockheed Martin Launches Commercial Ground Control Software For Satellite Constellations
    • March 14, 2023
  • /
  • Platforms
  • Architecture
  • Engineering
  • Programming
  • Tools
  • About

Input your search keywords and press Enter.