Qlik Sense Enterprise on Kubernetes での監視メトリック
すべての Qlik Sense Enterprise on Kubernetes サービスは、アクティビティ、正常性、パフォーマンスのデータの監視に使用可能なメトリックを公開します。
それらのデータは、オープン ソースのコンポーネントを使用して表示および照合することが可能です。以下の例は、Prometheus と Grafana を使用してメトリックをリアルタイムで収集および分析する方法を示しています。
Prometheus を使用したメトリックの表示
Prometheus は、メトリックの収集および分析に使用可能なツールキットを監視し、アラートを生成するシステムです。設定されたターゲットから所定の間隔でメトリックを収集し、ルール数式を評価し、結果を表示します。条件が満たされた場合にはアラートをトリガーすることもできます。
Prometheus は、サービスに追加された Kubernetes の注釈を探すことによってメトリックを研削します。
prometheus.io/port=8080
prometheus.io/scrape=truePrometheus チャートのインストール
次のコマンドを実行して stable/prometheus チャートをインストールします。
情報メモRBAC などのクラスターの設定を調整します。
helm upgrade --install prometheus stable/prometheus --set=rbac.create=true,alertmanager.enabled=false,pushgateway.enabled=falseメトリックの表示
次のコマンドによってメトリックを表示します。
export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,release=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:9090 to access prometheus"
kubectl port-forward $POD_NAME 9090Grafana を使用したメトリックの表示
Grafana は、メトリックの監視と分析のための別のツールです。
Grafana のインストール
次のコマンドを実行して Grafana をインストールします。
helm upgrade --install grafana stable/grafana -f grafana.yaml上のコマンドで参照されるサンプルの YAML ファイルでは、次の機能を実行できます。
- Prometheus メトリックを見るように Grafana を設定する。
- GO Services ダッシュボードをプリロードして Golang メトリックを公開する。
- 一般的なメトリックを含む Kubernetes ダッシュボードをプリロードする。
- より固有の POD メトリックを含む Kubernetes コンテナー詳細ダッシュボードをプリロードする。
Example:
datasources:
datasources.yaml:
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
url: http://prometheus-server
access: proxy
isDefault: true
dashboardProviders:
dashboardproviders.yaml:
apiVersion: 1
providers:
- name: 'default'
orgId: 1
folder: ''
type: file
disableDeletion: false
editable: true
options:
path: /var/lib/grafana/dashboards
dashboards:
kubernetes-container-details:
json: |
{
"__inputs": [],
"__requires": [
{
"type": "grafana",
"id": "grafana",
"name": "Grafana",
"version": "5.1.3"
},
{
"type": "panel",
"id": "graph",
"name": "Graph",
"version": "5.0.0"
},
{
"type": "panel",
"id": "singlestat",
"name": "Singlestat",
"version": "5.0.0"
}
],
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"description": "Monitors Kubernetes cluster using Prometheus. Shows overall cluster CPU / Memory / Filesystem usage as well as individual pod, containers, systemd services statistics. Uses cAdvisor metrics only.",
"editable": true,
"gnetId": 315,
"graphTooltip": 0,
"id": null,
"iteration": 1529438741801,
"links": [
{
"icon": "external link",
"tags": [],
"targetBlank": true,
"title": "Kubernetes QoS",
"type": "link",
"url": "https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/#qos-classes"
}
],
"panels": [
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 0
},
"id": 62,
"panels": [],
"title": "QoS (Experimental)",
"type": "row"
},
{
"cacheTimeout": null,
"colorBackground": true,
"colorValue": false,
"colors": [
"#d44a3a",
"rgba(237, 129, 40, 0.89)",
"#299c46"
],
"datasource": "$datasource",
"format": "decbytes",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 2,
"w": 3,
"x": 0,
"y": 1
},
"id": 54,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"expr": "kube_pod_container_resource_requests_memory_bytes{container=~\"$containername\",pod=~'$podname'}",
"format": "time_series",
"instant": true,
"intervalFactor": 1,
"legendFormat": "",
"refId": "A"
}
],
"thresholds": "0,1",
"title": "Memory Request",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": true,
"colorValue": false,
"colors": [
"#d44a3a",
"rgba(237, 129, 40, 0.89)",
"#299c46"
],
"datasource": "$datasource",
"format": "decbytes",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 2,
"w": 3,
"x": 3,
"y": 1
},
"id": 55,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"expr": "kube_pod_container_resource_limits_memory_bytes{container=~\"$containername\",pod=~'$podname'}",
"format": "time_series",
"instant": false,
"intervalFactor": 1,
"legendFormat": "",
"refId": "A"
}
],
"thresholds": "1,2",
"title": "Memory Limit",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": true,
"colorValue": false,
"colors": [
"#d44a3a",
"#cca300",
"#299c46"
],
"datasource": "$datasource",
"format": "none",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 2,
"w": 3,
"x": 6,
"y": 1
},
"id": 56,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"expr": "\n(\nkube_pod_container_resource_limits_memory_bytes{container=~\"$containername\",pod=~'$podname'} == bool\nkube_pod_container_resource_requests_memory_bytes{container=~\"$containername\",pod=~'$podname'}\n) +\n(\nkube_pod_container_resource_limits_memory_bytes{container=~\"$containername\",pod=~'$podname'} >= bool 1 and\nkube_pod_container_resource_requests_memory_bytes{container=~\"$containername\",pod=~'$podname'} >= bool 1\n)\n",
"format": "time_series",
"hide": false,
"instant": true,
"intervalFactor": 1,
"legendFormat": "",
"refId": "A"
}
],
"thresholds": "1,2",
"title": "QoS Guaranteed",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
},
{
"op": "=",
"text": "1-No",
"value": "1"
},
{
"op": "=",
"text": "Yes",
"value": "2"
},
{
"op": "=",
"text": "0-No",
"value": "0"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": true,
"colorValue": false,
"colors": [
"#d44a3a",
"#cca300",
"#299c46"
],
"datasource": "$datasource",
"format": "none",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 2,
"w": 3,
"x": 9,
"y": 1
},
"id": 57,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"expr": "\n(\nkube_pod_container_resource_limits_memory_bytes{container=~\"$containername\",pod=~'$podname'} != bool\nkube_pod_container_resource_requests_memory_bytes{container=~\"$containername\",pod=~'$podname'}\n) \n+\n(\nkube_pod_container_resource_limits_memory_bytes{container=~\"$containername\",pod=~'$podname'} >= bool 1 and\nkube_pod_container_resource_requests_memory_bytes{container=~\"$containername\",pod=~'$podname'} >= bool 1\n)\n",
"format": "time_series",
"instant": true,
"intervalFactor": 1,
"legendFormat": "",
"refId": "A"
}
],
"thresholds": "1,2",
"title": "QoS Burstable",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
},
{
"op": "=",
"text": "0 - No",
"value": "0"
},
{
"op": "=",
"text": "1 - No",
"value": "1"
},
{
"op": "=",
"text": "2 - Yes",
"value": "2"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": true,
"colorValue": false,
"colors": [
"#d44a3a",
"#cca300",
"#299c46"
],
"datasource": "$datasource",
"format": "none",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 2,
"w": 3,
"x": 12,
"y": 1
},
"id": 58,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"expr": "1 + \n(kube_pod_container_resource_limits_memory_bytes{container=~\"$containername\",pod=~'$podname'} < bool 1 OR on() vector(1)) +\n(kube_pod_container_resource_requests_memory_bytes{container=~\"$containername\",pod=~'$podname'} < bool 1 OR on() vector(1))",
"format": "time_series",
"hide": false,
"instant": true,
"intervalFactor": 1,
"legendFormat": "",
"refId": "A"
}
],
"thresholds": "1,2",
"title": "QoS BestEffort",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "0 - No",
"value": "0"
},
{
"op": "=",
"text": "1 - No",
"value": "1"
},
{
"op": "=",
"text": "2 - Yes",
"value": "2"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": true,
"colorValue": false,
"colors": [
"#d44a3a",
"rgba(237, 129, 40, 0.89)",
"#299c46"
],
"datasource": "$datasource",
"format": "none",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 2,
"w": 3,
"x": 0,
"y": 3
},
"id": 60,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"postfix": " Cores",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"expr": "kube_pod_container_resource_requests_cpu_cores{container=~\"$containername\",pod=~'$podname'}",
"format": "time_series",
"instant": false,
"intervalFactor": 1,
"legendFormat": "",
"refId": "A"
}
],
"thresholds": "0,0.0001",
"title": "CPU Request",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": true,
"colorValue": false,
"colors": [
"#d44a3a",
"rgba(237, 129, 40, 0.89)",
"#299c46"
],
"datasource": "$datasource",
"format": "none",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 2,
"w": 3,
"x": 3,
"y": 3
},
"id": 59,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"postfix": " Cores",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"expr": "kube_pod_container_resource_limits_cpu_cores{container=~\"$containername\",pod=~'$podname'}",
"format": "time_series",
"instant": false,
"intervalFactor": 1,
"legendFormat": "",
"refId": "A"
}
],
"thresholds": "0,0.0001",
"title": "CPU Limit",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 5
},
"id": 39,
"panels": [],
"title": "Containers",
"type": "row"
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"decimals": 2,
"editable": true,
"error": false,
"fill": 0,
"grid": {},
"gridPos": {
"h": 6,
"w": 11,
"x": 0,
"y": 6
},
"id": 27,
"isNew": true,
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"max": false,
"min": false,
"rightSide": false,
"show": true,
"sideWidth": 200,
"sortDesc": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 2,
"links": [],
"nullPointMode": "connected",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "Limit",
"color": "#0a437c",
"fill": 5,
"legend": false,
"linewidth": 0,
"zindex": -3
},
{
"alias": "Request",
"color": "#5195ce",
"fill": 5,
"legend": false,
"linewidth": 0,
"zindex": -2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum (\n container_memory_working_set_bytes{\n image!=\"\",name=~\"^k8s_.*\",\n container_name!=\"POD\",container_name=~\"$containername\",pod_name=~'$podname'\n }\n) \nby (container_name, pod_name)",
"format": "time_series",
"hide": false,
"interval": "10s",
"intervalFactor": 5,
"legendFormat": " {{ container_name }} >> Pod: {{ pod_name }} |",
"metric": "container_memory_usage:sort_desc",
"refId": "A",
"step": 10
},
{
"expr": "sum (\n container_memory_working_set_bytes{\n image!=\"\",name=~\"^k8s_.*\",\n container_name!=\"POD\",container_name=~\"$containername\",pod_name=~'$podname'\n }\n)\nby (kubernetes_io_hostname, name, image)",
"format": "time_series",
"hide": false,
"interval": "10s",
"intervalFactor": 5,
"legendFormat": "docker: {{ kubernetes_io_hostname }} | {{ image }} ({{ name }})",
"metric": "container_memory_usage:sort_desc",
"refId": "B",
"step": 10
},
{
"expr": "sum(\n kube_pod_container_resource_limits_memory_bytes{\n container=~\"$containername\"\n }\n)\nby (pod)",
"format": "time_series",
"hide": false,
"instant": false,
"intervalFactor": 5,
"legendFormat": "Limit",
"refId": "C"
},
{
"expr": "sum(\n kube_pod_container_resource_requests_memory_bytes{\n container=~\"$containername\"\n }\n)\nby (pod)",
"format": "time_series",
"hide": false,
"instant": false,
"intervalFactor": 5,
"legendFormat": "Request",
"refId": "D"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Containers memory usage",
"tooltip": {
"msResolution": false,
"shared": true,
"sort": 2,
"value_type": "cumulative"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "decbytes",
"label": null,
"logBase": 1,
"max": null,
"min": "0",
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"decimals": 3,
"editable": true,
"error": false,
"fill": 0,
"grid": {},
"gridPos": {
"h": 6,
"w": 11,
"x": 11,
"y": 6
},
"height": "",
"id": 24,
"isNew": true,
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"hideEmpty": false,
"hideZero": false,
"max": false,
"min": false,
"rightSide": false,
"show": true,
"sideWidth": null,
"sortDesc": false,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "connected",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "Request",
"color": "#1f78c1",
"fill": 5,
"legend": false,
"linewidth": 0,
"zindex": -3
},
{
"alias": "Limit",
"color": "#0a437c",
"fill": 5,
"legend": false,
"linewidth": 0,
"zindex": -1
},
{
"alias": "CPU Throttle",
"color": "#bf1b00"
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum (\n irate (container_cpu_usage_seconds_total{\n image!=\"\",name=~\"^k8s_.*\",\n container_name!=\"POD\",\n container_name=~\"$containername\",\n pod_name=~'$podname'}[5m])\n) \nby (container_name, pod_name)",
"format": "time_series",
"hide": false,
"interval": "",
"intervalFactor": 2,
"legendFormat": " {{ container_name }} >> pod: {{ pod_name }} ",
"metric": "container_cpu",
"refId": "A",
"step": 10
},
{
"expr": "kube_pod_container_resource_limits_cpu_cores{\n container=~\"$containername\",pod=~'$podname'\n}",
"format": "time_series",
"hide": false,
"interval": "",
"intervalFactor": 5,
"legendFormat": "Limit",
"refId": "C"
},
{
"expr": "kube_pod_container_resource_requests_cpu_cores{\n container=~\"$containername\",pod=~'$podname'\n}",
"format": "time_series",
"hide": false,
"interval": "",
"intervalFactor": 5,
"legendFormat": "Request",
"refId": "D"
},
{
"expr": "sum (\n irate (container_cpu_cfs_throttled_seconds_total{\n image!=\"\",name=~\"^k8s_.*\",\n container_name!=\"POD\",\n container_name=~\"$containername\",\n pod_name=~'$podname'}[5m])\n) \nby (container_name, pod_name)",
"format": "time_series",
"hide": false,
"intervalFactor": 2,
"legendFormat": "CPU Throttle",
"refId": "E"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Container CPU usage",
"tooltip": {
"msResolution": true,
"shared": true,
"sort": 2,
"value_type": "cumulative"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "percentunit",
"label": "cores",
"logBase": 1,
"max": null,
"min": "0",
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 12
},
"id": 66,
"panels": [],
"title": "Pod Orchestration",
"type": "row"
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"fill": 5,
"gridPos": {
"h": 7,
"w": 11,
"x": 0,
"y": 13
},
"id": 49,
"legend": {
"alignAsTable": false,
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 0,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": true,
"steppedLine": false,
"targets": [
{
"expr": "kube_pod_status_scheduled{pod=~\"$podname\",condition=\"true\"}",
"format": "time_series",
"intervalFactor": 10,
"legendFormat": "",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Pod Scheduled",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": "0",
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"fill": 1,
"gridPos": {
"h": 7,
"w": 11,
"x": 11,
"y": 13
},
"id": 51,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "kube_pod_container_status_restarts_total{container=~\"$containername\"}",
"format": "time_series",
"intervalFactor": 5,
"legendFormat": "{{container}}",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Pod Restarts",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": "0",
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 20
},
"id": 35,
"panels": [],
"title": "Pods",
"type": "row"
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"decimals": 2,
"editable": true,
"error": false,
"fill": 0,
"grid": {},
"gridPos": {
"h": 7,
"w": 11,
"x": 0,
"y": 21
},
"id": 25,
"isNew": true,
"legend": {
"alignAsTable": false,
"avg": true,
"current": true,
"max": false,
"min": false,
"rightSide": false,
"show": true,
"sideWidth": 200,
"sort": "current",
"sortDesc": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "connected",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum(container_memory_working_set_bytes{image!=\"\",name=~\"^k8s_.*$containername.*\",name!~'^k8s_POD.*',pod_name=~\"$podname\"}) without (beta_kubernetes_io_arch)",
"format": "time_series",
"interval": "10s",
"intervalFactor": 5,
"legendFormat": "{{container_name}}",
"metric": "container_memory_usage:sort_desc",
"refId": "A",
"step": 10
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Pods memory usage",
"tooltip": {
"msResolution": false,
"shared": true,
"sort": 2,
"value_type": "cumulative"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "bytes",
"label": null,
"logBase": 1,
"max": null,
"min": "0",
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"fill": 1,
"gridPos": {
"h": 7,
"w": 11,
"x": 11,
"y": 21
},
"id": 52,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum(kube_pod_container_status_terminated_reason{container=~\"$containername\"}) by (reason)",
"format": "time_series",
"intervalFactor": 5,
"legendFormat": "{{ reason }}",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Pod Termination Reasons",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": "0",
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"decimals": 2,
"editable": true,
"error": false,
"fill": 0,
"grid": {},
"gridPos": {
"h": 7,
"w": 11,
"x": 11,
"y": 28
},
"id": 16,
"isNew": true,
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"max": false,
"min": false,
"rightSide": false,
"show": true,
"sideWidth": 200,
"sortDesc": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "connected",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum(rate(container_network_receive_bytes_total{image!=\"\",name=~\"^k8s_.*$containername.*\",pod_name=~\"$podname\"}[5m])) by (pod_name,container_name)",
"format": "time_series",
"hide": false,
"interval": "10s",
"intervalFactor": 5,
"legendFormat": "Rx {{ pod_name }} {{container_name}} {{region}}",
"metric": "network",
"refId": "A",
"step": 10
},
{
"expr": "0 - sum (rate (container_network_transmit_bytes_total{image!=\"\",name=~\"^k8s_.*$containername.*\",pod_name=~\"$podname\"}[5m])) by (pod_name,container_name)",
"format": "time_series",
"hide": false,
"interval": "10s",
"intervalFactor": 5,
"legendFormat": "Tx {{ pod_name }} {{container_name}}",
"metric": "network",
"refId": "B",
"step": 10
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Pods network I/O (1m avg)",
"tooltip": {
"msResolution": false,
"shared": true,
"sort": 2,
"value_type": "cumulative"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "Bps",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
}
],
"refresh": false,
"schemaVersion": 16,
"style": "dark",
"tags": [
"kubernetes"
],
"templating": {
"list": [
{
"current": {
"text": "Prometheus",
"value": "Prometheus"
},
"hide": 0,
"label": null,
"name": "datasource",
"options": [],
"query": "prometheus",
"refresh": 1,
"regex": "",
"type": "datasource"
},
{
"allValue": ".*",
"current": {},
"datasource": "$datasource",
"hide": 0,
"includeAll": false,
"label": null,
"multi": false,
"name": "containername",
"options": [],
"query": "label_values(container_cpu_usage_seconds_total{image!=\"\",name=~\"^k8s_.*\",container_name!=\"POD\"},container_name)",
"refresh": 1,
"regex": "",
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allValue": ".*",
"current": {},
"datasource": "$datasource",
"hide": 0,
"includeAll": true,
"label": null,
"multi": true,
"name": "podname",
"options": [],
"query": "label_values(container_cpu_usage_seconds_total{image!=\"\",name=~\"^k8s_.*\",container_name=~\"$containername\"},pod_name)",
"refresh": 1,
"regex": "",
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
}
]
},
"time": {
"from": "now-1h",
"to": "now"
},
"timepicker": {
"refresh_intervals": [
"5s",
"10s",
"30s",
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
],
"time_options": [
"5m",
"15m",
"1h",
"6h",
"12h",
"24h",
"2d",
"7d",
"30d"
]
},
"timezone": "browser",
"title": "Kubernetes Container Details",
"uid": "qpvjpoNmk",
"version": 1
}
kubernetes:
json: |
{
"__inputs": [],
"__requires": [
{
"type": "grafana",
"id": "grafana",
"name": "Grafana",
"version": "5.1.3"
},
{
"type": "panel",
"id": "singlestat",
"name": "Singlestat",
"version": "5.0.0"
},
{
"type": "panel",
"id": "text",
"name": "Text",
"version": "5.0.0"
}
],
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"description": "Monitors Kubernetes cluster using Prometheus. Shows overall cluster CPU / Memory / Filesystem usage as well as individual pod, containers, systemd services statistics. Uses cAdvisor metrics only.",
"editable": true,
"gnetId": 315,
"graphTooltip": 1,
"id": null,
"iteration": 1529435082244,
"links": [],
"panels": [
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 0
},
"id": 88,
"panels": [],
"title": "",
"type": "row"
},
{
"content": "",
"gridPos": {
"h": 1,
"w": 12,
"x": 0,
"y": 1
},
"id": 64,
"links": [],
"mode": "markdown",
"title": "Masters",
"transparent": false,
"type": "text"
},
{
"content": "",
"gridPos": {
"h": 1,
"w": 12,
"x": 12,
"y": 1
},
"id": 65,
"links": [],
"mode": "markdown",
"title": "Nodes",
"type": "text"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"#299c46",
"rgba(237, 129, 40, 0.89)",
"#d44a3a"
],
"datasource": "$datasource",
"format": "none",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 3,
"w": 2,
"x": 0,
"y": 2
},
"id": 54,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"expr": "count(machine_memory_bytes{kubernetes_io_hostname=~\"^$Node$\",kubernetes_io_role=\"master\"})",
"format": "time_series",
"instant": true,
"intervalFactor": 1,
"refId": "A"
}
],
"thresholds": "",
"title": "Masters",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": true,
"colors": [
"rgba(50, 172, 45, 0.97)",
"rgba(237, 129, 40, 0.89)",
"rgba(245, 54, 54, 0.9)"
],
"datasource": "$datasource",
"editable": true,
"error": false,
"format": "percentunit",
"gauge": {
"maxValue": 1,
"minValue": 0,
"show": true,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 3,
"w": 8,
"x": 2,
"y": 2
},
"height": "180px",
"id": 4,
"interval": null,
"isNew": true,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"expr": "sum (container_memory_working_set_bytes{id=\"/\",kubernetes_io_hostname=~\"^$Node$\",kubernetes_io_role=\"master\"}) \n/ \nsum (machine_memory_bytes{kubernetes_io_hostname=~\"^$Node$\",kubernetes_io_role=\"master\"})",
"format": "time_series",
"hide": false,
"interval": "10s",
"intervalFactor": 1,
"refId": "A",
"step": 10
}
],
"thresholds": "0.65, 0.9",
"title": "Master memory usage",
"transparent": false,
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"rgba(50, 172, 45, 0.97)",
"rgba(237, 129, 40, 0.89)",
"rgba(245, 54, 54, 0.9)"
],
"datasource": "$datasource",
"decimals": 2,
"editable": true,
"error": false,
"format": "none",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 3,
"w": 2,
"x": 10,
"y": 2
},
"height": "1px",
"id": 90,
"interval": null,
"isNew": true,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"postfix": "",
"postfixFontSize": "30%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"expr": "sum(machine_cpu_cores{kubernetes_io_role='master'})",
"format": "time_series",
"instant": true,
"interval": "10s",
"intervalFactor": 1,
"refId": "A",
"step": 10
}
],
"thresholds": "",
"title": "Cores",
"type": "singlestat",
"valueFontSize": "50%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"#299c46",
"rgba(237, 129, 40, 0.89)",
"#d44a3a"
],
"datasource": "$datasource",
"format": "none",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 3,
"w": 2,
"x": 12,
"y": 2
},
"id": 55,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"expr": "count(machine_memory_bytes{kubernetes_io_hostname=~\"^$Node$\",kubernetes_io_role=\"node\"})",
"format": "time_series",
"instant": true,
"intervalFactor": 1,
"refId": "A"
}
],
"thresholds": "",
"title": "Nodes",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"rgba(50, 172, 45, 0.97)",
"rgba(237, 129, 40, 0.89)",
"rgba(245, 54, 54, 0.9)"
],
"datasource": "$datasource",
"decimals": 2,
"editable": true,
"error": false,
"format": "none",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 3,
"w": 2,
"x": 14,
"y": 2
},
"height": "1px",
"id": 12,
"interval": null,
"isNew": true,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"postfix": "",
"postfixFontSize": "30%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"expr": "sum(machine_cpu_cores{kubernetes_io_role='node'})",
"format": "time_series",
"instant": true,
"interval": "10s",
"intervalFactor": 1,
"refId": "A",
"step": 10
}
],
"thresholds": "",
"title": "Cores",
"type": "singlestat",
"valueFontSize": "50%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"rgba(50, 172, 45, 0.97)",
"rgba(237, 129, 40, 0.89)",
"rgba(245, 54, 54, 0.9)"
],
"datasource": "$datasource",
"decimals": 2,
"editable": true,
"error": false,
"format": "bytes",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 3,
"w": 2,
"x": 16,
"y": 2
},
"height": "1px",
"id": 10,
"interval": null,
"isNew": true,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"expr": "sum (machine_memory_bytes{kubernetes_io_hostname=~\"^$Node$\",kubernetes_io_role=\"node\",})",
"format": "time_series",
"interval": "10s",
"intervalFactor": 1,
"refId": "A",
"step": 10
}
],
"thresholds": "",
"title": "Total",
"type": "singlestat",
"valueFontSize": "50%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"rgba(50, 172, 45, 0.97)",
"rgba(237, 129, 40, 0.89)",
"rgba(245, 54, 54, 0.9)"
],
"datasource": "$datasource",
"decimals": 2,
"editable": true,
"error": false,
"format": "bytes",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 3,
"w": 2,
"x": 18,
"y": 2
},
"height": "1px",
"id": 89,
"interval": null,
"isNew": true,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"postfix": "",
"postfixFontSize": "20%",
"prefix": "",
"prefixFontSize": "20%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"expr": "sum (container_memory_working_set_bytes{id=\"/\",kubernetes_io_hostname=~\"^$Node$\",kubernetes_io_role=\"node\"})",
"format": "time_series",
"interval": "10s",
"intervalFactor": 1,
"refId": "A",
"step": 10
}
],
"thresholds": "",
"title": "Used",
"type": "singlestat",
"valueFontSize": "50%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": true,
"colors": [
"rgba(50, 172, 45, 0.97)",
"rgba(237, 129, 40, 0.89)",
"rgba(245, 54, 54, 0.9)"
],
"datasource": "$datasource",
"editable": true,
"error": false,
"format": "percentunit",
"gauge": {
"maxValue": 1,
"minValue": 0,
"show": true,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 3,
"w": 4,
"x": 20,
"y": 2
},
"height": "180px",
"id": 49,
"interval": null,
"isNew": true,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"expr": "sum (container_memory_working_set_bytes{id=\"/\",kubernetes_io_hostname=~\"^$Node$\",kubernetes_io_role=\"node\"})\n/\nsum (machine_memory_bytes{kubernetes_io_hostname=~\"^$Node$\",kubernetes_io_role=\"node\"})",
"format": "time_series",
"hide": false,
"instant": true,
"interval": "10s",
"intervalFactor": 1,
"refId": "A",
"step": 10
}
],
"thresholds": "0.65, 0.90",
"title": "Node memory usage",
"transparent": false,
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"collapsed": true,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 5
},
"id": 60,
"panels": [
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"fill": 0,
"gridPos": {
"h": 5,
"w": 6,
"x": 0,
"y": 6
},
"id": 47,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "avg(container_memory_working_set_bytes{id=\"/\",kubernetes_io_hostname=~\"^$Node$\",\nkubernetes_io_role=\"master\"}) by (instance)",
"format": "time_series",
"hide": false,
"intervalFactor": 5,
"legendFormat": "{{ instance }}",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Master Memory",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "decbytes",
"label": null,
"logBase": 1,
"max": null,
"min": "0",
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"decimals": 3,
"editable": true,
"error": false,
"fill": 0,
"grid": {},
"gridPos": {
"h": 5,
"w": 6,
"x": 6,
"y": 6
},
"height": "",
"id": 17,
"isNew": true,
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"max": false,
"min": false,
"rightSide": true,
"show": false,
"sort": "current",
"sortDesc": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum (rate (container_cpu_usage_seconds_total{image!=\"\",name=~\"^k8s_.*\",kubernetes_io_role=\"master\"}[5m])) by (kubernetes_io_hostname,namespace)",
"format": "time_series",
"hide": false,
"interval": "",
"intervalFactor": 5,
"legendFormat": "{{namespace}} - {{kubernetes_io_hostname}}",
"metric": "container_cpu",
"refId": "A",
"step": 10
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Master CPU usage",
"tooltip": {
"msResolution": true,
"shared": true,
"sort": 2,
"value_type": "cumulative"
},
"transparent": false,
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "percentunit",
"label": "cores",
"logBase": 1,
"max": null,
"min": "0",
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"fill": 0,
"gridPos": {
"h": 5,
"w": 6,
"x": 12,
"y": 6
},
"id": 48,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "avg(container_memory_working_set_bytes{id=\"/\",kubernetes_io_hostname=~\"^$Node$\", kubernetes_io_role=\"node\"}) by (instance)",
"format": "time_series",
"hide": false,
"instant": false,
"intervalFactor": 5,
"legendFormat": "{{ instance }}",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Node Memory",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "decbytes",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"decimals": 3,
"editable": true,
"error": false,
"fill": 0,
"grid": {},
"gridPos": {
"h": 5,
"w": 6,
"x": 18,
"y": 6
},
"height": "",
"id": 78,
"isNew": true,
"legend": {
"alignAsTable": false,
"avg": false,
"current": false,
"max": false,
"min": false,
"rightSide": false,
"show": false,
"sort": "current",
"sortDesc": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum (\n rate (\n container_cpu_usage_seconds_total{id=\"/\",kubernetes_io_role=\"node\"}[5m]\n )\n) by (kubernetes_io_hostname)",
"format": "time_series",
"hide": false,
"interval": "",
"intervalFactor": 5,
"legendFormat": "{{kubernetes_io_hostname}}",
"metric": "container_cpu",
"refId": "A",
"step": 10
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Node CPU usage",
"tooltip": {
"msResolution": true,
"shared": true,
"sort": 2,
"value_type": "cumulative"
},
"transparent": false,
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "percentunit",
"label": "cores",
"logBase": 1,
"max": null,
"min": "0",
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
}
],
"title": "Memory",
"type": "row"
},
{
"collapsed": true,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 6
},
"id": 35,
"panels": [
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"decimals": 3,
"editable": true,
"error": false,
"fill": 0,
"grid": {},
"gridPos": {
"h": 5,
"w": 6,
"x": 0,
"y": 7
},
"height": "",
"id": 86,
"isNew": true,
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"hideEmpty": false,
"hideZero": false,
"max": false,
"min": false,
"rightSide": false,
"show": true,
"sideWidth": null,
"sortDesc": false,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum (\n rate (container_cpu_cfs_throttled_seconds_total{\n image!=\"\",name=~\"^k8s_.*\",\n container_name!=\"POD\",\n kubernetes_io_role=\"master\"}[2m])\n) by (instance, namespace)",
"format": "time_series",
"hide": false,
"intervalFactor": 5,
"legendFormat": "{{namespace}} - {{instance}}",
"refId": "E"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Master CPU Throttle",
"tooltip": {
"msResolution": true,
"shared": true,
"sort": 2,
"value_type": "cumulative"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"decimals": 2,
"format": "percentunit",
"label": "cores",
"logBase": 1,
"max": null,
"min": "0",
"show": true
},
{
"format": "none",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": true,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"decimals": 3,
"editable": true,
"error": false,
"fill": 0,
"grid": {},
"gridPos": {
"h": 5,
"w": 6,
"x": 6,
"y": 7
},
"height": "",
"id": 81,
"isNew": true,
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"hideEmpty": false,
"hideZero": false,
"max": false,
"min": false,
"rightSide": false,
"show": true,
"sideWidth": null,
"sortDesc": false,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "Request",
"color": "#1f78c1",
"fill": 5,
"legend": false,
"linewidth": 0,
"zindex": -3
},
{
"alias": "Limit",
"color": "#0a437c",
"fill": 5,
"legend": false,
"linewidth": 0,
"zindex": -1
},
{
"alias": "CPU Throttle",
"color": "#bf1b00"
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum (\n rate (container_cpu_cfs_throttled_seconds_total{\n image!=\"\",\n container_name!=\"POD\",\n kubernetes_io_role=~\"master\"}[5m])\n) \nby (namespace, pod_name)",
"format": "time_series",
"hide": false,
"intervalFactor": 5,
"legendFormat": "{{namespace}} - {{pod_name}}",
"refId": "E"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Master Pod CPU Throttling",
"tooltip": {
"msResolution": true,
"shared": true,
"sort": 2,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "percentunit",
"label": "cores",
"logBase": 1,
"max": null,
"min": "0",
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"decimals": 3,
"editable": true,
"error": false,
"fill": 0,
"grid": {},
"gridPos": {
"h": 5,
"w": 6,
"x": 12,
"y": 7
},
"height": "",
"id": 85,
"isNew": true,
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"hideEmpty": false,
"hideZero": false,
"max": false,
"min": false,
"rightSide": false,
"show": true,
"sideWidth": null,
"sortDesc": false,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum (\n rate (container_cpu_cfs_throttled_seconds_total{\n image!=\"\",name=~\"^k8s_.*\",\n container_name!=\"POD\",\n kubernetes_io_role=\"node\"}[2m])\n) by (instance, namespace)",
"format": "time_series",
"hide": false,
"intervalFactor": 5,
"legendFormat": "{{namespace }} - {{instance}}",
"refId": "E"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Node CPU Throttling",
"tooltip": {
"msResolution": true,
"shared": true,
"sort": 2,
"value_type": "cumulative"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"decimals": 2,
"format": "percentunit",
"label": "cores",
"logBase": 1,
"max": null,
"min": "0",
"show": true
},
{
"format": "none",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": true,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"decimals": 3,
"editable": true,
"error": false,
"fill": 0,
"grid": {},
"gridPos": {
"h": 5,
"w": 6,
"x": 18,
"y": 7
},
"height": "",
"id": 80,
"isNew": true,
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"hideEmpty": false,
"hideZero": false,
"max": false,
"min": false,
"rightSide": false,
"show": true,
"sideWidth": null,
"sort": "current",
"sortDesc": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "Request",
"color": "#1f78c1",
"fill": 5,
"legend": false,
"linewidth": 0,
"zindex": -3
},
{
"alias": "Limit",
"color": "#0a437c",
"fill": 5,
"legend": false,
"linewidth": 0,
"zindex": -1
},
{
"alias": "CPU Throttle",
"color": "#bf1b00"
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum (\n rate (container_cpu_cfs_throttled_seconds_total{\n image!=\"\",name=~\"^k8s_.*\",\n container_name!=\"POD\",\n namespace=~\"$namespace\",\n kubernetes_io_role=~\"node\"}[5m])\n) \nby (namespace, pod_name)",
"format": "time_series",
"hide": false,
"intervalFactor": 5,
"legendFormat": "{{namespace}} - {{pod_name}}",
"refId": "E"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Node Pod CPU Throttling",
"tooltip": {
"msResolution": true,
"shared": true,
"sort": 2,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "percentunit",
"label": "cores",
"logBase": 1,
"max": null,
"min": "0",
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
}
],
"title": "CPU",
"type": "row"
},
{
"collapsed": true,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 7
},
"id": 67,
"panels": [
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"decimals": 2,
"editable": true,
"error": false,
"fill": 1,
"grid": {},
"gridPos": {
"h": 5,
"w": 12,
"x": 0,
"y": 8
},
"height": "200px",
"id": 77,
"isNew": true,
"legend": {
"alignAsTable": false,
"avg": true,
"current": true,
"max": false,
"min": false,
"rightSide": false,
"show": false,
"sideWidth": 200,
"sort": "current",
"sortDesc": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 2,
"links": [],
"nullPointMode": "connected",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "Total Node Received",
"color": "#fce2de",
"fill": 1,
"linewidth": 0,
"zindex": -3
},
{
"alias": "Total Node Sent",
"color": "#fce2de",
"fill": 1,
"linewidth": 0,
"zindex": -3
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum (rate (container_network_receive_bytes_total{kubernetes_io_hostname=~\"^$Node$\",kubernetes_io_role=\"master\"}[5m])) by (kubernetes_io_hostname)",
"format": "time_series",
"interval": "10s",
"intervalFactor": 10,
"legendFormat": "Received {{kubernetes_io_hostname}}",
"metric": "network",
"refId": "A",
"step": 10
},
{
"expr": "- sum (rate (container_network_transmit_bytes_total{kubernetes_io_hostname=~\"^$Node$\",kubernetes_io_role=\"master\"}[5m])) by (kubernetes_io_hostname)",
"format": "time_series",
"interval": "10s",
"intervalFactor": 10,
"legendFormat": "Sent {{kubernetes_io_hostname}}",
"metric": "network",
"refId": "B",
"step": 10
},
{
"expr": "sum (rate (container_network_receive_bytes_total{kubernetes_io_hostname=~\"^$Node$\",kubernetes_io_role=\"master\"}[5m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "Total Node Received",
"refId": "C"
},
{
"expr": "- sum (rate (container_network_transmit_bytes_total{kubernetes_io_hostname=~\"^$Node$\",kubernetes_io_role=\"master\"}[5m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "Total Node Sent",
"refId": "D"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Master Network I/O pressure",
"tooltip": {
"msResolution": false,
"shared": true,
"sort": 0,
"value_type": "cumulative"
},
"transparent": false,
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "bps",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "Bps",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"decimals": 2,
"editable": true,
"error": false,
"fill": 1,
"grid": {},
"gridPos": {
"h": 5,
"w": 12,
"x": 12,
"y": 8
},
"height": "200px",
"id": 52,
"isNew": true,
"legend": {
"alignAsTable": false,
"avg": true,
"current": true,
"max": false,
"min": false,
"rightSide": false,
"show": false,
"sideWidth": 200,
"sort": "current",
"sortDesc": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 2,
"links": [],
"nullPointMode": "connected",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "Total Node Received",
"color": "#fce2de",
"fill": 1,
"linewidth": 0,
"zindex": -3
},
{
"alias": "Total Node Sent",
"color": "#fce2de",
"fill": 1,
"linewidth": 0,
"zindex": -3
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum (rate (container_network_receive_bytes_total{kubernetes_io_hostname=~\"^$Node$\",kubernetes_io_role=\"node\"}[5m])) by (kubernetes_io_hostname)",
"format": "time_series",
"interval": "10s",
"intervalFactor": 10,
"legendFormat": "Received {{kubernetes_io_hostname}}",
"metric": "network",
"refId": "A",
"step": 10
},
{
"expr": "- sum (rate (container_network_transmit_bytes_total{kubernetes_io_hostname=~\"^$Node$\",kubernetes_io_role=\"node\"}[5m])) by (kubernetes_io_hostname)",
"format": "time_series",
"interval": "10s",
"intervalFactor": 10,
"legendFormat": "Sent {{kubernetes_io_hostname}}",
"metric": "network",
"refId": "B",
"step": 10
},
{
"expr": "sum (rate (container_network_receive_bytes_total{kubernetes_io_hostname=~\"^$Node$\",kubernetes_io_role=\"node\"}[5m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "Total Node Received",
"refId": "C"
},
{
"expr": "- sum (rate (container_network_transmit_bytes_total{kubernetes_io_hostname=~\"^$Node$\",kubernetes_io_role=\"node\"}[5m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "Total Node Sent",
"refId": "D"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Node Network I/O pressure",
"tooltip": {
"msResolution": false,
"shared": true,
"sort": 0,
"value_type": "cumulative"
},
"transparent": false,
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "bps",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "Bps",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
}
],
"title": "Network",
"type": "row"
},
{
"collapsed": true,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 8
},
"id": 62,
"panels": [
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"fill": 2,
"gridPos": {
"h": 6,
"w": 4,
"x": 0,
"y": 9
},
"id": 72,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "Scaled Up",
"color": "#508642"
},
{
"alias": "Scaled Down (empty)",
"color": "#bf1b00",
"transform": "negative-Y"
},
{
"alias": "Scaled Down (underutilized)",
"color": "#e0752d",
"transform": "negative-Y"
},
{
"alias": "Unschedulable Pods",
"color": "rgb(255, 198, 0)",
"fill": 10,
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "increase(cluster_autoscaler_scaled_down_nodes_total[5m])",
"format": "time_series",
"hide": false,
"intervalFactor": 2,
"legendFormat": "Scaled Down ({{reason}})",
"refId": "A"
},
{
"expr": "increase(cluster_autoscaler_scaled_up_nodes_total[5m])",
"format": "time_series",
"hide": false,
"intervalFactor": 2,
"legendFormat": "Scaled Up",
"refId": "B"
},
{
"expr": "cluster_autoscaler_unschedulable_pods_count",
"format": "time_series",
"hide": false,
"intervalFactor": 2,
"legendFormat": "Unschedulable Pods",
"refId": "C"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "AutoScaler",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "none",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"fill": 0,
"gridPos": {
"h": 6,
"w": 4,
"x": 4,
"y": 9
},
"id": 71,
"legend": {
"avg": false,
"current": false,
"hideEmpty": true,
"hideZero": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "cluster_autoscaler_nodes_count",
"format": "time_series",
"intervalFactor": 2,
"legendFormat": "{{state}}",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Instance States",
"tooltip": {
"shared": true,
"sort": 2,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "none",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"fill": 0,
"gridPos": {
"h": 6,
"w": 4,
"x": 8,
"y": 9
},
"id": 76,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 3,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "up{job=~\"prometheus\"}",
"format": "time_series",
"intervalFactor": 5,
"legendFormat": "{{job}}",
"refId": "A"
}
],
"thresholds": [
{
"colorMode": "critical",
"fill": true,
"line": true,
"op": "lt",
"value": 0.5,
"yaxis": "right"
}
],
"timeFrom": null,
"timeShift": null,
"title": "Prometheus UP",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"decimals": 0,
"format": "short",
"label": "DOWN <---> UP",
"logBase": 1,
"max": "1",
"min": "0",
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"decimals": null,
"fill": 1,
"gridPos": {
"h": 6,
"w": 4,
"x": 12,
"y": 9
},
"id": 58,
"legend": {
"alignAsTable": false,
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum(kube_pod_container_resource_requests_cpu_cores{namespace=~'$namespace'})\n/\nsum(machine_cpu_cores{kubernetes_io_role='node'}) by (namespace)",
"format": "time_series",
"hide": false,
"interval": "",
"intervalFactor": 5,
"legendFormat": "Requests",
"refId": "A"
},
{
"expr": "sum(kube_pod_container_resource_limits_cpu_cores{namespace=~'$namespace'})\n/\nsum(machine_cpu_cores{kubernetes_io_role='node'}) by (namespace)",
"format": "time_series",
"hide": false,
"instant": false,
"interval": "",
"intervalFactor": 5,
"legendFormat": "Limits",
"refId": "B"
}
],
"thresholds": [
{
"colorMode": "critical",
"fill": true,
"line": true,
"op": "gt",
"value": 1,
"yaxis": "left"
}
],
"timeFrom": null,
"timeShift": null,
"title": "Pod CPU Provisioning Ratios",
"tooltip": {
"shared": true,
"sort": 2,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"decimals": null,
"format": "percentunit",
"label": null,
"logBase": 1,
"max": "1",
"min": "0",
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": true,
"colors": [
"rgba(50, 172, 45, 0.97)",
"rgba(237, 129, 40, 0.89)",
"rgba(245, 54, 54, 0.9)"
],
"datasource": "$datasource",
"decimals": 2,
"editable": true,
"error": false,
"format": "percentunit",
"gauge": {
"maxValue": 1.2,
"minValue": 0,
"show": true,
"thresholdLabels": true,
"thresholdMarkers": true
},
"gridPos": {
"h": 6,
"w": 4,
"x": 16,
"y": 9
},
"height": "180px",
"id": 6,
"interval": null,
"isNew": true,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"expr": "sum(kube_pod_container_resource_requests_cpu_cores{namespace=~'$namespace'})\n/\nsum(machine_cpu_cores{kubernetes_io_role='node'})",
"format": "time_series",
"interval": "10s",
"intervalFactor": 1,
"refId": "A",
"step": 10
}
],
"thresholds": "0.75,0.90,1",
"title": "CPU Requests",
"transparent": false,
"type": "singlestat",
"valueFontSize": "50%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"decimals": null,
"fill": 1,
"gridPos": {
"h": 6,
"w": 4,
"x": 20,
"y": 9
},
"id": 57,
"legend": {
"alignAsTable": false,
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum(kube_pod_container_resource_requests_memory_bytes{namespace=~'$namespace'})\n/\nsum(machine_memory_bytes{kubernetes_io_role='node'})",
"format": "time_series",
"hide": false,
"intervalFactor": 5,
"legendFormat": "Requests",
"refId": "A"
},
{
"expr": "sum(kube_pod_container_resource_limits_memory_bytes{namespace=~'$namespace'})\n/\nsum(machine_memory_bytes{kubernetes_io_role='node'})",
"format": "time_series",
"hide": false,
"instant": false,
"intervalFactor": 5,
"legendFormat": "Limits",
"refId": "B"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Pod RAM Provisioning Ratios",
"tooltip": {
"shared": true,
"sort": 2,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"decimals": null,
"format": "percentunit",
"label": null,
"logBase": 1,
"max": "1",
"min": "0",
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
}
],
"title": "Provisioning",
"type": "row"
},
{
"collapsed": true,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 9
},
"id": 37,
"panels": [
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"decimals": 3,
"editable": true,
"error": false,
"fill": 0,
"grid": {},
"gridPos": {
"h": 7,
"w": 24,
"x": 0,
"y": 10
},
"height": "",
"id": 24,
"isNew": true,
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"hideEmpty": false,
"hideZero": false,
"max": false,
"min": false,
"rightSide": true,
"show": true,
"sideWidth": null,
"sort": "current",
"sortDesc": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 2,
"links": [],
"nullPointMode": "connected",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": true,
"targets": [
{
"expr": "sum (rate (container_cpu_usage_seconds_total{image!=\"\",name=~\"^k8s_.*\",container_name!=\"POD\",kubernetes_io_hostname=~\"^$Node$\"}[5m])) by (container_name, pod_name)",
"format": "time_series",
"hide": false,
"interval": "10s",
"intervalFactor": 10,
"legendFormat": "pod: {{ pod_name }} | {{ container_name }}",
"metric": "container_cpu",
"refId": "A",
"step": 10
},
{
"expr": "sum (rate (container_cpu_usage_seconds_total{image!=\"\",name!~\"^k8s_.*\",kubernetes_io_hostname=~\"^$Node$\"}[5m])) by (kubernetes_io_hostname, name, image)",
"format": "time_series",
"hide": false,
"interval": "10s",
"intervalFactor": 10,
"legendFormat": "docker: {{ kubernetes_io_hostname }} | {{ image }} ({{ name }})",
"metric": "container_cpu",
"refId": "B",
"step": 10
},
{
"expr": "sum (rate (container_cpu_usage_seconds_total{rkt_container_name!=\"\",kubernetes_io_hostname=~\"^$Node$\"}[5m])) by (kubernetes_io_hostname, rkt_container_name)",
"format": "time_series",
"interval": "10s",
"intervalFactor": 10,
"legendFormat": "rkt: {{ kubernetes_io_hostname }} | {{ rkt_container_name }}",
"metric": "container_cpu",
"refId": "C",
"step": 10
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Containers CPU usage",
"tooltip": {
"msResolution": true,
"shared": true,
"sort": 2,
"value_type": "cumulative"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "none",
"label": "cores",
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
}
],
"title": "Containers CPU usage",
"type": "row"
},
{
"collapsed": true,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 10
},
"id": 38,
"panels": [
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"decimals": 3,
"editable": true,
"error": false,
"fill": 0,
"grid": {},
"gridPos": {
"h": 13,
"w": 24,
"x": 0,
"y": 11
},
"id": 20,
"isNew": true,
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"max": false,
"min": false,
"rightSide": false,
"show": true,
"sort": "current",
"sortDesc": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 2,
"links": [],
"nullPointMode": "connected",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": true,
"targets": [
{
"expr": "sum (rate (container_cpu_usage_seconds_total{id!=\"/\",kubernetes_io_hostname=~\"^$Node$\"}[5m])) by (id)",
"format": "time_series",
"hide": false,
"interval": "10s",
"intervalFactor": 10,
"legendFormat": "{{ id }}",
"metric": "container_cpu",
"refId": "A",
"step": 10
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "All processes CPU usage",
"tooltip": {
"msResolution": true,
"shared": true,
"sort": 2,
"value_type": "cumulative"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "none",
"label": "cores",
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
}
],
"repeat": null,
"title": "All processes CPU usage",
"type": "row"
},
{
"collapsed": true,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 11
},
"id": 39,
"panels": [
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"decimals": 2,
"editable": true,
"error": false,
"fill": 0,
"grid": {},
"gridPos": {
"h": 7,
"w": 24,
"x": 0,
"y": 12
},
"id": 25,
"isNew": true,
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"max": false,
"min": false,
"rightSide": true,
"show": true,
"sideWidth": 200,
"sort": "current",
"sortDesc": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 2,
"links": [],
"nullPointMode": "connected",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": true,
"targets": [
{
"expr": "sum (container_memory_working_set_bytes{image!=\"\",name=~\"^k8s_.*\",kubernetes_io_hostname=~\"^$Node$\"}) by (pod_name)",
"format": "time_series",
"interval": "10s",
"intervalFactor": 5,
"legendFormat": "{{ pod_name }}",
"metric": "container_memory_usage:sort_desc",
"refId": "A",
"step": 10
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Pods memory usage",
"tooltip": {
"msResolution": false,
"shared": true,
"sort": 2,
"value_type": "cumulative"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "bytes",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
}
],
"title": "Pods memory usage",
"type": "row"
},
{
"collapsed": true,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 12
},
"id": 41,
"panels": [
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"decimals": 2,
"editable": true,
"error": false,
"fill": 0,
"grid": {},
"gridPos": {
"h": 7,
"w": 24,
"x": 0,
"y": 13
},
"id": 27,
"isNew": true,
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"max": false,
"min": false,
"rightSide": true,
"show": true,
"sideWidth": 200,
"sort": "current",
"sortDesc": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 2,
"links": [],
"nullPointMode": "connected",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": true,
"targets": [
{
"expr": "sum (container_memory_working_set_bytes{image!=\"\",name=~\"^k8s_.*\",container_name!=\"POD\",kubernetes_io_hostname=~\"^$Node$\"}) by (container_name, pod_name)",
"format": "time_series",
"interval": "10s",
"intervalFactor": 5,
"legendFormat": "pod: {{ pod_name }} | {{ container_name }}",
"metric": "container_memory_usage:sort_desc",
"refId": "A",
"step": 10
},
{
"expr": "sum (container_memory_working_set_bytes{image!=\"\",name!~\"^k8s_.*\",kubernetes_io_hostname=~\"^$Node$\"}) by (kubernetes_io_hostname, name, image)",
"format": "time_series",
"interval": "10s",
"intervalFactor": 5,
"legendFormat": "docker: {{ kubernetes_io_hostname }} | {{ image }} ({{ name }})",
"metric": "container_memory_usage:sort_desc",
"refId": "B",
"step": 10
},
{
"expr": "sum (container_memory_working_set_bytes{rkt_container_name!=\"\",kubernetes_io_hostname=~\"^$Node$\"}) by (kubernetes_io_hostname, rkt_container_name)",
"format": "time_series",
"interval": "10s",
"intervalFactor": 5,
"legendFormat": "rkt: {{ kubernetes_io_hostname }} | {{ rkt_container_name }}",
"metric": "container_memory_usage:sort_desc",
"refId": "C",
"step": 10
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Containers memory usage",
"tooltip": {
"msResolution": false,
"shared": true,
"sort": 2,
"value_type": "cumulative"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "bytes",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
}
],
"title": "Containers memory usage",
"type": "row"
},
{
"collapsed": true,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 13
},
"id": 42,
"panels": [
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"decimals": 2,
"editable": true,
"error": false,
"fill": 0,
"grid": {},
"gridPos": {
"h": 13,
"w": 24,
"x": 0,
"y": 14
},
"id": 28,
"isNew": true,
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"max": false,
"min": false,
"rightSide": false,
"show": true,
"sideWidth": 200,
"sort": "current",
"sortDesc": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 2,
"links": [],
"nullPointMode": "connected",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": true,
"targets": [
{
"expr": "sum (container_memory_working_set_bytes{id!=\"/\",kubernetes_io_hostname=~\"^$Node$\"}) by (id)",
"format": "time_series",
"interval": "10s",
"intervalFactor": 1,
"legendFormat": "{{ id }}",
"metric": "container_memory_usage:sort_desc",
"refId": "A",
"step": 10
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "All processes memory usage",
"tooltip": {
"msResolution": false,
"shared": true,
"sort": 2,
"value_type": "cumulative"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "bytes",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
}
],
"title": "All processes memory usage",
"type": "row"
}
],
"refresh": "5m",
"schemaVersion": 16,
"style": "dark",
"tags": [
"kubernetes"
],
"templating": {
"list": [
{
"current": {
"text": "Prometheus",
"value": "Prometheus"
},
"hide": 0,
"label": null,
"name": "datasource",
"options": [],
"query": "prometheus",
"refresh": 1,
"regex": "",
"type": "datasource"
},
{
"allValue": ".*",
"current": {},
"datasource": "$datasource",
"hide": 0,
"includeAll": true,
"label": null,
"multi": false,
"name": "Node",
"options": [],
"query": "label_values(kubernetes_build_info, kubernetes_io_hostname)",
"refresh": 2,
"regex": "",
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allValue": ".*",
"current": {},
"datasource": "$datasource",
"hide": 0,
"includeAll": true,
"label": null,
"multi": true,
"name": "namespace",
"options": [],
"query": "label_values(namespace)",
"refresh": 2,
"regex": "",
"sort": 5,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
}
]
},
"time": {
"from": "now-30m",
"to": "now"
},
"timepicker": {
"refresh_intervals": [
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
],
"time_options": [
"5m",
"15m",
"1h",
"6h",
"12h",
"24h",
"2d",
"7d",
"30d"
]
},
"timezone": "browser",
"title": "Kubernetes",
"uid": "N2Q_tTNmk",
"version": 5
}
go-services:
json: |
{
"__inputs": [],
"__requires": [
{
"type": "grafana",
"id": "grafana",
"name": "Grafana",
"version": "5.1.3"
},
{
"type": "panel",
"id": "graph",
"name": "Graph",
"version": "5.0.0"
},
{
"type": "panel",
"id": "singlestat",
"name": "Singlestat",
"version": "5.0.0"
}
],
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"gnetId": null,
"graphTooltip": 0,
"id": null,
"iteration": 1529435791391,
"links": [],
"panels": [
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 0
},
"id": 27,
"panels": [],
"repeat": null,
"title": "Service uptimes",
"type": "row"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": true,
"colors": [
"rgba(31, 153, 186, 0.97)",
"rgb(30, 198, 44)",
"rgba(223, 183, 33, 0.9)"
],
"datasource": "$datasource",
"decimals": 2,
"description": "Minimum (youngest instance) service uptime",
"format": "s",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 4,
"w": 4.8,
"x": 0,
"y": 1
},
"id": 6,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"minSpan": 2,
"nullPointMode": "connected",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"repeat": "service",
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"expr": "min(time() - process_start_time_seconds{job='$service'}) by (job)",
"format": "time_series",
"intervalFactor": 5,
"legendFormat": "",
"refId": "A",
"step": 600
}
],
"thresholds": "",
"title": "$service uptime",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 5
},
"id": 28,
"panels": [],
"repeat": null,
"title": "Dashboard Row",
"type": "row"
},
{
"aliasColors": {},
"bars": true,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"decimals": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 6
},
"id": 3,
"legend": {
"alignAsTable": false,
"avg": false,
"current": false,
"hideEmpty": true,
"hideZero": true,
"max": false,
"min": false,
"rightSide": false,
"show": false,
"total": false,
"values": false
},
"lines": false,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "avg(time() - process_start_time_seconds{job=~'$service'}) by (job, region)",
"format": "time_series",
"hide": false,
"interval": "",
"intervalFactor": 5,
"legendFormat": "{{job}}",
"refId": "B",
"step": 25
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Go Services Uptime",
"tooltip": {
"shared": false,
"sort": 0,
"value_type": "individual"
},
"transparent": false,
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "series",
"name": null,
"show": true,
"values": [
"current"
]
},
"yaxes": [
{
"format": "dtdurations",
"label": "Uptime",
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "none",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 14
},
"id": 29,
"panels": [],
"repeat": null,
"title": "Dashboard Row",
"type": "row"
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"fill": 1,
"gridPos": {
"h": 7,
"w": 12,
"x": 0,
"y": 15
},
"id": 1,
"legend": {
"avg": true,
"current": false,
"hideEmpty": true,
"hideZero": true,
"max": false,
"min": false,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum(go_goroutines{job=~'$service'}) by (job)",
"format": "time_series",
"hide": false,
"interval": "",
"intervalFactor": 5,
"legendFormat": "{{job}}",
"refId": "A",
"step": 50
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Total No. of Go Routines",
"tooltip": {
"shared": true,
"sort": 2,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "none",
"label": "# of Go Routines",
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"fill": 1,
"gridPos": {
"h": 7,
"w": 12,
"x": 12,
"y": 15
},
"id": 2,
"legend": {
"avg": true,
"current": false,
"hideEmpty": true,
"hideZero": true,
"max": false,
"min": false,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum(process_open_fds{job=~'$service'}) by (job)",
"format": "time_series",
"hide": false,
"interval": "",
"intervalFactor": 5,
"legendFormat": "{{job}}",
"refId": "A",
"step": 50
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Total No. of Open File Descriptors",
"tooltip": {
"shared": true,
"sort": 2,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "none",
"label": "# of File Descriptors",
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 22
},
"id": 30,
"panels": [],
"repeat": null,
"title": "Dashboard Row",
"type": "row"
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"fill": 1,
"gridPos": {
"h": 7,
"w": 12,
"x": 0,
"y": 23
},
"id": 5,
"legend": {
"avg": true,
"current": false,
"hideEmpty": true,
"hideZero": true,
"max": false,
"min": false,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum(process_virtual_memory_bytes{job=~'$service'}) by (job)",
"format": "time_series",
"hide": false,
"interval": "",
"intervalFactor": 5,
"legendFormat": "{{job}}",
"refId": "A",
"step": 50
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Virtual Memory Usage",
"tooltip": {
"shared": true,
"sort": 2,
"value_type": "individual"
},
"transparent": false,
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "decbytes",
"label": "Virtual Memory",
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": "",
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"fill": 1,
"gridPos": {
"h": 7,
"w": 12,
"x": 12,
"y": 23
},
"id": 4,
"legend": {
"avg": true,
"current": false,
"hideEmpty": true,
"hideZero": true,
"max": false,
"min": false,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum(process_resident_memory_bytes{job=~'$service'}) by (job)",
"format": "time_series",
"hide": false,
"interval": "",
"intervalFactor": 5,
"legendFormat": "{{job}}",
"refId": "A",
"step": 50
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Resident Memory Usage (RAM)",
"tooltip": {
"shared": true,
"sort": 2,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "decbytes",
"label": "Resident Memory (RAM)",
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": "",
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
}
],
"refresh": false,
"schemaVersion": 16,
"style": "dark",
"tags": [
"go",
"service",
"docker"
],
"templating": {
"list": [
{
"current": {
"text": "Prometheus",
"value": "Prometheus"
},
"hide": 0,
"label": "Data Source",
"name": "datasource",
"options": [],
"query": "prometheus",
"refresh": 1,
"regex": "",
"type": "datasource"
},
{
"allValue": null,
"current": {},
"datasource": "$datasource",
"hide": 0,
"includeAll": true,
"label": "Service",
"multi": true,
"name": "service",
"options": [],
"query": "label_values(go_goroutines, job)",
"refresh": 2,
"regex": "",
"sort": 1,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
}
]
},
"time": {
"from": "now-3h",
"to": "now"
},
"timepicker": {
"refresh_intervals": [
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
],
"time_options": [
"5m",
"15m",
"1h",
"6h",
"12h",
"24h",
"2d",
"7d",
"30d"
]
},
"timezone": "",
"title": "Go Services",
"uid": "v-lW2oNmz",
"version": 3
}メトリックの表示
次のコマンドを実行して管理者ユーザーのパスワードを取得します。
kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo同じシェルで、次のコマンドを実行して Grafana の URL を取得します。
export POD_NAME=$(kubectl get pods --namespace default -l "app=grafana,release=grafana" -o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:3000 to access grafana"
export GRAFANA_PASSWORD=$(kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo)
echo "Login as admin:$GRAFANA_PASSWORD"
kubectl port-forward $POD_NAME 3000