I have 2 versions of an app running in istio with pod labels of version=0.0.1 and 0.0.2. I am running the prometheus adapter to pull the records from prom as part of the custom metrics aggregation.
I want to be able to scale each deployment separately, but I'm not sure if I've set that up to pull the metric correctly.
First, here is all the relevant information.
HPA definition
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: translate-deployment-v2-hpa
spec:
minReplicas: 1
maxReplicas: 10
metrics:
- type: Pods
pods:
metric:
name: istio_requests_per_second
# selector: {matchLabels: {destination_version: 0.0.2}}
target:
type: AverageValue
averageValue: 10
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: translate-deployment-v2
Promethues Adapter confi
prometheus:
url: http://prometheus.istio-system.svc.cluster.local
rules:
default: false
custom:
# this rule matches cumulative cAdvisor metrics measured in seconds
- seriesQuery: 'istio_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}'
seriesFilters: []
resources:
# template: <<.Resource>>
# skip specifying generic resource<->label mappings, and just
# attach only pod and namespace resources by mapping label names to group-resources
overrides:
kubernetes_namespace: {resource: "namespace"}
kubernetes_pod_name: {resource: "pod"}
# specify that the `container_` and `_seconds_total` suffixes should be removed.
# this also introduces an implicit filter on metric family names
name:
# we use the value of the capture group implicitly as the API name
# we could also explicitly write `as: "$1"`
matches: "^(.*)_total"
as: "${1}_per_second"
# matches: ""
# as: ""
# specify how to construct a query to fetch samples for a given series
# This is a Go template where the `.Series` and `.LabelMatchers` string values
# are available, and the delimiters are `<<` and `>>` to avoid conflicts with
# the prometheus query language
metricsQuery: "sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)"
The istio/envoy metric which is being pulled, istio_requests_total which I'm renaming to istio_requests_total
istio_requests_total{app="istio-ingressgateway",chart="gateways",connection_security_policy="unknown",destination_app="translate-pod",destination_canonical_revision="0.0.1",destination_canonical_service="translate-pod",destination_principal="spiffe://cluster.local/ns/default/sa/default",destination_service="translate-service.default.svc.cluster.local",destination_service_name="translate-service",destination_service_namespace="default",destination_version="0.0.1",destination_workload="translate-deployment",destination_workload_namespace="default",heritage="Tiller",install_operator_istio_io_owning_resource="unknown",instance="172.17.0.5:15020",istio="ingressgateway",istio_io_rev="default",job="kubernetes-pods",kubernetes_namespace="istio-system",kubernetes_pod_name="istio-ingressgateway-6cfd75fc57-flmhp",operator_istio_io_component="IngressGateways",pod_template_hash="6cfd75fc57",release="istio",reporter="source",request_protocol="http",response_code="200",response_flags="-",service_istio_io_canonical_name="istio-ingressgateway",service_istio_io_canonical_revision="latest",source_app="istio-ingressgateway",source_canonical_revision="latest",source_canonical_service="istio-ingressgateway",source_principal="spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account",source_version="unknown",source_workload="istio-ingressgateway",source_workload_namespace="istio-system"}
Finally the query being shown in the prometheus-adapter logs which is coming from the HPA to pull the metric
I0108 19:58:32.125963 1 httplog.go:89] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/istio_requests_per_second?labelSelector=app%3Dtranslate-pod%2Cversion%3D0.0.2" latency="9.604608ms" userAgent="kube-controller-manager/v1.19.2 (linux/amd64) kubernetes/f574309/system:serviceaccount:kube-system:horizontal-pod-autoscaler" srcIP="172.17.0.1:56526" resp=200
I am basing my HPA on the istio_requests_total metric. In the query from the HPA there are two labelSelectors being sent app=translate-pod and version=0.0.2, but neither of these are present in the istio_requests_total metrics set of labels ? So what is the purpose of the HPA sending this, and is it actually being used anywhere ?
Also, in my HPA definition above, if I uncomment the selector, then the query sent from the HPA is changed to the following and has another added query param of metricLabelSelector, but this also doesn't change the result and this modified query with the additional param and the one above both return the same value. So again, what is the purpose of the selector in this context and how does one use it ?
I0108 20:05:53.121384 1 httplog.go:89] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/istio_requests_per_second?labelSelector=app%3Dtranslate-pod%2Cversion%3D0.0.2&metricLabelSelector=destination_version%3D0.0.2" latency="8.687004ms" userAgent="kube-controller-manager/v1.19.2 (linux/amd64) kubernetes/f574309/system:serviceaccount:kube-system:horizontal-pod-autoscaler" srcIP="172.17.0.1:56526" resp=200
What I want to be able to do is to scale the 0.0.1 version and 0.0.2 versions separately based on load, and to be able to pull the istio_request_metric correctly based on some filter, I'm not sure if what I am doing is correct, even though it returns values and actually scales currently, for each version.
In general, I dont understand how to filter a metric based on labels that you pass in from the HPA or if that is even possible. I can see the HPA passing in labels which are set on the pods the HPA is going to scale, but those don't match any labels in the metric I'm pulling so I know something is off.
