Changes

v1.77.0

Released at 05-05-2022

• FEATURE: vmagent: add support for sending data to remote storage with AWS sigv4 authorization. See this feature request.

• FEATURE: vmagent: allow filtering targets by target url and by target labels with time series selector on http://vmagent:8429/targets page. This may be useful when vmagent scrapes big number of targets. See this feature request.

• FEATURE: vmagent: reduce -promscrape.config reload duration when the config contains big number of jobs (aka scrape_config sections) and only a few of them are changed. Previously all the jobs were restarted. Now only the jobs with changed configs are restarted. This should reduce the probability of data miss because of slow config reload. See this issue.

• FEATURE: vmagent: improve service discovery speed for big number of scrape targets. This should help when vmagent discovers big number of targets (e.g. thousands) in Kubernetes cluster. The service discovery speed now should scale with the number of CPU cores available to vmagent.

• FEATURE: vmagent: add ability to attach node-level labels and annotations to discovered Kubernetes pod targets in the same way as Prometheus 2.35 does. See this feature request and this pull request.

• FEATURE: vmagent: add support for tls_config and proxy_url options at oauth2 section in the same way as Prometheus does. See oauth2 docs.

• FEATURE: vmagent: add support for min_version option at tls_config section in the same way as Prometheus does. See tls_config docs.

• FEATURE: vmagent: expose vmagent_remotewrite_rate_limit metric at http://vmagent:8429/metrics, which can be used for alerting rules such as rate(vmagent_remotewrite_conn_bytes_written_total) / vmagent_remotewrite_rate_limit > 0.8 when -remoteWrite.rateLimit command-line flag is set. See this pull request.

• FEATURE: vmalert: add support for DNS-based discovery for notifiers in the same way as Prometheus does (aka dns_sd_configs). See these docs and this feature request.

• FEATURE: vmalert: add -replay.disableProgressBar command-line flag, which allows disabling progressbar in rules' backfilling mode. See this issue.

• FEATURE: allow specifying TLS cipher suites for incoming https requests via -tlsCipherSuites command-line flag. See this feature request.

• FEATURE: allow specifying TLS cipher suites for mTLS connections between cluster components via -cluster.tlsCipherSuites command-line flag. See these docs.

• FEATURE: vmstorage: add -snapshotsMaxAge command-line flag for automatic removal of snapshots older than the given age.

• FEATURE: vmui: show an empty graph on the selected time range when there is no data on it. Previously No data to show placeholder was shown instead of the graph in this case. This prevented from zooming and scrolling of such a graph.

• FEATURE: vmui: show the selected last N minutes/hours/days in the top right corner. Previously the start - end duration was shown instead, which could be hard to interpret. See this feature request.

• FEATURE: vmui: execute the query when enter button is pressed in the same way as Prometheus does. Multi-line query can be entered by pressing shift-enter in the query input field.

• FEATURE: expose vm_indexdb_items_added_total and vm_indexdb_items_added_size_bytes_total counters at /metrics page, which can be used for monitoring the rate for addition of new entries in indexdb (aka inverted index) alongside the total size in bytes for the added entries. See this feature request.

• FEATURE: vmctl: show data pocessing speed during data migration.

• FEATURE: MetricsQL: add drop_common_labels() function, which drops common label="name" pairs from the passed time series. See these docs.

• FEATURE: MetricsQL: add tlast_change_over_time(m[d]) function, which returns the timestamp of the last change of m on the given lookbehind window d. See these docs.

• FEATURE: leave the last raw sample per each -dedup.minScrapeInterval discrete interval when the deduplication is enabled. This aligns better with the staleness rules in Prometheus comparing to the previous behaviour when the first sample per each -dedup.minScrapeInterval was left.

• FEATURE: VictoriaMetrics cluster: add ability to disable peer TLS certificate verification with -cluster.tlsInsecureSkipVerify command-line flag. See mTLS docs for details. See this feature request.

• FEATURE: add a handler for /api/v1/status/buildinfo endpoint, which is used by Grafana starting from v8.5.0 . See this pull request.

• FEATURE: add ability to proxy alerting API requests from Grafana to vmalert by passing -vmalert.proxyURL command-line flag to single-node VictoriaMetrics or to vmselect at cluster version of VictoriaMetrics. See this issue.

• BUGFIX: export staleness markers as null values from JSON export API. Previously they were exported as NaN values. This could break the exported JSON parsing, since NaN values aren't supported by JSON specification.

• BUGFIX: VictoriaMetrics cluster: close vmselect->vmstorage connections if they were idle for more than 30 seconds. Expose vm_tcpdialer_conns_idle metric at http://vmselect:8481/metrics with the number of idle connections to vmstorage. See this issue.

• BUGFIX: vmctl: return non-zero exit code on error. This allows handling vmctl errors in shell scripts. Previously vmctl was returning 0 exit code on error. See this issue.

• BUGFIX: vmctl: prevent from indefinite hang on Ctrl+C. See this issue.

• BUGFIX: vmagent: properly show scrape_timeout and scrape_interval options at http://vmagent:8429/config page. Previously these options weren't displayed even if they were set in -promscrape.config.

• BUGFIX: vmagent: handle non-standard http redirect status codes, which may be returned by scrape targets, in the same way as Prometheus does. See this issue.

• BUGFIX: vmalert: skip template execution during rules' validation. This should prevent from error evaluating annotation template errors when some template functions expect non-empty args. See this issue.

• BUGFIX: vmalert: fixed truncating alerts expression in table, updated table cell layout. See this issue.

• BUGFIX: MetricsQL: properly handle joins on time series filtered by values. For example, kube_pod_container_resource_requests{resource="cpu"} * on (namespace,pod) group_left() (kube_pod_status_phase{phase=~"Pending|Running"}==1). This query could result in duplicate time series on the right side error even if ==1 filter leaves only a single time series per (namespace,pod) labels. Now such query is properly executed.

• BUGFIX: MetricsQL: properly handle scalar default vector, scalar if vector and scalar ifnot vector queries. Previously such queries could return unexpected results from the vector part.

• BUGFIX: Official Grafana dashboards for VictoriaMetrics: take into account indexdb when calculating disk space usage. See this issue.

See full changelog.

How to run VictoriaMetrics

Unpack the victoria-metrics-*.tar.gz archive and read these docs.

vmutils-*.tag.gz archive contains the following tools:

• vmagent
• vmalert
• vmauth
• vmbackup
• vmctl
• vmrestore

vmutils-*-enterprise.tar.gz archive contains the following additional enterprise tools:

• vmbackupmanager
• vmgateway

The corresponding docker images are available here. Cluster version is available here.