Changes

It isn't recommended to use VictoriaMetrics cluster v1.81.0 because of the bug, which may result in vmselect crashes under high load. Upgrade to v1.81.2 instead.

v1.81.0

Released at 31-08-2022

Update note 1: vmalert by default hides values of -remoteWrite.url, -remoteRead.url and -datasource.url in logs and at http://vmalert:8880/flags for security reasons. See the corresponding SECURITY change in the Chagelog below for additional info.

Update note 2: vmalert by default points alert source url to /vmalert/alert?... aka web UI instead of /vmalert/api/v1/alert?... aka JSON handler. The old behavior can be achieved by setting -external.alert.source=vmalert/api/v1/alert?group_id={{.GroupID}}&alert_id={{.AlertID}} command-line flag.

• SECURITY: vmalert: do not expose -remoteWrite.url, -remoteRead.url and -datasource.url command-line flag values in logs and at http://vmalert:8880/flags page by default, since they may contain sensitive data such as auth keys. This aligns vmalert behaviour with vmagent, which doesn't expose -remoteWrite.url command-line flag value in logs and at http://vmagent:8429/flags page by default. Specify -remoteWrite.showURL, -remoteRead.showURL and -datasource.showURL command-line flags for showing values for the corresponding -*.url flags in logs. Thanks to @mble for the pull request.

• SECURITY: upgrade base docker image (alpine) from 3.16.1 to 3.16.2. See alpine 3.16.2 release notes.

• FEATURE: return shorter error messages to Grafana and to other clients requesting /api/v1/query and /api/v1/query_range endpoints. This should simplify reading these errors by humans. The long error message with full context is still written to logs.

• FEATURE: add the ability to fine-tune the number of points, which can be generated per each matching time series during subquery evaluation. This can be done with the -search.maxPointsSubqueryPerTimeseries command-line flag. See this feature request.

• FEATURE: vmagent: improve the performance for relabeling rules with commonly used regular expressions in regex and if fields such as some_string, prefix.*, prefix.+, foo|bar|baz, .*foo.* and .+foo.+.

• FEATURE: vmagent: reduce CPU usage when discovering big number of Kubernetes targets with big number of labels and annotations.

• FEATURE: vmagent: add ability to accept multitenant data via OpenTSDB /api/put protocol at /insert/<tenantID>/opentsdb/api/put http endpoint if multitenant support is enabled at vmagent. Thanks to @chengjianyun for the pull request.

• FEATURE: monitoring: expose vm_hourly_series_limit_max_series, vm_hourly_series_limit_current_series, vm_daily_series_limit_max_series and vm_daily_series_limit_current_series metrics when -search.maxHourlySeries or -search.maxDailySeries limits are set. This allows alerting when the number of unique series reaches the configured limits. See these docs for details.

• FEATURE: VictoriaMetrics cluster: reduce the amounts of logging at vmstorage when vmselect connects/disconnects to vmstorage.

• FEATURE: VictoriaMetrics cluster: improve performance for heavy queries on systems with many CPU cores.

• FEATURE: vmagent: add ability to use {{label_name}} placeholders in the replacement option of relabeling rules. This simplifies constructing label values from multiple existing label values. See these docs for details.

• FEATURE: vmagent: generate additional per-target metrics - scrape_series_limit, scrape_series_current and scrape_series_limit_samples_dropped if series limit is set according to these docs. This simplifies alerting on targets with the exceeded series limit. See these docs for details on these metrics.

• FEATURE: vmagent: add support for MX record types in dns_sd_configs in the same way as Prometheus 2.38 does.

• FEATURE: vmagent: add __meta_kubernetes_service_port_number meta-label for role: service in kubernetes_sd_configs in the same way as Prometheus 2.38 does.

• FEATURE: vmagent: add __meta_kubernetes_pod_container_image meta-label for role: pod in kubernetes_sd_configs in the same way as Prometheus 2.38 does.

• FEATURE: vmagent: retry HTTP requests after some wait time during service discovery and during target scrapes if the server returns 429 HTTP status code (aka Too many requests). See this issue.

• FEATURE: vmui: add a legend in the top right corner for shortcut keys. See this feature request.

• FEATURE: vmalert: add toTime() template function in the same way as Prometheus 2.38 does. See these docs.

• FEATURE: vmalert: add $alertID and $groupID template variables. These variables may be used for templating annotations or -external.alert.source command-line flag. See the full list of supported variables here.

• FEATURE: vmalert: add $activeAt template variable. See this feature request. See the full list of supported variables here. Thanks to @laixintao for the pull request.

• FEATURE: vmalert: point alert source to vmalert's UI at /vmalert/alert?... instead of JSON handler at /vmalert/api/v1/alert?.... This improves user experience. The old behavior can be achieved by setting -external.alert.source=vmalert/api/v1/alert?group_id={{.GroupID}}&alert_id={{.AlertID}} command-line flag.

• BUGFIX: prevent from excess CPU usage when the storage enters read-only mode.

• BUGFIX: improve performance for requests to /api/v1/labels and /api/v1/label/.../values when the filter in the match[] query arg matches small number of time series. The performance for this case has been reduced in v1.78.0. See this and this issues.

• BUGFIX: increase the default limit on the number of concurrent merges for small parts from 8 to 16. This should help resolving potential issues with heavy data ingestion. See this comment from @lukepalmer .

• BUGFIX: MetricsQL: fix panic when incorrect arg is passed as phi into histogram_quantiles function. See this issue.

See full changelog.

How to run VictoriaMetrics

Unpack the victoria-metrics-*.tar.gz archive and read these docs.

vmutils-*.tag.gz archive contains the following tools:

• vmagent
• vmalert
• vmauth
• vmbackup
• vmctl
• vmrestore

vmutils-*-enterprise.tar.gz archive contains the following additional enterprise tools:

• vmbackupmanager
• vmgateway

The corresponding docker images are available here. Cluster version is available here.