Released at 2024-02-14

• SECURITY: upgrade Go builder from Go1.21.6 to Go1.22.0. See the list of issues addressed in Go1.21.7, plus the changelog for Go1.22.0.

• FEATURE: all VictoriaMetrics components: add support for TLS client certificate verification at -httpListenAddr (aka mTLS). See these docs and this feature request.

• FEATURE: all VictoriaMetrics components: add support for accepting http requests over multiple distinct TCP addresses. This can be done by starting VictoriaMetrics component with multiple -httpListenAddr command-line flags. For example, ./victoria-metrics -httpListenAddr=some-host:12345 -httpListenAddr=localhost:8428 starts VictoriaMetrics, which accepts incoming http requests at both some-host:12345 and localhost:8428. See this feature request.

• FEATURE: all VictoriaMetrics components: add support for empty command-line flag values in short array notation. For example, -remoteWrite.sendTimeout=',20s,' specifies three -remoteWrite.sendTimeout values - the first one and the last one have default values (30s in this case), while the second one is set to 20s.

• FEATURE: all VictoriaMetrics components: do not close connections to -httpListenAddr every 2 minutes. This behavior didn't help spreading load among multiple backend servers behind load-balancing TCP proxy. Instead, it could lead to hard-to-debug issues like this one. If you still need periodically closing client connections because of some reason, then pass the desired timeout to -http.connTimeout command-line flag.

• FEATURE: vmauth: add support for mTLS-based request routing to different backends depending on the subject of the TLS certificate provided by the client. See these docs and this feature request.

• FEATURE: vmagent and single-node VictoriaMetrics: add support for data ingestion via DataDog lambda extension aka /api/beta/sketches endpoint. See these docs and this feature request. Thanks to @AndrewChubatiuk for the pull request.

• FEATURE: vmagent: add -remoteWrite.tlsHandshakeTimeout command-line flag for tuning the timeout needed for establishing TLS connections to -remoteWrite.url. Bigger tls handshake timeouts should reduce the probability of http: TLS handshake error from ...: EOF errors at the remote storage side under high load. See this issue.

• FEATURE: VictoriaMetrics cluster: add -disableReroutingOnUnavailable command-line flag to vminsert, which can be used for reducing resource usage spikes at vmstorage nodes during rolling restart. See these docs. Thanks to @Muxa1L for the pull request.

• FEATURE: add -search.resetRollupResultCacheOnStartup command-line flag for resetting query cache on startup. See this feature request.

• FEATURE: MetricsQL: propagate label filters across label_set and alias functions. For example, label_set(q1, "a", "b") + q2{c="d"} is automatically transformed to label_set(q1{c="d"}, "a", "b") + q2{a="b",c="d"} now. This should improve performance for such queries. See this issue.

• FEATURE: MetricsQL: add sum_eq_over_time, sum_gt_over_time and sum_le_over_time functions. See this feature request.

• FEATURE: dashboards/vmagent: add Targets scraped/s stat panel showing the number of targets scraped by the vmagent per-second.

• FEATURE: dashboards/all: add new panel CPU spent on GC. It should help identifying cases when too much CPU is spent on garbage collection, and advice users on how this can be addressed.

• FEATURE: vmalert: support filtering for /api/v1/rules API. See the pull request by @victoramsantos.

• FEATURE: vmbackup: support client-side TLS configuration for creating and deleting snapshots via -snapshot.tls* cmd-line flags. See this feature request. Thanks to @khushijain21 for the pull request.

• BUGFIX: vmagent: reduce CPU usage when -promscrape.dropOriginalLabels command-line flag is set. This issue has been introduced in v1.96.0 when addressing this feature request.

• BUGFIX: vmauth: properly release memory during config reload. See this issue.

• BUGFIX: vmauth: properly expose vmauth_unauthorized_user_concurrent_requests_capacity, vmauth_unauthorized_user_concurrent_requests_current, vmauth_user_concurrent_requests_capacity and vmauth_user_concurrent_requests_current metrics after config reload. Previously these metrics didn't work after config reload.

• BUGFIX: MetricsQL: properly propagate label filters from multiple arguments passed to aggregate functions. For example, sum({job="foo"}, {job="bar"}) by (job) + a was improperly optimized to sum({job="foo"}, {job="bar"}) by (job) + a{job="foo"} before being executed. This could lead to unexpected results. See this issue.

• BUGFIX: MetricsQL: properly handle precision errors when calculating changes, changes_prometheus, increases_over_time and resets functions. See this issue.

• BUGFIX: all VictoriaMetrics components: consistently return 200 http status code from /-/reload endpoint. Previously single-node VictoriaMetrics was returning 204 http status code. See this feature request.

• BUGFIX: properly store staleness markers for self-scraped metrics on single-node VictoriaMetrics shutdown. See this issue.

• BUGFIX: prevent from possible too big indexBlockSize panic when samples with too long label values (~64Kb) are ingested into VictoriaMetrics.

• BUGFIX: vmui: fix the graph dragging for Firefox and Safari. See this issue.

• BUGFIX: vmui: fix handling invalid timezone. See this issue.

• BUGFIX: vmui: fix the bug where the select does not open. See this issue.

• BUGFIX: vmui: clear entered text in select after selecting a value. See this issue.

• BUGFIX: vmui: improve the operation of the context for autocomplete. See this, this and this issues.

• BUGFIX: dashboards: update Storage full ETA panels for Single-node and Cluster dashboards to prevent them from showing negative or blank results caused by increase of deduplicated samples. Deduplicated samples were part of the expression to provide a better estimate for disk usage, but due to sporadic nature of deduplication in VictoriaMetrics it rather produced skewed results. See this pull request.

How to run VictoriaMetrics

Unpack the victoria-metrics-*.tar.gz archive and read these docs.

vmutils-*.tag.gz archive contains the following tools:

• vmagent
• vmalert
• vmalert-tool
• vmauth
• vmbackup
• vmctl
• vmrestore

vmutils-*-enterprise.tar.gz archive contains the following additional enterprise tools:

• vmbackupmanager
• vmgateway

The corresponding docker images are available here. Cluster version is available here.