Released at 2024-04-04

Update note 1: the -datasource.lookback command-line flag at vmalert is no-op starting from this release. This flag will be removed in the future, so please switch to eval_delay option. See this issue for more details.

• SECURITY: upgrade Go builder from Go1.21.7 to Go1.22.2. See the list of issues addressed in Go1.22.1 and the list of issues addressed in Go1.22.2.

• FEATURE: downsampling: add ability to configure distinct downsampling per distinct sets of time series and/or tenants. See this feature request and these docs.

• FEATURE: vmauth: allow discovering ip addresses for backend instances hidden behind a shared hostname, via discover_backend_ips: true option. This allows evenly spreading load among backend instances. See these docs and this feature request.

• FEATURE: vmauth: allow routing incoming requests based on HTTP query args via src_query_args option at url_map. See these docs and this feature request.

• FEATURE: vmauth: allow routing incoming requests based on HTTP request headers via src_headers option at url_map. See these docs.

• FEATURE: vmauth: add ability to read auth tokens from arbitrary HTTP request headers via -httpAuthHeader command-line flag. Previously auth tokens were read only from Authorization HTTP request header. See these docs for details.

• FEATURE: vmauth: add ability to authorize by opaque HTTP request header value via auth_token option in -auth.config.

• FEATURE: stream aggregation: reduce memory usage by up to 5x when aggregating over big number of unique time series. The memory usage reduction is most visible when stream deduplication is enabled.

• FEATURE: stream aggregation: allow using -streamAggr.dedupInterval and -remoteWrite.streamAggr.dedupInterval command-line flags without the need to specify -streamAggr.config and -remoteWrite.streamAggr.config. See these docs.

• FEATURE: stream aggregation: add -streamAggr.dropInputLabels command-line flag, which can be used for dropping the listed labels from input samples before applying stream de-duplication and aggregation. This is faster and easier to use alternative to input_relabel_configs. See these docs.

• FEATURE: stream aggregation: add dedup_interval option, which allows configuring individual deduplication intervals per each stream aggregation config.

• FEATURE: stream aggregation: use the same logic in stream deduplication as in the deduplication at VictoriaMetrics. See this issue.

• FEATURE: stream aggregation: ignore out of order samples when calculating increase, increase_prometheus, total and total_prometheus outputs. Thanks to @edma2 for the pull request.

• FEATURE: stream aggregation: add an ability to ignore input samples with old timestamps outside the current aggregation interval. See these docs for details.

• FEATURE: stream aggregation: add keep_metric_names option, which can be set at stream aggregation config in order to keep the original metric names in the output aggregated samples instead of using the default output metric naming scheme.

• FEATURE: stream aggregation: align the time of aggregated data flush to the specified aggregation interval. For example, if interval is set to 1m, then the aggregated data will be flushed at the end of every minute. The alginment can be disabled by setting no_align_flush_to_interval: true option at stream aggregation config. See these docs for details.

• FEATURE: stream aggregation: add unique_samples output, which can be used for calculating the number of unique sample values over the given interval.

• FEATURE: stream aggregation: add increase_prometheus and total_prometheus outputs, which can be used for increase and total aggregations when the first sample of every new time series must be ignored.

• FEATURE: stream aggregation: expose vm_streamaggr_flush_timeouts_total and vm_streamaggr_dedup_flush_timeouts_total counters at /metrics page, which can be used for detecting flush timeouts for stream aggregation states. Expose also vm_streamaggr_flush_duration_seconds and vm_streamaggr_dedup_flush_duration_seconds histograms for monitoring the real flush durations of stream aggregation states.

• FEATURE: vmui: improve trace display for better visual separation of branches. See this pull request.

• FEATURE: vmagent: ability to limit the ingestion rate via -maxIngestionRate command-line flag. See this pull request.

• FEATURE: vmagent: use the provided -remoteWrite.tlsServerName as Host header in requests to -remoteWrite.url. This allows sending data to https remote storage by IP address instead of hostname. Thanks to @minor-fixes for initial idea and the pull request.

• FEATURE: vmagent: add -remoteWrite.shardByURL.ignoreLabels command-line flag, which can be used for specifying the ignored list of labels when sharding by -remoteWrite.url is enabled. Thanks to @edma2 for the idea and the pull request.

• FEATURE: vmagent: automatically reload updated root CA certificates from files without the need to restart vmagent. See this issue.

• FEATURE: optimize /api/v1/labels and /api/v1/label/.../values when match[] filters contains metric name. For example, /api/v1/label/instance/values?match[]=up now works much faster than before. See this issue.

• FEATURE: vmagent: accept Prometheus remote write protocol data at /api/v1/push endpoint. This simplifies writing data to vmagent from Grafana Mimir. Thanks to @edma2 for the pull request.

• FEATURE: vmctl: support client-side TLS configuration for native protocol. See this feature request. Thanks to @khushijain21 for the pull request.

• FEATURE: vmctl: support client-side TLS configuration for VictoriaMetrics destination specified via --vm-* cmd-line flags used in InfluxDB, Remote Read protocol, OpenTSDB, Prometheus and Promscale migration modes.

• FEATURE: vmctl: split explore phase in vm-native mode by time intervals when --vm-native-step-interval is specified. This should reduce probability of exceeding complexity limits for number of selected series during explore phase. See this issue.

• FEATURE: vmgateway: add -logInvalidAuthTokens command-line flag, which can be used for logging invalid auth tokens. This is useful for debugging of auth token format issues. See this feature request.

• FEATURE: graphite: add support for aggregateSeriesLists, diffSeriesLists, multiplySeriesLists and sumSeriesLists functions. Thanks to @rbizos for the pull request.

• FEATURE: OpenTelemetry: add -opentelemetry.usePrometheusNaming command-line flag, which can be used for enabling automatic conversion of the ingested metric names and labels into Prometheus-compatible format. See these docs and this issue.

• BUGFIX: prevent from automatic deletion of newly registered time series when it is queried immediately after the addition. The probability of this bug has been increased significantly after v1.99.0 because of optimizations related to registering new time series. See this and this issue.

• BUGFIX: vmagent: properly set Host header in requests to scrape targets if it is specified via headers option. Thanks to @fholzer for the bugreport and the fix.

• BUGFIX: vmagent: properly set Host header in requests to scrape targets when server_name option at tls_config is set. Previously the Host header was set incorrectly to the target hostname in this case.

• BUGFIX: do not drop match[] filter at /api/v1/series if -search.ignoreExtraFiltersAtLabelsAPI command-line flag is set, since missing match[] filter breaks /api/v1/series requests.

• BUGFIX: vmagent: return proper responses for AWS Firehose requests according to these docs. See this pull request and this issue.

• BUGFIX: vmctl: properly parse TLS key and CA files for InfluxDB and OpenTSDB migration modes.

• BUGFIX: vmui: fix VictoriaLogs UI query handling to correctly apply _time filter across all queries. See this issue.

• BUGFIX: vmui: fix issue where step parameter wasn't updated after moving the focus outside the step input field in Firefox.

• BUGFIX: vmui: fix freezing when pasting a query with autocomplete enabled. See this issue.

• BUGFIX: vmui: fix auto-suggestion trigger issue after whitespace char. See this issue.

• BUGFIX: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: limit duration of requests to /api/v1/labels, /api/v1/label/.../values or /api/v1/series with -search.maxLabelsAPIDuration duration. Before, -search.maxExportDuration value was used by mistake. The bug has been introduced in v1.99.0. Thanks to @kbweave for the pull request.

• BUGFIX: properly wait for force merge to be completed during the shutdown. See this issue for the details.

• BUGFIX: vmalert: set correct endsAt value in notifications sent to the Alertmanager. Previously, a rule with evaluation intervals lower than 10s could never be triggered. See this issue for details.

• BUGFIX: vmalert: properly account for -rule.resendDelay for alerting rules that are constantly switching state from inactive to firing. Before, notifications for such rules could have been skipped if state change happened more often than -rule.resendDelay. See this pull request for details.

• BUGFIX: vmalert: respect -remoteWrite.maxBatchSize at shutdown period. See this issue. Thanks to @jiekun for the pull request.

How to run VictoriaMetrics

Unpack the victoria-metrics-*.tar.gz archive and read these docs.

vmutils-*.tag.gz archive contains the following tools:

• vmagent
• vmalert
• vmalert-tool
• vmauth
• vmbackup
• vmctl
• vmrestore

vmutils-*-enterprise.tar.gz archive contains the following additional enterprise tools:

• vmbackupmanager
• vmgateway

The corresponding docker images are available here. Cluster version is available here.