Changes

v1.85.0

Released at 11-12-2022

Update note 1: this release drops support for direct upgrade from VictoriaMetrics versions prior v1.28.0. Please upgrade to v1.84.0, wait until finished round 2 of background conversion line is emitted to log by single-node VictoriaMetrics or by vmstorage, and then upgrade to newer releases.

Update note 2: this release splits type="indexdb" metrics into type="indexdb/inmemory" and type="indexdb/file" metrics. This may break old dashboards and alerting rules, which contain label filter on {type="indexdb"}. Such label filter must be substituted with {type=~"indexdb.*"}, so it matches indexdb from the previous releases and indexdb/inmemory + indexdb/file from new releases. It is recommended upgrading to the latest available dashboards and alerting rules mentioned in these docs, since they already contain fixed label filters.

Update note 3: this release deprecates relabel_debug and metric_relabel_debug config options in scrape_configs. The -relabelDebug, -remoteWrite.relabelDebug and -remoteWrite.urlRelabelDebug command-line options are also deprecated. Use more powerful target-level relabel debugging and metric-level relabel debugging instead as documented here.

• FEATURE: vmagent: provide enhanced target-level and metric-level relabel debugging. See these docs and this issue.

• FEATURE: leave a sample with the biggest value for identical timestamps per each -dedup.minScrapeInterval discrete interval when the deduplication is enabled. See this issue.

• FEATURE: add -inmemoryDataFlushInterval command-line flag, which can be used for controlling the frequency of in-memory data flush to disk. The data flush frequency can be reduced when VictoriaMetrics stores data to low-end flash device with limited number of write cycles (for example, on Raspberry PI). See this feature request.

• FEATURE: expose additional metrics for indexdb and storage parts stored in memory and for indexdb parts stored in files (see storage docs for technical details):

  • vm_active_merges{type="storage/inmemory"} - active merges for in-memory storage parts
  • vm_active_merges{type="indexdb/inmemory"} - active merges for in-memory indexdb parts
  • vm_active_merges{type="indexdb/file"} - active merges for file-based indexdb parts
  • vm_merges_total{type="storage/inmemory"} - the total merges for in-memory storage parts
  • vm_merges_total{type="indexdb/inmemory"} - the total merges for in-memory indexdb parts
  • vm_merges_total{type="indexdb/file"} - the total merges for file-based indexdb parts
  • vm_rows_merged_total{type="storage/inmemory"} - the total rows merged for in-memory storage parts
  • vm_rows_merged_total{type="indexdb/inmemory"} - the total rows merged for in-memory indexdb parts
  • vm_rows_merged_total{type="indexdb/file"} - the total rows merged for file-based indexdb parts
  • vm_rows_deleted_total{type="storage/inmemory"} - the total rows deleted for in-memory storage parts
  • vm_assisted_merges_total{type="storage/inmemory"} - the total number of assisted merges for in-memory storage parts
  • vm_assisted_merges_total{type="indexdb/inmemory"} - the total number of assisted merges for in-memory indexdb parts
  • vm_parts{type="storage/inmemory"} - the total number of in-memory storage parts
  • vm_parts{type="indexdb/inmemory"} - the total number of in-memory indexdb parts
  • vm_parts{type="indexdb/file"} - the total number of file-based indexdb parts
  • vm_blocks{type="storage/inmemory"} - the total number of in-memory storage blocks
  • vm_blocks{type="indexdb/inmemory"} - the total number of in-memory indexdb blocks
  • vm_blocks{type="indexdb/file"} - the total number of file-based indexdb blocks
  • vm_data_size_bytes{type="storage/inmemory"} - the total size of in-memory storage blocks
  • vm_data_size_bytes{type="indexdb/inmemory"} - the total size of in-memory indexdb blocks
  • vm_data_size_bytes{type="indexdb/file"} - the total size of file-based indexdb blocks
  • vm_rows{type="storage/inmemory"} - the total number of in-memory storage rows
  • vm_rows{type="indexdb/inmemory"} - the total number of in-memory indexdb rows
  • vm_rows{type="indexdb/file"} - the total number of file-based indexdb rows

• FEATURE: DataDog parser: add device tag when it is passed in the device field is present in the series object of the input request. Thanks to @PerGon for the provided pull request.

• FEATURE: vmagent: improve service discovery performance when discovering big number of targets (10K and more).

• FEATURE: vmagent: allow using series_limit option for limiting the number of series a single scrape target generates in stream parsing mode. See this feature request.

• FEATURE: vmagent: allow using sample_limit option for limiting the number of metrics a single scrape target can expose in every response sent over stream parsing mode.

• FEATURE: vmagent: add exported_ prefix to metric names exported by scrape targets if these metric names clash with automatically generated metrics such as up, scrape_samples_scraped, etc. This prevents from corruption of automatically generated metrics. See this issue.

• FEATURE: vmagent: make the host label optional in DataDog data ingestion protocol. See this issue.

• FEATURE: VictoriaMetrics cluster: improve error message when the requested path cannot be properly parsed, so users could identify the issue and properly fix the path. Now the error message links to url format docs. See this issue.

• FEATURE: VictoriaMetrics enterprise cluster: add -storageNode.discoveryInterval command-line flag to vmselect and vminsert to control load on DNS servers when automatic discovery of vmstorage nodes is enabled. See this issue.

• FEATURE: VictoriaMetrics enterprise cluster: allow reading and updating the list of vmstorage nodes at vmselect and vminsert nodes via file. See automatic discovery of vmstorage for details.

• FEATURE: vmalert: reduce memory and CPU usage by up to 50% on setups with thousands of recording/alerting groups. See this issue.

• FEATURE: vmalert: add -remoteWrite.sendTimeout command-line flag, which allows configuring timeout for sending data to -remoteWrite.url. See this issue.

• FEATURE: vmctl: add ability to migrate data between VictoriaMetrics clusters with automatic tenants discovery. See these docs and this issue.

• FEATURE: vmctl: add ability to copy data from sources via Prometheus remote_read protocol. See these docs. The related issues: one and two.

• FEATURE: vmui: allow changing timezones for the requested data. See this issue.

• FEATURE: vmui: provide fast path for hiding results for all the queries except the given one by clicking eye icon with ctrl key pressed. See this feature request.

• FEATURE: MetricsQL: add range_trim_spikes(phi, q) function for trimming phi percent of the largest spikes per each time series returned by q. See these docs.

• FEATURE: MetricsQL: allow passing inf arg into limitk, topk, bottomk and other functions, which accept numeric arg, which limits the number of output time series. See this feature request.

• FEATURE: vmgateway: add support for JWT token signature verification. See these docs for details.

• FEATURE: put the version of VictoriaMetrics in the first message of query trace. This should simplify debugging.

• BUGFIX: vmagent: fix the The request did not have a subscription or a valid tenant level resource provider error when discovering Azure targets with azure_sd_configs. See this issue.

• BUGFIX: vmalert: properly pass HTTP headers during the alert state restore procedure. See this issue.

• BUGFIX: vmalert: properly specify rule evaluation step during the replay mode. The step value was previously overriden by -datasource.queryStep command-line flag.

• BUGFIX: vmalert: properly return the error message from remote-write failures. Before, error was ignored and only vmalert_remotewrite_errors_total was incremented.

• BUGFIX: vmui: fix sticky tooltip sizing, which could prevent from closing the tooltip. See this issue.

• BUGFIX: vmui: properly put multi-line queries in the url, so it could be copy-n-pasted and opened without issues in a new browser tab. Previously the url for multi-line query couldn't be opened. See this issue.

• BUGFIX: vmui: correctly handle up and down keypresses when editing multi-line queries. See this issue.

See full changelog.

How to run VictoriaMetrics

Unpack the victoria-metrics-*.tar.gz archive and read these docs.

vmutils-*.tag.gz archive contains the following tools:

• vmagent
• vmalert
• vmauth
• vmbackup
• vmctl
• vmrestore

vmutils-*-enterprise.tar.gz archive contains the following additional enterprise tools:

• vmbackupmanager
• vmgateway

The corresponding docker images are available here. Cluster version is available here.