5.8.2

Release Date: 2024-11-12

Enhancements

Core MQTT Functionalities

• #14059 Added a new configuration option for the retainer to cap message expiry intervals for retained messages. This enables garbage collection to remove messages sooner if storage is running low.

• #14072 Updated the virtual machine to use Unicode for its printable range. This improvement enhances the readability of certain binary data in messages. For instance, a binary previously displayed as <<116,101,115,116,228,184,173,230,150,135>> will now be formatted as <<"test中文"/utf8>>, providing clearer representation.

MQTT Durable Sessions

• #14130 Reduced CPU usage for idle durable sessions.

  Previously, idle durable sessions periodically woke up to refresh the list of DS streams. With this change, stream discovery is now event-based, significantly lowering CPU consumption during idle periods. Additionally, the update reduces the delay in notifying sessions of new streams, effectively eliminating the long-tail latency in end-to-end processing.

REST API

• #13889 Enhanced the performance of the /api/v5/monitor_current and /api/v5/metrics APIs.

  Previously, these APIs queried clustered nodes sequentially in a loop. Now, the queries are sent in parallel, reducing response time. The latency is now primarily dependent on the slowest node in the cluster.

  Additionally, a node parameter was added to the /api/v5/monitor_current API, allowing targeted queries to a single node instead of the entire cluster. For instance, using ?aggregate=false&node=emqx@node1.domain.name will return data exclusively for the specified node.

EMQX Clustering

• #13903 Added logs to inform the user when a replicant node cannot find a core node with the same release version as its own.

Security

• #13923 Added zone support in authentication, authorization, and mountpoint templates.

  Previously, to reference a client's zone in authentication or authorization rules, users needed to access it through client_attrs. Now, the ${zone} placeholder can be used directly in these templates, simplifying rule creation and enabling zone-specific configurations.

  For example, the following ACL rule uses ${zone} to dynamically apply permissions based on a client’s assigned zone: {allow, all, all, ["${zone}/${username}/#"]}.

• #14102 Added support for SSL private key passphrase from a secret file.

  EMQX can now read the passphrase from a secret file if password is configured as ...ssl_options.password = "file://{path-to-secret-file}".

Data Integration

• #14065 Added a new queuing_bytes metric for data integration. This metric shows the RAM and/or disk resources consumed by buffering for a specific action.

Observability

• #14096 Exposed emqx_conf_sync_txid as a Prometheus metric, allowing for monitoring the configuration file synchronization status of each node in the cluster.

MQTT over QUIC

• #13814 Connection Scope Keepalive for MQTT over QUIC Multi-Stream:

  Introduced a new feature to keep MQTT connections alive when data streams remain active, even if the control stream is idle.

  Previously, clients were required to send MQTT.PINGREQ on idle control streams to keep the connection alive. Now, a shared state tracks activity across all streams for each connection. This shared state is used to determine if the connection is still alive, reducing the risk of keepalive timeouts due to Head-of-Line (HOL) blocking.

• #14112 Added support ssl_options.hibernate_after in QUIC listener to reduce memory footprint of QUIC transport.

Bug Fixes

Core MQTT Functionality

• #13931 Updated the gen_rpc library to version 3.4.1, which includes a fix to prevent client socket initialization errors from escalating to the node level on the server side.

• #13969 Optimized the periodic cleanup of expired retained messages to ensure efficient resource usage, particularly in cases with a large volume of expired messages.

• #14068 Added the handle_frame_error/2 callback to all gateway implementation modules to handle message parsing errors.

• #14037 Improved the internal database bootstrap process to better tolerate temporary unavailability of peer nodes, particularly when a new node joins an existing cluster.

• #14116 Fixed an issue where the default configuration for the retainer was generated incorrectly after joining a cluster.

MQTT Durable Sessions

• #14042 Fix crash in the durable session after updates to subscription parameters (such as QoS, no_local, upgrade_qos, ...).

• #14052 Corrected memory usage reporting from cgroups when in use.

• #14055 Updated the /clients_v2 API to properly respect all filtering arguments when querying offline clients with durable sessions. Previously, only the username filter was applied, while other filtering arguments were ignored.

• #14151 Fixed handling of the conn_state filter in the /clients_v2 API for offline clients with durable sessions. Previously, these clients could be incorrectly selected with conn_state=connected.

• #14057 Resolved a compatibility issue that prevented the Messages DS database from starting due to a slightly different database configuration schema. This issue occurred when upgrading EMQX from version 5.7.x with session durability enabled.

REST API

• #14023 Fixed an issue with the GET /monitor HTTP API where returned values could appear higher than actual values, depending on the requested time window. For data points within a 1-hour window, this distortion is only visual on the Dashboard. However, for data points older than 1 hour, the data distortion is permanent.

  The affected metrics include:

  • disconnected_durable_sessions
  • subscriptions_durable
  • subscriptions
  • topics
  • connections
  • live_connections

EMQX Clustering

• #13996 Fixed an intermittent crash occurring when using emqx conf fix to resolve configuration discrepancies, particularly if a configuration key was missing on one of the nodes.

Security

• #13922 Updated the CRL (Certificate Revocation List) cache to use the full Distribution Point (DP) URL as the cache key. Previously, only the path part of the URL was used, causing conflicts when multiple DPs shared the same path.

• #13924 Fixed an issue where JWK keys could leak into debug logs upon JWT authentication failure.

Data Integration

• #13916 Fixed an issue where the parent metric failed was not incremented when a rule’s failed.no_result or failed.exception metrics were updated.

• #14001 Resolved a race condition where a resource (such as a connector, action, source, authentication, or authorization) could falsely report a connected, healthy channel after a brief disconnection. This issue could result in excessive action_not_found log entries when the race condition occurred.

• #13913 Fixed an issue with the actions and source HTTP APIs where a 500 status code would be returned if a timeout occurred while attempting to update or delete a resource.

• #14101 Resolved an issue where deleting a resource would fail if a source and an action were both created with the same name.

Observability

• #13909 Fixed log formatting for cases where the payload cannot be displayed as readable UTF-8 Unicode characters.

• #14061 Improved log information when emqx_cm:request_stepdown/3 fails.

  In scenarios where a client channel needs to terminate another channel with the same ClientID, a race condition may occur if the target channel has already been closed or terminated. In such cases, error logs and stack traces that provide no useful information will no longer be generated.

• #14070 Removed the connector's state from error and warning logs due to its potential length. For issue analysis, the connector's state can now be accessed through emqx_resource:list_instances_verbose/0. Below is an example of a log entry before this change:

  pid: <0.43914.0>, connector: connector:sqlserver:connector-05a2e105, reason: [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Argument data type varchar is invalid for argument 2 of dateadd function. SQLSTATE IS: 42000, state: {"resource_opts":{"start_timeout":5000,"start_after_created":true,"health_check_interval":15000},"pool_name":"connector:sqlserver:connector-05a2e105","installed_channels":{"action:sqlserver:action-4b033621:connector:sqlserver:connector-05a2e105":{"sql_templates":{"batch_insert_temp":{"send_message":{"batch_insert_tks":["{str,<<\" ( \">>}","{var,[<<\"messageId\">>]}","{str,<<\", \">>}","{var,[<<\"measurement\">>]}","{str,<<\", \">>}","{var,[<<\"Analog_IN_Fault_1\">>]}","{str,<<\", \">>}","{var,[<<\"Analog_IN_Fault_2\">>]}","{str,<<\", \">>}","{var,[<<\"Analog_IN_Fault_3\">>]}","{str,<<\", \">>}","{var,[<<\"Analog_IN_Fault_4\">>]}","{str,<<\", \">>}","{var,[<<\"Analog_IN_PV_1\">>]}","{str,<<\", \">>}","{var,[<<\"Analog_IN_PV_2\">>]}","{str,<<\", \">>}","{var,[<<\"Analog_IN_PV_3\">>]}","{str,<<\", \">>}","{var,[<<\"Analog_IN_PV_4\">>]}","{str,<<\", DATEADD(MS, \">>}","{var,[<<\"ms_shift\">>]}","{str,<<\", DATEADD(S, \">>}","{var,[<<\"s_shift\">>]}","{str,<<\", '19700101 00:00:00:000') ))\">>}"],"batch_insert_part":"insert into TransactionLog(MessageId, Measurement, Fault1, Fault2, Fault3, Fault4, Value1, Value2, Value3, Value4, DateStamp) \r\n"}}}}}},msg: invalid_request
  

• #14099 Removed an error-level log entry that was triggered when validation of UTF-8 strings in MQTT messages failed.

  Example of the removed log entry:

  {"time":"2024-10-11T06:05:07.610048+00:00","level":"error","msg":"supervisor: {esockd_connection_sup,0.53591191.0}, errorContext: connection_shutdown, reason: #{cause => invalid_topic,reason => malformed_utf8_string_length}, offender: [{pid,0.53591191.0},...]", ..., "error_logger":{"type":"supervisor_report","tag":"error_report"}}
  

• #14091 Implemented a fix to remove function_clause from log messages when users provide unsupported write syntax.

  Example of unsupported syntax:

  weather,location=us-midwest,season=summer temperature=82 ${timestamp}u 
  

  Before this fix, the error log would contain the function_clause error, as shown:

  pid: <0.558392.0>, info: {"stacktrace":["{emqx_bridge_influxdb_connector,parse_timestamp,[[1719350482910000000,<<\"u\">>]],[{file,\"emqx_bridge_influxdb_connector.erl\"},{line,692}]}", ...], ..., "error":"{error,function_clause}"}, tag: ERROR, msg: resource_exception
  

  This change improves log clarity by omitting function_clause in cases of syntax errors.