e5.0.3

Enhancements

• #10128 Add support for OCSP stapling for SSL MQTT listeners.

• #10156 Change the configuration overlay order:

  If it is a new installation of EMQX, emqx.conf + Environment variables overlays on top of API Updated Configs (cluster.hocon)

  If EMQX is upgraded from an older version (i.e., the cluster-override.conf file still exists in EMQX's data directory), then it’s the same as before, that is cluster-override.conf overlays on top of emqx.conf + Environment variables.

  Please note that data/configs/cluster-override.conf is considered deprecated. After upgrade, you are encouraged to update emqx.conf to delete configs which are overridden by cluster-override.conf and move the configs in cluster-override.conf to cluster.hocon. After upgrade, EMQX will continue to read local-override.conf (if it exists) as before, but you are encouraged to merge the configs to emqx.conf.

• #10164 Add CRL check support for TLS MQTT listeners.

• #10207 Improve OpenAPI (swagger) document readability. Prior to this change, there were a few summary docs which are lengthy and lack of translation, now it makes use of the more concise label field from schema i18n database instead.

• #10210 Eliminated a few harmless error level logs. Prior to this change, there might be some Mnesia callback (hook) failures occasionally occurring when stopping/restarting Mria. Now the callbacks (hooks) are unregistered prior to stop. See also Mria PR.

• #10224 Add the option to customize clusterIP in Helm chart, so that a user may set it to a fixed IP.

• #10263 Add command eval-ex for Elixir expression evaluation.

• #10278 Refactor the directory structure of all gateways.

• #10206 Support async query mode for all data bridges.

  Prior to this change, setting the query mode of a resource such as a bridge to sync would force the buffer to call the underlying connector in a synchronous way, even if it supports async calls.

• #10306 Add support for async query mode for most bridges.

  This is a follow-up change after #10206. Before this change, some bridges (Cassandra, MongoDB, MySQL, Postgres, Redis, RocketMQ, TDengine) were only allowed to be created with a sync query mode. Now async mode is also supported.

• #10318 Prior to this enhancement, only double quotes (") were allowed in rule engine SQL language's FROM clause. Now it also supports single quotes (').

• #10336 Add /rule_engine API endpoint to manage configuration of rule engine.

• #10354 More specific error messages when configure with bad max_heap_size value. Log current value and the max value when the message_queue_too_long error is thrown.

• #10358 Hide flapping_detect/conn_congestion/stats configuration. Deprecate flapping_detect.enable.

• #10359 Metrics now are not implicitly collected in places where API handlers don't make any use of them. Instead, a separate backplane RPC gathers cluster-wide metrics.

• #10373 Deprecate the trace.payload_encode configuration. Add payload_encode=[text,hidden,hex] option when creating a trace via HTTP API.

• #10381 Hide the auto_subscribe configuration items so that they can be modified later only through the HTTP API.

• #10391 Hide a large number of advanced options to simplify the configuration file.

  That includes rewrite, topic_metric, persistent_session_store, overload_protection, flapping_detect, conn_congestion, stats,auto_subscribe, broker_perf, shared_subscription_group, slow_subs, ssl_options.user_lookup_fun and some advance items in node and dashboard section, #10358, #10381, #10385.

• #10404 Change the default queue mode for buffer workers to memory_only. Before this change, the default queue mode was volatile_offload. When under high message rate pressure and when the resource is not keeping up with such rate, the buffer performance degraded a lot due to the constant disk operations.

• #10140 Integrate Cassandra into bridges as a new backend. At the current stage only support Cassandra version 3.x, not yet 4.x.

• #10143 Add RocketMQ data integration bridge.

• #10165 Support escaped special characters in InfluxDB data bridge write_syntax. This update allows to use escaped special characters in string elements in accordance with InfluxDB line protocol.

• #10211 Hide broker.broker_perf config and API documents. The two configs route_lock_type and trie_compaction are rarely used and requires a full cluster restart to take effect. They are not suitable for being exposed to users. Detailed changes can be found here: https://gist.github.com/zmstone/01ad5754b9beaeaf3f5b86d14d49a0b7/revisions.

• #10294 When configuring a MongoDB bridge, you can now use the ${field} syntax to reference fields in the message. This enables you to select the collection to insert data into dynamically.

• #10363 Implement Microsoft SQL Server bridge.

• #10573 Improved performance of Webhook bridge when using synchronous query mode. This also should improve the performance of other bridges when they are configured with no batching.

Bug Fixes

• #10145 Add field status_reason to GET /bridges/:id response in case this bridge is in status disconnected if internal health-check reports an error condition. Include this same error condition in message when creating an alarm for a failing bridge.

• #10172 Fix the incorrect regular expression in default ACL rule to allow specify username(dashboard) to subscribe $SYS/#.

• #10174 Upgrade library esockd from 5.9.4 to 5.9.6. Fix an unnecessary error level logging when a connection is closed before proxy protocol header is sent by the proxy.

• #10195 Add labels to API schemas where description contains raw HTML, which would break formatting of generated documentation otherwise.

• #10196 Use lower-case for schema summaries and descriptions to be used in menu of generated online documentation.

• #10209 Fix bug where a last will testament (LWT) message could be published when kicking out a banned client.

• #10225 Allow installing a plugin if its name matches the beginning of another (already installed) plugin name. For example: if plugin emqx_plugin_template_a is installed, it must not block installing plugin emqx_plugin_template.

• #10226 Handle validation error in /bridges API and return 400 instead of 500.

• #10242 Fixed a log data field name clash. Prior to this fix, some debug logs may report a wrong Erlang PID which may affect troubleshooting session takeover issues.

• #10257 Fixed the issue where auto_observe was not working in LwM2M Gateway.

  Before the fix, OBSERVE requests were sent without a token, causing failures that LwM2M clients could not handle.

  After the fix, LwM2M Gateway can correctly observe the resource list carried by client, furthermore, unknown resources will be ignored and printing the following warning log:

  2023-03-28T18:50:27.771123+08:00 [warning] msg: ignore_observer_resource, mfa: emqx_lwm2m_session:observe_object_list/3, line: 522, peername: 127.0.0.1:56830, clientid: testlwm2mclient, object_id: 31024, reason: no_xml_definition
  

• #10286 Enhance logging behaviour during boot failure. When EMQX fails to start due to corrupted configuration files, excessive logging is eliminated and no crash dump file is generated.

• #10297 Keeps eval command backward compatible with v4 by evaluating only Erlang expressions, even on Elixir node. For Elixir expressions, use eval-ex command.

• #10300 Fixed issue with Elixir builds that prevented plugins from being configured via environment variables.

• #10315 Fix crash checking limit and page parameters in /mqtt/delayed/messages API call.

• #10317 Do not expose listener level authentications before extensive verification.

• #10323 For security reasons, the value of the password field in the API examples is replaced with ******.

• #10410 Fix config check failed when gateways are configured in emqx.conf. This issue was first introduced in v5.0.22 via #10278, the boot-time config check was missing.

• #10533 Fixed an issue that could cause (otherwise harmless) noise in the logs.

  During some particularly slow synchronous calls to bridges, some late replies could be sent to connections processes that were no longer expecting a reply, and then emit an error log like:

  2023-04-19T18:24:35.350233+00:00 [error] msg: unexpected_info, mfa: emqx_channel:handle_info/2, line: 1278, peername: 172.22.0.1:36384, clientid: caribdis_bench_sub_1137967633_4788, info: {#Ref<0.408802983.1941504010.189402>,{ok,200,[{<<"cache-control">>,<<"max-age=0, ...">>}}
  

  Those logs are harmless, but they could flood and worry the users without need.

• #10449 Validate the ssl_options and header configurations when creating authentication http (authn_http). Prior to this, incorrect ssl configuration could result in successful creation but the entire authn being unusable.

• #10548 Fixed a race condition in the HTTP driver that would result in an error rather than a retry of the request. Related fix in the driver: https://github.com/emqx/ehttpc/pull/45

• #10201 In TDengine data bridge, removed the redundant database name from the SQL template.

• #10270 ClickHouse data bridge has got a fix that makes the error message better when users click the test button in the settings dialog.

• #10324 Previously, when attempting to reconnect to a misconfigured ClickHouse bridge through the dashboard, users would not receive an error message. This issue is now resolved, and error messages will now be displayed.

• #10438 Fix some configuration item terminology errors in the DynamoDB data bridge:

  • Changed database to table
  • Changed username to aws_access_key_id
  • Changed password to aws_secret_access_key