Enhancements

• #10389 Unified the configuration formats for cluster.core_nodes and cluster.statics.seeds. Now they both support formats in array ["emqx1@127.0.0.1", "emqx2@127.0.0.1"] and the comma-separated string "emqx1@127.0.0.1,emqx2@127.0.0.1".

• #10392 Introduced a new function to convert a formatted date to an integer timestamp: date_to_unix_ts/3.

  date_to_unix_ts(TimeUnit, FormatString, InputDateTimeString)

• #10426 Optimized the configuration priority mechanism to fix the issue where the configuration changes made to etc/emqx.conf do not take effect after restarting EMQX.

  More information about the new mechanism: Configure Override Rules

• #10457 Deprecated the integration with StatsD.

• #10458 Set the level of plugin configuration options to low, users usually manage the plugins through the dashboard, rarely modify them manually, so we lowered the level.

• #10491 Renamed etcd.ssl to etcd.ssl_options to keep all SSL options consistent in the configuration file.

• #10512 Improved the storage format of Unicode characters in data files, Now we can store Unicode characters. For example: SELECT * FROM "t/1" WHERE clientid = "-测试专用-".

• #10568 Added shutdown_counter printout to emqx ctl listeners command.

• #10588 Increased the time precision of trace logs from second to microsecond. For example, change from 2023-05-02T08:43:50+00:00 to 2023-05-02T08:43:50.237945+00:00.

• #10623 Renamed max_message_queue_len to max_mailbox_size in the force_shutdown configuration. The old name is kept as an alias, so this change is backward compatible.

• #10713 Hide the resource_option.request_timeout of the webhook and it will use the value of http request_timeout.

• #10075 Added node rebalance/node evacuation functionality. See also: EIP doc

• #10378 Implemented Pulsar Producer Bridge and only producer role is supported now.

• #10408 Introduced 3 built-in functions in the rule engine SQL-like language for creating values of the MongoDB date type.

• #10409 #10337 Supported Protocol Buffers and Apache Avro schemas in Schema Registry.

• #10425 Implemented OpenTSDB data bridge.

• #10498 Implemented Oracle Database Bridge.

• #10560 Added enterprise data bridge for Apache IoTDB.

• #10417 Improved get config items performance by eliminating temporary references.

• #10430 Simplified the configuration of the retainer feature. Marked flow_control as a non-importance field.

• #10511 Improved the security and privacy of some resource logs by masking sensitive information in the log.

• #10525 Reduced resource usage per MQTT packet handling.

• #10528 Reduced memory footprint in hot code path. The hot path includes the code that is frequently executed in core functionalities such as message handling, connection management, authentication, and authorization.

• #10591 #10625 Improved the configuration of the limiter.

  • Reduced the complexity of the limiter's configuration.

  • Updated the configs/limiter API to suit this refactor.

  • Reduced the memory usage of the limiter configuration.

• #10487 Optimized the instance of limiter for whose rate is infinity to reduce memory and CPU usage.

• #10490 Removed the default limit of connect rate which used to be 1000/s.

• #10077 Added support for QUIC TLS password-protected certificate file.

Bug Fixes

• #10340 Fixed the issue that could lead to crash logs being printed when stopping EMQX via systemd.

  2023-03-29T16:43:25.915761+08:00 [error] Generic server memsup terminating. Reason: {port_died,normal}. Last message: {'EXIT',<0.2117.0>,{port_died,normal}}. State: [{data,[{"Timeout",60000}]},{items,{"Memory Usage",[{"Allocated",929959936},{"Total",3832242176}]}},{items,{"Worst Memory User",[{"Pid",<0.2031.0>},{"Memory",4720472}]}}]. 2023-03-29T16:43:25.924764+08:00 [error] crasher: initial call: memsup:init/1, pid: <0.2116.0>, registered_name: memsup, exit: {{port_died,normal},[{gen_server,handle_common_reply,8,[{file,"gen_server.erl"},{line,811}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [os_mon_sup,<0.2114.0>], message_queue_len: 0, messages: [], links: [<0.2115.0>], dictionary: [], trap_exit: true, status: running, heap_size: 4185, stack_size: 29, reductions: 187637; neighbours: 2023-03-29T16:43:25.924979+08:00 [error] Supervisor: {local,os_mon_sup}. Context: child_terminated. Reason: {port_died,normal}. Offender: id=memsup,pid=<0.2116.0>.

• #10369 Fixed error in /api/v5/monitor_current API endpoint that happens when some EMQX nodes are down.

  Prior to this fix, sometimes the request returned HTTP code 500 and the following message:

  {"code":"INTERNAL_ERROR","message":"error, badarg, [{erlang,'++',[{error,nodedown},[{node,'emqx@10.42.0.150'}]], ...

• #10407 Fixed the crash issue of the alarm system.

  • Leverage Mnesia dirty operations and circumvent extraneous calls to enhance 'emqx_alarm' performance.

  • Use 'emqx_resource_manager' for reactivating alarms that have already been triggered.

  • Implement the newly developed, fail-safe 'emqx_alarm' API to control the activation and deactivation of alarms, thus preventing 'emqx_resource_manager' from crashing due to alarm timeouts.

  • The alarm system is susceptible to crashing under these concurrent conditions:

    • A significant number of resources fail, such as when bridges continuously attempt to trigger alarms due to recurring errors.

    • The system is under an extremely high load.

• #10420 Fixed HTTP path handling when composing the URL for the HTTP requests in authentication and authorization modules.

  • Avoid unnecessary URL normalization since we cannot assume that external servers treat original and normalized URLs equally. This led to bugs like #10411.

  • Fixed the issue that path segments could be HTTP encoded twice.

• #10422 Fixed a bug where external plugins could not be configured via environment variables in a lone-node cluster.

• #10448 Fixed a compatibility issue of limiter configuration introduced by e5.0.3 which broke the upgrade from previous versions if the capacity is infinity.

  In e5.0.3 we have replaced capacity with burst. After this fix, a capacity = infinity config will be automatically converted to equivalent burst = 0.

• #10462 Deprecated config broker.shared_dispatch_ack_enabled. This was designed to avoid dispatching messages to a shared-subscription session that has the client disconnected. However, since e5.0.0, this feature is no longer helpful because the shared-subscription messages in an expired session will be redispatched to other sessions in the group. See also: https://github.com/emqx/emqx/pull/9104 .

• #10463 Improved bridges API error handling. If Webhook bridge URL is not valid, the bridges API will return '400' error instead of '500'.

• #10484 Fixed the issue that the priority of the configuration cannot be set during the rolling upgrade. For example, when authorization is modified in e5.0.2 and then upgraded e5.0.3 through the rolling upgrade, the authorization will be restored to the default.

• #10495 Added the limiter API /configs/limiter which was deleted by mistake back.

• #10500 Added several fixes, enhancements, and features in Mria:

  • Protect mria:join/1,2 with a global lock to prevent conflicts between two nodes trying to join each other simultaneously Mria PR

  • Implement new function mria:sync_transaction/4,3,2, which blocks the caller until a transaction is imported to the local node (if the local node is a replicant, otherwise, it behaves exactly the same as mria:transaction/3,2) Mria PR

  • Optimize mria:running_nodes/0 Mria PR

  • Optimize mria:ro_transaction/2 when called on a replicant node Mria PR.

• #10518 Added the following fixes and features in Mria:

  • Call mria_rlog:role/1 safely in mria_membership to ensure that mria_membership gen_server won't crash if RPC to another node fails Mria PR

  • Add an extra field to ?rlog_sync table to facilitate extending this functionality in future Mria PR.

• #10556 Wrapped potentially sensitive data in emqx_connector_http if Authorization headers are being passed at initialization.

• #10571 Stopped emitting useless crash report when EMQX stops.

• #10659 Fixed the issue where EMQX cannot start when sysmon.os.mem_check_interval is disabled.

• #10717 Fixed an issue where the buffering layer processes could use a lot of CPU when inflight window is full.

• #10724 A summary has been added for all endpoints in the HTTP API documentation (accessible at "http://<emqx_host_name>:18083/api-docs").

• #10726 Health Check Interval and Auto Restart Interval now support the range from 1ms to 1 hour.

• #10728 Fixed an issue where the rule engine was unable to access variables exported by FOREACH - DO clause.

  Given a payload: {"date": "2023-05-06", "array": ["a"]}, as well as the following SQL statement:

  FOREACH payload.date as date, payload.array as elem DO date, elem FROM "t/#" -- {"date": "2023-05-06", "array": ["a"]}

  Prior to the fix, the date variable exported by FOREACH could not be accessed in the DO clause of the above SQL, resulting in the following output for the SQL statement: [{"elem": "a","date": "undefined"}].

• #10742 Correctness check of the rules is enforced before saving the authorization file source. Previously, Saving wrong rules could lead to EMQX restart failure.

• #10743 Fixed an issue where trying to get bridge info or metrics could result in a crash when a node is joining a cluster.

• #10755 Fixed data bridge resource update race condition.

  In the 'delete + create' process for EMQX resource updates, long bridge creation times could cause dashboard request timeouts. If a bridge resource update was initiated before completion of its creation, it led to an erroneous deletion from the runtime, despite being present in the config file.

  This fix addresses the race condition in bridge resource updates, ensuring the accurate identification and addition of new resources, and maintaining consistency between runtime and configuration file statuses.

• #10761 Fixed the issue where the default value of SSL certificate for Dashboard Listener was not correctly interpolated, which caused HTTPS to be inaccessible when verify_peer and cacertfile were using the default configuration.

• #10672 Fixed the issue where the lack of a default value for ssl_options in listeners results in startup failure. For example, such command(EMQX_LISTENERS__WSS__DEFAULT__BIND='0.0.0.0:8089' ./bin/emqx console) would have caused a crash before.

• #10738 TDEngine data bridge now supports "Supertable" and "Create Tables Automatically". Before this fix, an insert with a supertable in the template will fail, like this:

  • insert into ${clientid} using msg TAGS (${clientid}) values (${ts},${msg}).

• #10746 Add missing support of the event $events/delivery_dropped into the rule engine test API rule_test.

• #10747 Ported some time formating fixes in Rule-Engine functions from version 4.4.

• #10760 Fix "internal error 500" when getting bridge statistics page while a node is joining the cluster.

• #10801 Avoid double percent-decode for topic name in API /topics/{topic} and /topics.

• #10817 Fix a config value handling for bridge resource option auto_restart_interval, now it can be set to infinity.