Manticore Search 6.0.0

Released: Feb 7 2023

➡️➡️➡️ DOWNLOAD HERE ⬅️⬅️⬅️

Starting with this release, Manticore Search comes with Manticore Buddy, a sidecar daemon written in PHP that handles high-level functionality that does not require super low latency or high throughput. Manticore Buddy operates behind the scenes, and you may not even realize it is running. Although it is invisible to the end user, it was a significant challenge to make Manticore Buddy easily installable and compatible with the main C++-based daemon. This major change will allow the team to develop a wide range of new high-level features, such as shards orchestration, access control and authentication, and various integrations like mysqldump, DBeaver, Grafana mysql connector. For now it already handles SHOW QUERIES, BACKUP and Auto schema.

This release also includes more than 130 bug fixes and numerous features, many of which can be considered major.

Major Changes

• 🔬 Experimental: you can now execute Elasticsearch-compatible insert and replace JSON queries which enables using Manticore with tools like Logstash (version < 7.13), Filebeat and other tools from the Beats family. Enabled by default. You can disable it using SET GLOBAL ES_COMPAT=off.
• Support for Manticore Columnar Library 2.0.0 with numerous fixes and improvements in Secondary indexes. ⚠️ BREAKING CHANGE: Secondary indexes are ON by default as of this release. Make sure you do ALTER TABLE table_name REBUILD SECONDARY if you are upgrading from Manticore 5. See below for more details.
• Commit c436 Auto-schema: you can now skip creating a table, just insert the first document and Manticore will create the table automatically based on its fields. Read more about this in detail here. You can turn it on/off using searchd.auto_schema.
• Vast revamp of cost-based optimizer which lowers query response time in many cases.
  • Issue #1008 Parallelization performance estimate in CBO.
  • Issue #1014 CBO is now aware of secondary indexes and can act smarter.
  • Commit cef9 Encoding stats of columnar tables/fields are now stored in the meta data to help CBO make smarter decisions.
  • Commit 2b95 Added CBO hints for fine-tuning its behaviour.
• Telemetry: we are excited to announce the addition of telemetry in this release. This feature allows us to collect anonymous and depersonalized metrics that will help us improve the performance and user experience of our product. Rest assured, all data collected is completely anonymous and will not be linked to any personal information. This feature can be easily turned off in the settings if desired.
• Commit 5aaf ALTER TABLE table_name REBUILD SECONDARY to rebuild secondary indexes whenever you want, for example:
  • when you migrate from Manticore 5 to the newer version,
  • when you did UPDATE (i.e. in-place update, not replace) of an attribute in the index
• Issue #821 New tool manticore-backup for backing up and restoring Manticore instance
• SQL command BACKUP to do backups from inside Manticore.
• SQL command SHOW QUERIES as an easy way to see running queries rather than threads.
• Issue #551 SQL command KILL to kill a long-running SELECT.
• Dynamic max_matches for aggregation queries to increase accuracy and lower response time.

Minor changes

• Issue #822 SQL commands FREEZE/UNFREEZE to prepare a real-time/plain table for a backup.

• Commit c470 New settings accurate_aggregation and max_matches_increase_threshold for controlled aggregation accuracy.

• Issue #718 Support for signed negative 64-bit IDs. Note, you still can't use IDs > 2^63, but you can now use ids in the range of from -2^63 to 0.

• As we recently added support for secondary indexes, things became confusing as "index" could refer to a secondary index, a full-text index, or a plain/real-time index. To reduce confusion, we are renaming the latter to "table". The following SQL/command line commands are affected by this change. Their old versions are deprecated, but still functional:

  • index <table name> => table <table name>,
  • searchd -i / --index => searchd -t / --table,
  • SHOW INDEX STATUS => SHOW TABLE STATUS,
  • SHOW INDEX SETTINGS => SHOW TABLE SETTINGS,
  • FLUSH RTINDEX => FLUSH TABLE,
  • OPTIMIZE INDEX => OPTIMIZE TABLE,
  • ATTACH TABLE plain TO RTINDEX rt => ATTACH TABLE plain TO TABLE rt,
  • RELOAD INDEX => RELOAD TABLE,
  • RELOAD INDEXES => RELOAD TABLES.

  We are not planning to make the old forms obsolete, but to ensure compatibility with the documentation, we recommend changing the names in your application. What will be changed in a future release is the "index" to "table" rename in the output of various SQL and JSON commands.

• Queries with stateful UDFs are now forced to be executed in a single thread.

• Issue #1011 Refactoring of all related to time scheduling as a prerequisite for parallel chunks merging.

• ⚠️ BREAKING CHANGE: Columnar storage format has been changed. You need to rebuild those tables that have columnar attributes.

• ⚠️ BREAKING CHANGE: Secondary indexes file format has been changed, so if you are using secondary indexes for searching and have searchd.secondary_indexes = 1 in your configuration file, be aware that the new Manticore version will skip loading the tables that have secondary indexes. It's recommended to:

  • Before you upgrade change searchd.secondary_indexes to 0 in the configuration file.
  • Run the instance. Manticore will load up the tables with a warning.
  • Run ALTER TABLE <table name> REBUILD SECONDARY for each index to rebuild secondary indexes.

  If you are running a replication cluster, you'll need to run ALTER TABLE <table name> REBUILD SECONDARY on all the nodes or follow this instruction with just change: run the ALTER .. REBUILD SECONDARY instead of the OPTIMIZE.

• ⚠️ BREAKING CHANGE: The binlog version has been updated, so any binlogs from previous versions will not be replayed. It is important to ensure that Manticore Search is stopped cleanly during the upgrade process. This means that there should be no binlog files in /var/lib/manticore/binlog/ except for binlog.meta after stopping the previous instance.

• Issue #849 SHOW SETTINGS: helper command for manticore-backup.

• Issue #1007 SET GLOBAL CPUSTATS=1/0 turns on/off cpu time tracking; SHOW THREADS now doesn't show CPU statistics when the cpu time tracking is off.

• Issue #1009 RT table RAM chunk segments can now be merged while the RAM chunk is being flushed.

• Issue #1012 Added secondary index progress to the output of indexer.

• Issue #1013 Previously a table record could be removed by Manticore from the index list if it couldn't start serving it on start. The new behaviour is to keep it in the list to try to load it on the next start.

• indextool --docextract returns all the words and hits belonging to requested document.

• Commit 2b29 Environment variable dump_corrupt_meta enables dumping a corrupted table meta data to log in case searchd can't load the index.

• Commit c7a3 DEBUG META can show max_matches and pseudo sharding statistics.

• Commit 6bca A better error instead of the confusing "Index header format is not json, will try it as binary...".

• Commit bef3 Ukirainian lemmatizer path has been changed.

• Commit 4ae7 Secondary indexes statistics has been added to SHOW META.

• Commit 2e7c JSON interface can now be easily visualized using Swagger Editor https://manual.manticoresearch.com/dev/Openapi#OpenAPI-specification.

Changes related with Manticore Columnar Library

• Refactoring of Secondary indexes integration with Columnar storage.
• Commit efe2 Manticore Columnar Library optimization which can lower response time by partial preliminary min/max evaluation.
• Commit 2757 If a disk chunk merge is interrupted, the daemon now cleans up the MCL-related tmp files.
• Commit e9c6 Columnar and secondary libraries versions are dumped to log on crash.
• Commit f5e8 Added support for quick doclist rewinding to secondary indexes.
• Commit 06df Queries like select attr, count(*) from plain_index (w/o filtering) are now faster in case you are using MCL.
• Commit 0a76 @@autocommit in HandleMysqlSelectSysvar for compatibility with .net connector for mysql greater than 8.25

Packaging-related changes

• Commit 4d19 ⚠️ BREAKING CHANGE: Support for Debian Stretch and Ubuntu Xenial has been discontinued.
• RHEL 9 support including Centos 9, Alma Linux 9 and Oracle Linux 9.
• Issue #924 Debian Bookworm support.
• Issue #636 Packaging: arm64 builds for Linuxes and MacOS.
• PR #26 Multi-architecture (x86_64 / arm64) docker image.
• Simplified package building for contributors.
• It's now possible to install a specific version using APT.
• Commit a6b8 Windows installer (previously we provided just an archive).
• Switched to compiling using CLang 15.
• ⚠️ BREAKING CHANGE: Custom Homebrew formulas including the formula for Manticore Columnar Library. To install Manticore, MCL and any other necessary components, use the following command brew install manticoresoftware/manticore/manticoresearch manticoresoftware/manticore/manticore-extra.

Bugfixes

• Issue #479 Field with name text
• Issue #501 id can't be non bigint
• Issue #646 ALTER vs field with name "text"
• ❗Issue #652 Possible BUG: HTTP (JSON) offset and limit affects facet results
• ❗Issue #827 Searchd hangs/crashes under intensive loading
• ❗Issue #996 PQ index out of memory
• ❗Commit 1041 binlog_flush = 1 has been broken all the time since Sphinx. Fixed.
• MCL Issue #14 MCL: crash on select when too many ft fields
• MCL Issue #17 MCL: add SSE code to columnar scan
• Issue #470 sql_joined_field can't be stored
• Issue #713 Crash when using LEVENSHTEIN()
• Issue #743 Manticore crashes unexpected and cant to normal restart
• Issue #788 CALL KEYWORDS through /sql returns control char which breaks json
• Issue #789 mariadb can't create table FEDERATED
• Issue #796 WARNING: dlopen() failed: /usr/bin/lib_manticore_columnar.so: cannot open shared object file: No such file or directory
• Issue #797 Manticore crashes when search with ZONESPAN is done through api
• Issue #799 wrong weight when using multiple indexes and facet distinct
• Issue #801 SphinxQL group query hangs after SQL index reprocessing
• Issue #802 MCL: Indexer crashes in 5.0.2 and manticore-columnar-lib 1.15.4
• Issue #813 Manticore 5.0.2 FEDERATED returns empty set (MySQL 8.0.28)
• Issue #824 select COUNT DISTINCT on 2 indices when result is zero throws internal error
• Issue #826 CRASH on delete query
• Issue #843 MCL: Bug with long text field
• Issue #856 5.0.2 rtindex: Aggregate search limit behavior is not as expected
• Issue #863 Hits returned is Nonetype object even for searches that should return multiple results
• Issue #870 Crash with using Attribute and Stored Field in SELECT expression
• Issue #872 table gets invisible after crash
• Issue #877 Two negative terms in search query gives error: query is non-computable
• Issue #878 a -b -c is not working via json query_string
• Issue #886 pseudo_sharding with query match
• Issue #893 Manticore 5.0.2 min/max function doesn't work as expecting ...
• Issue #896 Field "weight" is not parsed correctly
• Issue #897 Manticore service crash upon start and keep restarting
• Issue #900 group by j.a, smth works wrong
• Issue #913 Searchd crash when expr used in ranker, but only for queries with two proximities
• Issue #916 net_throttle_action is broken
• Issue #919 MCL: Manticore crashes on query execution and other crashed during cluster recovery.
• Issue #925 SHOW CREATE TABLE outputs w/o backticks
• Issue #930 It's now possible to query Manticore from Java via JDBC connector
• Issue #933 bm25f ranking problems
• Issue #934 configless indexes frozen in watchdog on the first-load state
• Issue #937 Segfault when sorting facet data
• Issue #940 crash on CONCAT(TO_STRING)
• Issue #947 In some cases a single simple select could cause the whole instance stall, so you couldn't log in to it or run any other query until the running select is done.
• Issue #948 Indexer crash
• Issue #950 wrong count from facet distinct
• Issue #953 LCS is calculating incorrectly in built-in sph04 ranker
• Issue #955 5.0.3 dev crashing
• Issue #963 FACET with json on engine columnar crash
• Issue #982 MCL: 5.0.3 crash from secondary index
• PR #984 @@autocommit in HandleMysqlSelectSysvar
• PR #985 Fix thread-chunk distribution in RT indexes
• Issue #985 Fix thread-chunk distribution in RT indexes
• Issue #986 wrong default max_query_time
• Issue #987 Crash on when using regex expression in multithreaded execution
• Issue #988 Broken backward index compatibility
• Issue #989 indextool reports error checking columnar attributes
• Issue #990 memleak of json grouper clones
• Issue #991 Memleak of levenshtein func cloning
• Issue #992 Error message lost when loading meta
• Issue #993 Propagate errors from dynamic indexes/subkeys and sysvars
• Issue #994 Crash on count distinct over a columnar string in columnar storage
• Issue #995 MCL: min(pickup_datetime) from taxi1 crashes
• Issue #997 empty excludes JSON query removes columns from select list
• Issue #998 Secondary tasks run on current scheduler sometimes cause abnormal side effects
• Issue #999 crash with facet distinct and different schemas
• Issue #1000 MCL: Columnar rt index became damaged after run without columnar library
• Issue #1001 implicit cutoff is not working in json
• Issue #1002 Columnar grouper issue
• Issue #1003 Unable to delete last field from the index
• Issue #1004 wrong behaviour after --new-cluster
• Issue #1005 "columnar library not loaded", but it's not required
• Issue #1006 no error for delete query
• Issue #1010 Fixed ICU data file location in Windows builds
• PR #1018 Handshake send problem
• Issue #1020 Display id in show create table
• Issue #1024 crash 1 Crash / Segmentation Fault on Facet search with larger number of results.
• Commit 4739 Thread gets stuck on shutdown while replication is busy between nodes
• Commit ab87 Mixing floats and ints in a JSON range filter could make Manticore ignore the filter
• Commit d001 Float filters in JSON were inaccurate
• Commit 4092 Discard uncommitted txns if index altered (or it can crash)
• Commit 9692 Query syntax error when using backslash
• Commit 0c19 workers_clients could be wrong in SHOW STATUS
• Commit 1772 fixed a crash on merging ram segments w/o docstores
• Commit f45b Fixed missed ALL/ANY condition for equals JSON filter
• Commit 3e83 Replication could fail with got exception while reading ist stream: mkstemp(./gmb_pF6TJi) failed: 13 (Permission denied) if the searchd was started from a directory it can't write to.
• Commit 92e5 Since 4.0.2 crash log included only offsets. This commit fixes that.