This is a periodic rollup release. It contains many features, bug fixes, and improvements.

The core API has still not stabilized and will not be guaranteed until 1.0.

Merged PRs

go-mysql-server

• 1056: Allow REGEXP args to be converted fix for #1055
• 1054: Fix prep AS OF Prepared AS OFs errored when the asof target was not a bindvar. All of our previous tests treated ASOF also as a bindvar. companion PR: https://github.com/dolthub/dolt/pull/3592
• 1053: Added type wire tests This enforces that we're returning the correct data over the wire (which may differ from the storage values). Majority of the fixes made have been moved to the parent PR (https://github.com/dolthub/go-mysql-server/pull/1044) so that this one is not quite as large. They're all fairly straightforward, hence I don't think they require review. The expected test output has been validated against MySQL (using the shim). In addition, this also enforces that the results are coming from the Type.SQL() function by running the output of a sql.RowIter (gathered from the directly querying the engine) through the Type.SQL() function and ensuring it matches.
• 1052: enginetest: Parameterize query plan tests
• 1051: Better join commutativity count We excluded joins on the basis of join factors in the logical join tree, rather than distinct join subtrees subject to commutativity. The difference is that we have to permute n! to optimize the search space of all valid commutative trees, versus a k^n (k = table rows) execution runtime. 12! blocks analysis, k^12 is steep but can be OK depending on cardinality and indexes. We impose no limits on k^n joins.
• 1047: sql/parse: Test round-tripping sql.Type as string
• 1046: Introduced OrderedIndex interface to deal with indexes that don't return ordered results
• 1044: Type value changes & row type assertions This PR has two primary goals:
  1. All types now pass their values around in a form most suitable to that type.
  2. Enforce that the aforementioned types are always passed to integrators, such that integrators do not need to do type validation on their end. To elaborate on these points, some types already passed around values that were sensible and a best fit for that type, such as MEDIUMINT returning an int32. Other types, such as DECIMAL, passed around strings, which necessitated conversions for integrators to be able to properly persist the values. Not only that, there is currently no guarantee that a row's values each have their best fit type (a BIGINT can work with an int8, but it should be able to always expect int64). To make these guarantees, I'm adding a check at all GMS-integrator junctions that pass a sql.Row and verifying that the value types are exactly what that column expects. This may have the side effect of changing the output of a SELECT statement for integrators. As a SELECT statement simply returns a sql.RowIter, additional logic will be needed to convert all values to their canonical MySQL representation. This can easily be achieved by passing all values through Type.SQL() before display.
• 1043: enginetest: Validating Enginetest Harness
• 1042: exclude procedure aliases from showing information_schema.routines table Not show procedure aliases in information_schema.routines table Dolt PR tests this change
• 1041: cleanup enginetest Splits up enginetest/enginetest.go into initialization.go and evaluation.go. The intent is to leave only test banks in enginetest.go This is a pure refactor, no logical changes were made.
• 1038: Added a few foreign key tests
• 1037: Parallelize IndexedTableAccess on DoltDiff tables
• 1033: fix drop current database with case insensitive name
• 1032: Bug fixes for ModifyColumn related to primary keys
• 1029: fix some column values of information_schema.columns table Fixes column_key, character_maximum_length, data_type and column_type column values of information_schema.columns table
• 1027: Moved responsibility for rewriting tables on add / drop primary key into engine
• 1026: More tests converted to new format
• 1025: adds mod() function fix for: https://github.com/dolthub/dolt/issues/3423
• 1022: Rewrite table for drop column
• 1021: Allow SHOW CREATE PROCEDURE for external procedures + display a fake CREATE PROCEDURE A note regarding the fake CREATE PROCEDURE. It seems that Java's JDBC checks that the procedure creation statement is a valid statement, regardless of whether the statement remotely matches that of a stored procedure. So rather than returning a random statement, we return a "valid" CREATE PROCEDURE statement. Fixes https://github.com/dolthub/dolt/issues/3428 and https://github.com/dolthub/dolt/issues/3424
• 1020: example memory.NewTable add "PrimaryKey:true" flag. example memory.NewTable add "PrimaryKey:true" flag.
• 1019: UUID function is no longer based off current time fix for: https://github.com/dolthub/dolt/issues/3323
• 1018: support SRID for spatial type column definition Added SRID value syntax functionality for column definition. Added tests cover CREATE TABLE, ALTER TABLE ADD/MODIFY COLUMN, and INSERT statements. Fixes https://github.com/dolthub/dolt/issues/3425
• 1015: Thread safe procedure cache
• 1014: Added field for connected db to logger
• 1013: Moving transaction initialization before query analysis Moves transaction creation ahead of query analysis, so that queries can be executed with the latest committed state from other transactions. Fixes: https://github.com/dolthub/dolt/issues/3402
• 1011: Add FK Checks for Insert on Duplicate
• 1009: sql/analyzer: Fix reresolveTables to not drop IndexedTableAccess and DeferredAsOfTable nodes on its transform. enginetest: Add a test to assert IndexedTableAccess in a prepared statement behaves correctly when querying a table that has been modified since the prepare.
• 1008: Bug fix for rewriting table during modify column
• 1007: Refactor grant_tables to mysql_db Mostly renaming variables and files to better reflect what they are now. Added flatbuffer files for MySQL DB.
• 1005: Allowing "DEFAULT NULL" for blob column definitions Fixes: https://github.com/dolthub/dolt/issues/3388
• 1004: Support for rewriting tables on column modifications
• 1003: Replace statements that order by primary key, to use indexed table access
• 1002: New interface to rewrite table on certain schema change operations Interface isn't quite final but this can be checked in. Complementary dolt changes are done and tested but want to get feedback on this approach first.
• 1000: Regression test case for https://github.com/dolthub/dolt/issues/3247 https://github.com/dolthub/dolt/issues/3247 reported a panic that our test cases didn't cover. Another commit (https://github.com/dolthub/go-mysql-server/commit/324e43b896d344f745be0b52935640335e9fd546) already fixed the panic, so this PR just adds a quick test to ensure we don't regress with the same bug.
• 997: adding COLLATE to SHOW CREATE TABLE Fix for: https://github.com/dolthub/dolt/issues/3351 Also fix for: https://github.com/dolthub/dolt/issues/3352
• 995: CheckpointHarness and analyzer mutexes
• 992: Allow unresolved tables and procedures in trigger body in CREATE TRIGGER In order to allow non-existent tables and non-exitent procedures in trigger body in CREATE TRIGGER statement, we no longer run analyzer on trigger body, instead all validation checks are performed on a single rule, validateCreateTrigger. Added tests for cases of different action times and events.
• 990: support SELECT INTO 'variable' functionality Added functionality support for SELECT INTO variable only. Usages of 'outfile' or 'dumpfile' will return unsupported syntax error. Added tests for different cases of SELECT INTO syntax. Using SELECT INTO around unions will not work unless the result is a single row because we do not support order by and limit clauses for union statements even though we parse it correctly.
• 988: Fix SHOW CREATE ... AS OF ... prepared statements Fix for: https://github.com/dolthub/dolt/issues/3304 Note:
  • The parseColumnDefaults rule is repeated because ColumnDefaults are unresolved after unresolveTables and setTargetSchemas, for some reason this only happens when using a DoltHarness.
• 986: support XOR logic operator Added XOR MySQL logic operator
• 985: Fix Handler queries when Prepared Statements are Disabled Fix for: https://github.com/dolthub/dolt/issues/3305
• 983: Change Autocommit Query Plan This PR adds a new analyzer rule and query Node for autocommit. This fixes an issue where previously a query process was being killed (via a cancelCtx function) before a transaction committed.
• 982: adding flag options 2 and 4 for st_asgeojson fix for: https://github.com/dolthub/dolt/issues/3280
• 981: fix panics on CREATE PROCEDURE queries
  • Skips evaluating procedure-parameters in the analyzer step
  • Added case for handling UnresolvedFunction in fds to avoid panics
  • Fixes issue, https://github.com/dolthub/dolt/issues/3241 (when procedure parameters are used in filter expressions, it needs to be evaluated, but procedure parameters cannot be evaluated without nil reference map)
• 980: Add tests for handling join in parentheses Tests regarding this issue, https://github.com/dolthub/dolt/issues/2203 Some cases give incorrect result in Dolt.
• 979: Fixed overly aggressive type change check with foreign keys Previously PR https://github.com/dolthub/go-mysql-server/pull/977 The bug has existed since the GMS foreign key implementation, but only surfaced after the previous PR. When changing a column's type, I was only checking that the table was used in foreign keys, when I needed to check that the column is used. This fixes that bug.
• 978: Use transactional logic to rollback bad triggers, also tests Added a new rule that wraps everything in a TriggerRollback node if there is a trigger present. This new node will make a savepoint right before execution, and will return to normal if something breaks.
• 977: Foreign keys now allow limited type changing It seems that, specifically when using the string types, you may lengthen a field, but you may not change any other properties (cannot change character set or collation). Non-string types don't seem to allow lengthening, as you can't change an INT to a BIGINT. The implemented behavior is actually an approximation of MySQL's, as you cannot cross some length thresholds due to how MySQL stores strings. From what I can gather for MySQL, as long as the on-disk data does not change AND does not need to be checked, it is allowed. I'm sure more type changes are allowed, but it's not documented, and therefore would require a ton of trial and error, so just shipping this for now.
• 975: Fixing return types for spatial functions Fix for: https://github.com/dolthub/dolt/issues/3279 Adding tests that check the return type for functions
  • st_asgeojson
  • st_aswkb
  • st_aswkt
  • st_astext
  • st_dimension
  • st_latitude
  • st_longitude
  • st_swapxy
  • st_x
  • st_y
• 974: Fixed LOAD DATA errors There are three core issues with LOAD DATA that have been fixed:
  1. Columns could be rearranged, but we didn't handle the case where the rearranged columns were a subset.
  2. Line endings were assumed to be a single character, which is not true for Windows (which uses \r\n).
  3. Default values did not work. Nil default values "worked" due to nil being the default value when instantiating a slice of interface{}s.
• 971: support handling join table between parentheses Allow handling join tables in parentheses. It causes the JoinTableExpr to be wrapped in ParenTableExpr in the parser, so it needs to be unwrapped to get the JoinTableExpr inside Fixes https://github.com/dolthub/dolt/issues/3254
• 970: Changed all DDL nodes to return an OkResult This is important for compatibility with certain SQL clients (notably C connector).
• 969: Improving Error Message for Inserting Strings that are too Long includes the string that failed and for which column helps for: https://github.com/dolthub/dolt/issues/3270
• 968: Expanded AddColumn tests to include some writes on the new table
• 967: add TRADITIONAL to sql_mode set Added TRADITIONAL option to sql_mode set. Fixes https://github.com/dolthub/dolt/issues/3230
• 966: Arithmetic .Type() Perf Improvement SQL Logictests were timing out on huge CASE statements with ~40 nodes. This fixes the computational blowup, which I think was somehow exponential before.
• 965: Bug Fix: FilterTable pushdown to aliased tables Fixes: https://github.com/dolthub/go-mysql-server/issues/808 Max – Tagging you for your analyzer expertise. Could you check this out and see if you see a better approach to solving this? I saw a couple ways to address this, but this seemed like the least intrusive approach.
• 964: support more than 1 byte characters to insert for char()/varchar() types Allows inserting strings with 'exact number of character but consists of more bytes' values to char() or varchar() types Fixes https://github.com/dolthub/dolt/issues/3233
• 961: Fix underspecified insert tests No inserts were not being run with prepareds. A lot of them were broken, but should be running in GMS now at least. Edit: These pass Dolt also.
• 960: Parse charset introducer for string literals for expressions Fixes parsing charset introducer in front of string literals for handling some expressions such as column default expressions. Allows only utf8mb4, utf8mb3 and utf8 charsets. If other charsets are used, gives unsupported charset error.
• 958: Added variable to hide external stored procedures
• 957: Allow conversion of strings and hex to geometry types fix for: https://github.com/dolthub/dolt/issues/3231 we should be able to read mysql dumps produced using the --hex-blob flag.
• 956: Fixing Point method to actually print string Fix for: https://github.com/dolthub/dolt/issues/3228 Allows the use of point(), linestring(), and polygon() for default values for columns of type point, linestring, polygon, and geometry.
• 955: allow NULL value for geometry types Allow handling null values some geometry data types. For these types, DEFAULT NULL values should be allowed
• 954: Allow arbitrary parameter count for external stored procedures If an overloaded function contains a variadic variant, that variant is only used if an exact match is not found.
• 953: fixing geometry tests Fix for: https://github.com/dolthub/dolt/issues/3217 Types returned should be wrapped in geometry.
• 952: Remove checks for unique index We should follow up with an implementation for a unique index for the memory table. Will take a crack at this in a future edit.
• 951: Added support for external stored procedures This adds support for hooking functions into the stored procedure system. There were two considerations for the implementation, and I felt this implementation is superior (using the reflect package). The other primary consideration was to expect a specific function signature that takes a map[string]interface{} representing each parameter, and there'd be another map[string]sql.Type returned in the loading function that specifies the type of each parameter. I didn't like that the types were interfaces, and their definitions declared elsewhere, as I'd rather the engine handle as much as possible. I'd prefer to have this all statically-enforced, but Go's type system is sadly not flexible enough, so this is the next best thing. The reflection logic is fairly straightforward, and there are many comments, so I'm not too concerned with this being hard to maintain at all.
• 948: Allow insertion of negative numbers into auto_increment columns Fix for: https://github.com/dolthub/dolt/issues/3192 Added an additional check to just not run auto_increment logic when value provided is less than the Zero() value for column. Small change to memory/table.go to stop incrementing the current autoIncVal when the provided value is strictly less than the current autoIncVal.
• 947: Update drop table tests to add primary keys, Add more ordering to JSON script test results
• 946: enginetests: Added result ordering to a few more JSON script tests
• 943: Return correctly formatted column defaults in the information_schema.columns table The information_schema.columns contains the column column_default that represents the formatted string version of the column default. This correct representation is essential as tools (like table editors) use this column to generate representation of a table or their alter statements. This pr pushes the resolution of column defaults through the analyzer. Note that there are some open flaws
  1. The double quoting of literal column defaults
  2. The double parentheses for expressions due to improper parentheses handling in the codebase
  3. Missing backticks for columns. This is due to fact that all the way down to GetField for expressions we don't actually backtick the name. We should followup with a rewrite of the Column Default String method There should be subsequent followup work that simplifies string formatting with default values and expressions as a whole.
• 942: enginetest: Update JSON Scripts tests to add primary keys and ordered results
• 941: Fixes unhashable filter bug, adds semantic validation starter Fixes unhashable filter bug. Adds improvement on semantic operand validation. Targetted checks easier to implement.
• 940: Enable RowIter2 on IndexedTableAccess This is pretty limited and only works for static lookups, not joins, but is enough to benchmark with This change also does some refactoring of index analysis code, but no other functional changes Small bug fix to the String() method of SubstringIndex
• 939: ValidateJoinDepth rule; hard cutoff for > 12 table joins New analyzer rule validateJoinDepth that errors if we count more than 12 join leaves. Opaque node counting could be done more carefully. join search schema comparison was O(n^4) if i counted correctly, now should be more like O(n)
• 938: Quickperm, reduce join search memory usage
• 936: Allow auto_increment on non-primary key columns Fix for: https://github.com/dolthub/dolt/issues/2981 Since Unique Keys are not a part of sql.Schema and sql.Column, we need to check if there are any valid indexes defined over the columns we want to mark as auto_increment. Added skipped tests for queries that don't work. TODO:
  • Adding auto_increment adds NOT NULL constraint too
  • Support adding columns; need to be able to alter table t add column j int unique first
  • Autoincrement on keys defined over multiple columns; MySQL only allows leftmost column to be auto_incremented
• 935: update create view parsing and added definer var for create trigger stmt Updated CREATE VIEW parsing supporting algorithm, definer, security syntaxes, but does not support the functionality or affect create view query statement. Added Definer variable to Trigger. Added and updated engine tests covering these changes
• 934: More transform and join tests
• 932: Adding support for show create table <table> as of <version> Adds support for show create table <table> as of <version> to go-mysql-server Depends on:
  • https://github.com/dolthub/vitess/pull/151
• 930: Editing test data to work around failures with new Dolt storage format Without changing the test data, the new Dolt storage format doesn't recognize that there is any change and it errors out with "nothing to commit" during testdata setup. Editing the test data allows the testdata setup to complete successfully on the old Dolt storage format and on the new Dolt storage format. TestVersionedQueries is now configured to be skipped on the new Dolt storage format.
• 929: have _example/main.go work as expected Fixes: https://github.com/dolthub/go-mysql-server/issues/911
• 928: Updating versioned query tests based on behavior differences in Dolt harness When running the versioned query tests with Dolt harness there are two behavior differences that cause these tests to fail. To fix those, we need to:
  1. explicitly delete old table data and
  2. temporarily remove non-existent version test since Dolt reports a no branch error, while Memory reports a no table error.
• 927: Prevent Creation of Unique Keys Small fix for preventing the creation of FULLTEXT keys as well. TODO: There should also be a change in Dolt to prevent altering and modifying columns to use FULLTEXT keys
• 926: Have VIEWS show up in information_schema.columns Fix for: https://github.com/dolthub/dolt/issues/3109 VIEWS now show up, but the our current implementation of ViewDefinition makes it difficult to match MySQL capabilities.
• 924: InsertInto Bindvars InsertInto is special in that [Source] is not a child sql.Node, so [Source] needs to be traversed separately to touch every expression in a DFS search. This fixes a bug in that DFS search missing the non-Source node children.
• 923: Adding support for an as of expression in show columns from table Passing through as of expression in show columns statements. Depends on Vitess support in: https://github.com/dolthub/vitess/pull/150
• 920: handle more time formats fix for https://github.com/dolthub/dolt/issues/3102
• 918: Multi Alter Add/Drop Columns This pr does the following
  1. Modifies Add/Drop/Rename/Modify columns to pull the relevant table fresh from the database
  2. Adds error checks in validate_create_table to make sure that invalid columns are detected at analyze time This pr does not:
  3. Address multi-alter ddls and edge cases with default values. I found a ton of small errors while working through edge cases here that I think there is a good amount of work separate from the immediate multi-alter ddl needs. This pr is enough to get us unblocked on panics that used occur when mass add/drop pairs were created.
• 915: fix create/drop TRIGGER parsing to support database name specific query
• 914: Adding Geometry type Adds geometry type for GMS side. Includes case for geometry type for functions:
  • st_swaxy
  • st_srid
  • st_dimension
  • st_aswkt
  • st_aswkb
• 913: Apply IndexJoins with CrossJoin children
• 910: Ensure that the local index map uses the computed id We should use the computed id of index as key for the index map of the memory table. Previously we were using the given index name which would conflict when multiple indexes were assigned the empty string.
• 909: fix create/drop procedure parsing to support database name specific query
• 907: Add db name to the column name for show tables statement
  • Database name is now included in column name in SHOW TABLES statement
  • Fixed cardinality column of information_schema.statistics table returned wrong type before
• 906: Store the creation time for trigger statements Will have changes to Dolt side to store this for Dolt Databases. MySQL has nanosecond precision, which we wont have here. I mimicked stored procedure behavior creation time for mysqlshim: https://github.com/dolthub/go-mysql-server/blob/main/enginetest/mysqlshim/database.go#L169
• 904: IndexedJoins mark handled filters Applying an indexed join in applyJoinPlan should apply equality lookup expressions, but currently do not. pushdown mistakenly marked those expressions as handled and eliminated necessary filters. This PR prevents IndexedJoinAccess with nil lookups from marking filters as handled. A future improvement could the pushdown range logic to opportunistically apply lookups in applyJoinPlan, and let pushdown mark those as handled. This would improve perf by both removing both a table scan and filter from some index joins.
• 903: support show create procedure
• 899: run query on info schema routines table for SHOW PROCEDURE STATUS Also fixes SHOW PROCEDURE STATUS to show all procedures in all databases. Removes show_procedure_status.go file (removing the showProcedureStatus node)
• 898: Improve MultiAlters error handling with Alter Primary Key, Alter Index, Alter Default, Alter AutoIncrement This pr does the following
  1. Improves MultiAlter DDL analyzer by checking for failure modes in adding/dropping pks, adding/dropping indexes, adding/dropping defaults
  2. Pushes the table resolution of AlterPk, AlterIndex, AlterDefault,Alter AutoIncrement at execution time. This is due to the fact that the node should know whether columns were added or dropped in previous steps of a MultiAlter DDL.
  3. Adds a small bug fix with how columns were being dropped in the validate_create_table
  4. This pr does not include changes to multialters with ADD/DROP constraints due to additional complexity.
• 897: Code sample update to remove missing Auth package Updated code sample in README and _example/main.go to remove the missing Auth package. Reported in: https://github.com/dolthub/go-mysql-server/issues/888
• 896: adding 1 second to trigger creation time Fix for this issue: https://github.com/dolthub/dolt/issues/3047 Triggers weren't showing up in TablePlus because the creation time for triggers is always 0 in unix time, which is out of TablePlus supported timestamp range (for some reason). So this PR just makes the trigger creation time 1 second after the start of all time, which TablePlus considers valid.
• 895: fix SHOW INDEXES FROM otherdb.tab fails as database is initialized incorrectly in parsing Before Show Index statement was initializing the Database incorrectly as it would set it to either empty or current database. This was an issue for using the statement for database that user is not currently on. This PR also fixes couple of info schema table inconsistencies against Mysql.
• 893: /go.mod: require go 1.18
• 890: support SHOW FUNCTION STATUS functionality SHOW FUNCTION STATUS result is queried from INFORMATION_SCHEMA.ROUTINES table
• 887: fix non_unique column value definition in information_schema.statistics table fix non_unique being assigned incorrect value. Verified with Mysql docs, "0 if the index cannot contain duplicates (which .IsUnique() is true), 1 if it can."
• 886: Auto Increment Refactor
• 885: Ensure CheckPrivileges runs on resolved table functions While testing privilege checks for the new dolt_diff table function, I realized that CheckPrivileges was being run on UnresolvedTableFunction and not the actual resolved table function. Since the dolt_diff table function pulls data from underlying tables, it needs to be resolved before CheckPrivileges runs to ensure we can accurately check privileges.
• 884: added warnings for some syntaxes of ALTER TABLE and CREATE DATABASE statements
  • Added a warning for using CHARACTER SET, COLLATE or ENCRYPTION syntaxes when creating database. The database is still created.
  • Added a warning for using DISABLE KEYS or ENABLE KEYS options for alter table statement. Nothing changes for the table being altered.
  • Added a system variable that mysqldump uses, innodb_stats_auto_recalc
• 881: More tests for alter table bugs
  1. Split PkOrdinals tests into DDL and DML (new)
  2. New script test that triggers the primary key / secondary key ordinal mismatch
  3. Add ctx.Done() checks to a few key node iterators before operating on child rows (may have missed some, but tried to hit the table, index, and edit iterators) companion PR: https://github.com/dolthub/dolt/pull/3013
• 878: Fix logic with OR operation and add missing nil check to Like MySQL has some really weird behavior where the OR of two non bools evaluates too false. Improperly handling this situation was trickling up to a LIKE panic on the new query added to queries.go

  SELECT "A" OR "A";
  +------------+
  | "A" OR "A" |
  +------------+
  |          0 |
  +------------+
  

• 873: Add support for making fulltext return unsupported feature This pr addresses this issue Cc mysql docs
• 872: sql: expose convert native go value to value Expose convertToValue function from memory package as ConvertToValue. Return an error instead of panicing if the value type is not known.
• 871: Apply bindvars to subqueries Cherry pick from https://github.com/dolthub/go-mysql-server/pull/795 to fix Subquery bindvars specficially.
• 869: Fixed invalid hex number type association from number to blob Fixes https://github.com/dolthub/dolt/issues/2968
• 867: TransformUp is now sensitive to tree modifications TransformUp and related node/expression DFS helper functions expect the visit functions to return an additional boolean parameter indicating whether the visit changed the node:

  type TransformNodeFunc func(Node) (Node, bool, error)
  type TransformExprFunc func(Expression) (Expression, bool, error)
  type Transformer func(TransformContext) (sql.Node, bool, error)
  

  TransformUp's implementation uses the modification information to avoid re-creating a node with identical children:

  BenchmarkTransformOld
  BenchmarkTransformOld-12          	  396544	      2782 ns/op	    3000 B/op	      51 allocs/op
  BenchmarkTransformOldNoEdit
  BenchmarkTransformOldNoEdit-12    	  407797	      2731 ns/op	    2936 B/op	      50 allocs/op
  BenchmarkTransformNew
  BenchmarkTransformNew-12          	 4584258	       254.1 ns/op	      96 B/op	       5 allocs/op
  BenchmarkTransformNewNoEdit
  BenchmarkTransformNewNoEdit-12       	 4782098	       237.8 ns/op	      96 B/op	       5 allocs/op
  

  We use plan.InspectUp when possible, and then plan.TransformUp where possible, resorting to the more expensive plan.TransformUpCtx and plan.TransformUpCtxSchema only when necessary.
• 862: fix misspelled name in information_schema.innodb table
• 861: have methods for dolt to call to parse check expression To properly merge check constraints, Dolt needs to partially parse their expression strings. This PR gives Dolt access to some helper methods, to complete this operation. Should have no functional effect on GMS.
• 860: Add all non INNODB information_schema tables. This PR adds a bunch of information_schema tables that were previously missing. All are initialized with an empty row iter.
• 859: add all information_schema.innodb_* tables as empty tables
• 858: New Foreign Key Implementation Baseline PR containing foreign key progress. As foreign keys won't be ready to merge until they're fully completed, this PR will serve as the baseline that future foreign key PRs are based on, so that changes are digestible. Once a PR is approved, it will be added to this one. In addition, this serves as a synchronization point for merging main changes to manage conflicts.
• 856: Only skip updating defaults when added column has no default value AND is nullable Fix for bad PR #855 which fails the following Dolt test: http://github.com/dolthub/dolt/blob/51987ab84ef9d07b3c6ceae3ab65b9ca65ab55bd/go/libraries/doltcore/sqle/sqlddl_test.go#L482-L482 Verified this PR passes Dolt tests. Also, removes TestAddAndDropColumn test which will be replaced with an appropriate bat test on Dolt. Related to https://github.com/dolthub/dolt/issues/2939
• 855: Don't apply defaults to all rows when the added column has no default Related to https://github.com/dolthub/dolt/issues/2939 When a user was working with a large table, adding a column with no default caused an OOM crash. The crash is a result of the ADD COLUMN sql statement causing every row to be modified in the table. This is not-expected and a bug. We expect adding a column with no default to only modify the schema and not any Dolt row. The fix is to skip applying defaults to table if the added column has no default. We can call this a "fast" add column since only the schema has to be modified. In the future, we may also add a "fast" drop column where only the schema is modified. Currently, we modify every row for a DROP COLUMN statement. In the "fast" drop scenario, it might be important to ensure values aren't retained across ADD COLUMN and DROP COLUMN statements with the same column name. This PR also adds tests to ensure this.
• 854: Pk ordinals delete test
• 853: removing routine_definition column from information_schema.routines table in enginetest as it gives different values in GMS and Dolt The value in routine_definition comes out different:
  • in Dolt, Exchange(parallelism=3)\n └─ Project(6)\n └─ Table(dual)\n
  • in GMS, Project(6)\n └─ Table(dual)\n
• 852: add information_schema.routines table
• 851: add collation_character_set_applicability table in infoSchema
• 850: Added sql.FilteredIndex to allow integrators to specify index filter pushdowns
• 849: Implemented sort node, number type, and handler support for Row2, Expr2, Type2
• 848: Add FLUSH PRIVILEGES support
• 846: analyzer: Collect all range expressions for multi-column indexes
• 845: Fix InTuple optimizer bug Last week we fixed a dropped optimizer error return that was masking a tuple bug. We expected a slice of interfaces, but single element tuples return one value. We catch this in most circumstances, but missed the multicolumn case triggered by an AND expression. The error was passing logictests because we still return correct results even if we fail to push a filter into a table scan.
• 842: /.github/workflows/bump-dependency.yaml: pin go version
• 841: Bump vitess while automatic bumping is broken
• 840: /.github/workflows/bump-dependency.yaml: add GOOS env
• 839: server: Refactor handler to use buffered channel
• 838: Table function support Adds support for table_functions in go-mysql-server. Depends on:
  • https://github.com/dolthub/vitess/pull/131
• 837: Add IGNORE behavior for CHECK CONSTRAINTS and on duplicate key Fixes an issue where INSERT IGNORE was not correctly ignoring check constraints
• 836: fix set serialization
• 833: Fixing a race condition by no longer updating the UserVar's type during Eval The type was being set when retrieved in Eval for completeness, but it shouldn't be needed. When used as a value, the analyzer was already using the NewUserVarWithType method that explicitly loaded and set the type. The NewUserVar method (which doesn't set type) should only be used when the user var is the left hand side of a set var operation.
• 832: add processlist table to information_schema The result is the same as current SHOW PROCESSLIST query result
• 829: Added additional privilege tests Added new testing structure for privileges I'm calling "quick tests". A test can take up 6 lines rather than 18, which is a big difference when there's 100 of them. Plus they're quicker to write this way. They're the same is normal tests though.
• 828: Correct primary key column output for show create table Used the column order before, instead of the key order
• 827: fix ROLLBACK; fail with no database selected MySQL lets ROLLBACK when no database is selected
• 826: Don't call ProcessList.Update on every row spooled in a result set by default
• 825: add information_schema.statistics table Cardinality is currently set to total number of row in the table instead of an estimate number of unique values in the index
• 824: analyzer doesn't error for invalid value re: https://github.com/dolthub/go-mysql-server/issues/815 Originally we were failing to handle an analyzer error. Catching that error exposed a case where we want to manually prevent that error path from bubbling up.
• 823: sql: Added destination buffer parameter to Type.SQL()
• 820: Add uint types to LiteralToInt Re: issue https://github.com/dolthub/go-mysql-server/issues/799 This database and this query were parsing 128 as a uint8, which my type switch missed:

  $ dolt clone https://doltremoteapi.dolthub.com/post-no-preference/stocks
  $ cd stocks
  $ dolt sql
  stocks> select date, act_symbol, avg(close) OVER (PARTITION BY act_symbol ORDER BY date ROWS BETWEEN 128 PRECEDING AND CURRENT ROW) AS ma200 FROM ohlcv WHERE act_symbol='AAPL' having date = '2022-02-11';
  offset must be a non-negative integer; found: 128
  

  After fix:

  select date, act_symbol, avg(close) OVER (PARTITION BY act_symbol ORDER BY date ROWS BETWEEN 128 PRECEDING AND CURRENT ROW) AS ma200 FROM ohlcv WHERE act_symbol='AAPL' having date = '2022-02-11';
  +-------------------------------+------------+--------------------+
  | date                          | act_symbol | ma200              |
  +-------------------------------+------------+--------------------+
  | 2022-02-11 00:00:00 +0000 UTC | AAPL       | 158.29837209302272 |
  +-------------------------------+------------+--------------------+
  

  I haven't been able to create a testing database with the same type parsing behavior yet. Something about the stocks database or that specific query is yielding a types.Value with value=128 and type=sql.Uint8.
• 819: fix BETWEEN function of missing inclusive logic BETWEEN() does inclusive check including checks with NULL value
• 817: Changing UserVar to return its correct type Changing UserVar to return its correct type, when used as a value in an expression. Resolves: https://github.com/dolthub/go-mysql-server/issues/790
• 816: /go.{mod,sum}: bump vitess
• 813: /sql/analyzer: Added feature flag for single threaded execution
• 811: fix DropTable enginetest fix DropTableTest in enginetest, was causing failed test in Dolt
• 810: Extending script queries to allow easier testing of warning messages This change lets you specify the expected number of warnings and an expected substring from test queries. It also tidies up some of the logic around running tests with expected warnings. Example test syntax this allows you to write:

  Assertions: []enginetest.ScriptTestAssertion{
  {
  Query:                           "select * from dolt_diff_t;",
  ExpectedWarning:                 1105,
  ExpectedWarningsCount:           4,
  ExpectedWarningMessageSubstring: "unable to coerce value from field",
  SkipResultsCheck:			true,
  },
  },
  

  Needed for: https://github.com/dolthub/dolt/pull/2832
• 809: query plan test for duplicate indexes
• 806: Deduplicate IndexLookup filters Opportunistically eliminate filter expressions when sql.IndexedTableAccess is used for a table scan. The original logic respecting join scoping for applying indexes is unaffected, this simply connects out current filter elimination logic with the expressions used for range scan. This requires integrators implementing sql.IndexLookup to only return rows selected by the lookup filter. Changes were made to ranges and datetime filters To allow integrators to comply to this filtering standard. GMS can now distinguish between NULL and infinity. Tests are added for IS NULL and IS NOT NULL handling for single and multi column expressions. Engine datetimes are not truncated to support date index lookups with the same precision as MySQL. We currently fail to eliminate filter inversions. More work is needed to compare inverted filters to their range scan representation.
• 804: Revert "addColumn" This reverts commit f64b71875501714d69ed9b80c38e2564467d2794.
• 803: Add NamedWindows Node This moves window name validation, resolving, and embedding out of parsing and into the analyzer.
• 802: Add vitess to go mod to support SHOW STATUS LIKE
• 801: Added external functions for persisting Grant Tables This adds the functions necessary to be able to easily persist the data from Grant Tables as JSON for an integrator.
• 797: Added privilege checks for every currently supported statement This adds privileges for every supported statement.
• 796: Framework for RowIter2 and other new data representation interfaces
  • Implement Table2 for memory tables
  • Several low-level node types converted to Node2
  • Special top-level iterator to switch iteration type based on whether the tree is all Node2 or not
• 795: ComPrepare saves partial query plan Prepared statements hidden behind enable_prepared_statements config. Lacking:
  • Full support for versioned questions (select * asof '2021-03-21)). I fixed the panics, but many asof tests return incorrect results.
  • Window function parameters can't be bindVars currently (lag(x,?))
  • Needs a lot more insert ... select tests ComPrepare saves a partially optimized plan for prepared statements. Query execution will use the saved plan when possible. Connections have a private view of saved plans that are deleted on termination. All enginetests ran with prepared queries regarless of BindVar inclusion. Variety of bug fixes exposed by the prepared enginetests.
• 794: Fix drop table from different database Added DropTable case in getTransactionDatabase Added tests to check dropping table in out of scope database Changed ddlNode to CurDatabase that is update with every table being dropped Changed taking array of tables as string type to plan.UnresolvedTable type Updated resolve-table rule in analyzer to support DropTable filtering out non-existent tables.
• 793: Add type checking to auto increment
• 792: Named Windows Build WINDOW clause definitions from AST into plan.Window nodes. Coversion currently happens in parser. This breaks the separation of concerns between parsing and semantic validation to avoid creating new node types to differentiate between Over windows and Window clause windows. A proper refactor probably requires changing the the parse/exec representations for all related nodes to carry info from the parser further before semantic validation.
• 791: Database and table name visibility affected by privileges This limits the visibility of database and table names depending on the privileges associated with an account (whether directly on via an active role). In addition, all engine tests now run with grant tables enabled. Of course not all statements are checking for their respective privilege, but all of the other logic paths are being exercised (especially the new PrivilegedDatabase and ...Provider constructs). I removed some driver implementation as it was literally broken (errors in the file on main) and was obviously not relied on by anything (not even tests), so instead of fixing it for the changes I made, I just removed it altogether.
• 789: fix alter table columns in different database Removed ddlNode and added tests for AddColumn , DropColumn , RenameColumn , DropColumn that check updates with different database tables than current one. Added cases in engine.go for updating the transaction database for some plans including above and CreateIndex , AlterIndex
• 786: Added all static global, database, and table privileges This adds all static global, database, and table level privileges. They parse and persist but do not yet restrict on anything. Future PR will dive into the full set of privilege checking. One major point to note (which will probably bring about confusion otherwise) is that these new tables are implemented as essentially a "view" on top of the mysql.user table. This will significantly help when it comes privilege checking, especially in the context of determining active roles and such. To complement this, PrivilegeSet was almost entirely rewritten. The related files are data_editor_view.go and grant_table_shim.go. With the original direction, I could have 10 different table queries and 10 sets of privilege data to manage with only 2 roles. This way, I'll only have 3 queries and 3 sets of privilege data in the same scenario.
• 785: No longer mark subqueries in triggers as cacheable Resolves issue https://github.com/dolthub/dolt/issues/2674
• 784: fix alter table add pk Added case for getting non current database name in transaction for AlterPk
• 783: recursive CTEs RecursiveCte node impl with tests for joins, group by, and subquery aliasing. RecursiveCte's are not supported in DML statements. Nested recursive CTE's and nested subqueries are not supported yet. Aliasing is not neatly split down the middle on either side for RCte unions; only the dual table can have alias clashes on either side of the CTE for now. Cyclical recursion will timeout at depth 20. This will likely need a rewrite with better global aliasing to support infinitely nestable RecursiveCte's. Indexed table lookups are not applied to RecursiveCte tables currently because of a bug in the memory impl.
• 782: migrate invalidArgument in time.go to errors.go
• 781: Expose Typ field in sql.RangeColumnExpr
• 780: Expose enginetest.CreateIndex() for testing
• 779: fix json_contains error handling Fixed tests for "Fix JSON Extract on struct" PR
• 778: Properly implemented SHOW GRANTS and SHOW PRIVILEGES SHOW GRANTS was a shell that always returned the same result, and it has now been properly implemented (without USING since active roles aren't in yet). Also added SHOW PRIVILEGES.
• 777: Add PeerGrouper, simplify count and rank window funcs. Count has a default range framer now, matching MySQL and Postgres's behavior. Move custom peer grouping logic into a formal framer to simplify ranking agg functions, which should simplify other rank funcs in the future.
• 775: add groupby column check Any alias column that is non alias table column will not have table name assigned.
• 774: Prevent invalidating table check constraints when altering table schema Updated logic for dropping and renaming columns to ensure that no table checks are referenced. Further improvements should eventually be made to ensure that table check constraints aren't modified by other schema alterations, such as modifying a column's type.
• 773: Added roles, revoking, and dropping users and roles This adds roles, along with implementing the accompanying statements and functionality. In addition, we may now revoke privileges and roles, and also drop users and roles.
• 772: ST_LATITUDE and ST_LONGITUDE functions
• 771: RANGE window framing with rangeFrameBase Support RANGE window frames, adds unit tests for framing and enginetests for queries. Count needs to be rewritten to support RANGES. Better validation for range expression/bound matching would be helpful.
• 770: fix not between function and or logics Fixed handling NOT BETWEEN function was incorrect Added missing logic for OR operator
• 769: Fixed a bug (just introduced) in resolving default column expressions Unbreaks dolt The reason that this bug didn't break the in-memory implementation is it keeps its column defaults reified as Expressions, so it doesn't really use most of the related analyzer machinery
• 768: go.mod,.github: Bump to 1.17.
• 766: Fix panics and errors introduced by recent changes to column default resolution This will fix https://github.com/dolthub/dolt/issues/2659 when picked up by dolt
• 764: Window frame support This PR adds sql.WindowFrame and sql.WindowFramers to represent window frames from AST -> plan -> exec. Every unique and valid combination of window frame Start and End bounds has a plan node (sql.WindowFrame) and a framer (sql.WindowFramer) pair. The frames, framers, and constructor functions that map from AST -> plan and plan -> exec are generated by optgen.
• 763: Basic privilege granting and checking This adds an extremely basic implementation of privilege checking, along with the ability to grant a subset of privileges. In addition, connections carry along their authentication information, and direct queries on the engine (through use of the *sql.Context) also carry auth information for later privilege checking.
• 762: Refactored how grant tables are stored This refactors a lot of previously-written code to more manageably handle future modifications to the grant tables, and to ease their use and reduce potential development mistakes.
• 761: ST_SWAPXY function
• 760: ST_DIMENSION function TODO: missing cases for multi geometry types
• 759: Revert "add ability to override logger (#758)" This reverts commit 2a4f3ea458b57c97bc3f1b70d697e103f62be6ef.
• 758: add ability to override logger
• 756: Adding support for non-string key types in JSON_OBJECT Resolves: https://github.com/dolthub/dolt/issues/2626
• 755: sql/expression/function: json_{array,object}.go: Fix to handle JSONValue subexpressions as well.
• 753: go.mod: Bump vitess, pick up parser simplifications.
• 749: GroupBy normalization rule maintains parent Projection dependencies We flatten GroupBy aggregations to isolation agg functions, but in the process lose nested column dependencies. The downstream qualifyColumns rule was erroring, but without passthrough projections binary expressions like Arithemtic(Sum(x.i), y.i) will fail at execution time without a full set of input dependencies. For this query:`

  select sum(x.i) + y.i from mytable as x, mytable as y where x.i = y.i GROUP BY x.i
  

  We correctly identify that the GroupBy node has one primary aggregation, Sum, and project the Arithmetic separately.

  GroupBy -> Sum(x.i) + y.i -> ... TableScan (x,y)
  =>
  Project (Arithmetic(sum(x.i) + (y.i)))-> GroupBy(Sum(x.i)) -> ...
  

  The Project node fails downstream trying to lookup the y.i dependency we discarded in the transform. This PR adds dependencies back to cover the new parent Project for GroupBy flattening.

  GroupBy -> Sum(x.i) + y.i -> ... TableScan (x,y)
  =>
  Project (Arithmetic(sum(x.i) + (y.i)))-> GroupBy(Sum(x.i), y.i) -> ...
  

• 746: Window exec uses framing iterator and support agg functions in windows plan.Window is converted into new WindowIter backed by the framing setup. Codegen unary agg function nodes, including executable and unit testing harness. Rewrite window aggregation functions for framing. More unit tests that will make it easier to debug WindowBlock/Iter Doc comments for the new WindowIter exec stack.
• 745: Added password validation for logging in Now when creating a user, you can specify a password and log in according to the password given. Denies all incorrect password or invalid users. Also added a killswitch in the grant tables to disable this functionality since it's not yet ready.
• 744: /enginetest: Add a primary key to 'specialtable'
• 743: Fixes pushdownGroupByAliases bug with repeated projection column name. The analyzer rule stored column names in a hash map to mirror deps on either side of the GroupBy and Project child node. The return expressions were recreated without the table info, which is fine for most cases but breaks when there are deps from two tables with the same column name. This maintains table metadata for *expression.UnresolvedColumn throughout the rule, so that the output projections maintain table identity.
• 741: add JSON_ARRAY() function
• 740: one more reentrant lock removal
• 739: /sql/analyzer: Remove reentrant locking from Catalog
• 737: Adding functions st_asgeojson and st_geomfromgeojson
• 736: Fixed bug for JSON values in CASE expressions
• 735: server/context.go: getOrCreateSession: Use a safer .mu.Unlock construct in case s.NewSession panics.
• 734: Fixed example in README Fix mysql command line mismatch the example code
• 733: add JSON_MERGE_PRESERVE() function
• 732: Allow conversion of zero values of other types to date / timestamp values This is for compatibility with postgres dumps and other tools that expect it
• 731: Revert groupby change for partitionless agg functions GroupBy can't switch it's execution runtime to the same as Window without a significant perf hit for select count(primary_key) from table. We could simplify group by iter/its aggregation functions now that it's just a for loop off a child iter.
• 729: Lazy evaluation for column defaults
• 728: Adding SRID and Axis Options for WKT and WKB functions
• 727: Ensures that check constraints cant be added to table that violate th… …e rule
• 726: add int uint conversion
• 724: GroupBy window framing refactor This PR adds:
  • design for window framing
  • new agg functions for window frames
  • GroupBy refactor using framing (frame = partition) Aggregations will have two execution paths while I am swapping out the Window execution layer:
  • In GroupBy node, agg functions will use window frame setup
  • In Window nodes, agg functions will continue to use AggregationBuffer Other:
  • The aggregation package is a bit of a mess right now. I will delete all of the old aggregation and window code during the window refactor.
  • I was 50/50 on whether or not we should use a special iterator for groupBy. It was not too hard to extend the windowBlockIterator, which should drop-in work for regular windows.
• 723: Apply Defaults in the AddColumn node This pr
  1. Adds Defaults in the Add Column Node
  2. Adds a nullability check in the Update node
• 722: Bug fix for using TRIM function in CHECK constraints
• 721: Adding Support for SRID Adds srid field to existing spatial types, and st_srid function: https://dev.mysql.com/doc/refman/8.0/en/gis-general-property-functions.html#function_st-srid. Will add SRID option to wkb and wkt function in separate PR.
• 720: Allow deterministic functions in CHECK constraints
• 719: Fixed a bug in trigger execution involving update statements in BEGIN blocks This fixes https://github.com/dolthub/dolt/issues/2534
• 718: Adding Well Known Binary Conversion Functions Functions:
  • st_asbinary
  • st_aswkb
  • st_pointfromwkb
  • st_linefromwkb
  • st_polyfromwkb
  • st_geomfromwkb
• 717: add TIMESTAMPDIFF() function
• 716: Detect and give a reasonable error message for SHOW TABLES with no current db Fixes https://github.com/dolthub/dolt/issues/2559 This also moves the resolve databases rule into a different analyzer phase and fixes a string of issues that arise when that move occurs.
• 715: Add lag function. Spec: LAG(expr, offset(default = 1), default(default=nil)).
• 712: INSERT IGNORE data conversion behavior This PR addresses behavior for INSERT IGNORE with regards to data conversion errors.
  1. Changes column value to 0 values on incorrect types
  2. Truncates strings that are too long cc. https://dev.mysql.com/doc/refman/8.0/en/sql-mode.html#sql-mode-strict
• 711: Adding Well Known Text Conversions These are functions that allow a user to create spatial types using the Well Known Text (WKT) format. Functions:
  • point_from_text
  • line_from_text
  • poly_from_text
  • geom_from_text
  • as_text
  • as_wkt
• 710: Adding functions ST_X and ST_Y These are spatial type functions that work specifically with point.
• 708: sql/parse: Update sql/parse/parse.go for dropping vitess partial ddl support.
• 704: Basic CREATE USER implementation This adds the minimum for the CREATE USER statement to work, in that we have an in-memory representation of the grant tables, which may be accessed either directly (by the mysql database) or through the user statements (of which we only have CREATE USER for now).
• 701: add DATEDIFF() func
• 699: server/handler.go: Close the row iter in doQuery with the same context we run the query with. If a row iter implementation captures the |ctx| from RowIter() or Next() calls, and that |ctx| is canceled before the Close() call happens, then the implementation may return an error on the canceled context. It is better for our existing implementations if the Close() calls happen with the same context as RowIter() and Next().
• 697: sql/analyzer: Fix a bug where we do not use indexed joins for some queries with offset or sort+limit[+offset].
• 696: Fix boolean SQL conversion panic Fixes a panic happening when returning Boolean values from the database.
• 695: Add SQL func time_format Add SQL func time_format Also took the time to refactor time parsing to match MySQL behavior closer, even its weirdest parts. Fixes a bug in Time.Convert that led single-digit micro-second times to be parsed as tenth of seconds, e.g. 14:00:00.00007 was parsed as 14:00:00.7. Note: I'm trying to achieve minimum compatibility with Metabase's MySQL connector, which required some functions (see #694)
• 694: Add Locate SQL function Add Locate SQL function. I'm trying to achieve minimum compatibility with Metabase's MySQL connector, and it requires a couple new functions, so you'll see a few more PRs from me. Reference: https://www.w3schools.com/sql/func_mysql_locate.asp
• 693: Parsing statements dealing with users and privileges This just parses all of the statements from their vitess representation to their GMS representation.
• 691: sql: RowIter,IndexKeyIter,etc.: Add ctx parameter to Next().
• 690: Adding spatial types: point, linestring, and polygon Able to create table with column of point, linestring, and polygon type. Able to call some geometry constructors to create and display a point, linestring, and polygon. Added functions: st_pointfromtext, st_linestringfromtext, st_polygonfromtext and tests
• 689: Added test utility method for IndexTableAccess
• 686: server/handler,sql/plan/row_update_accumulator: Move the logic for generating proper OkResult values for RowsAffected and InsertId to the engine. This makes RowUpdateAccumulator responsible for returning the right OkResult, instead of having the server/handler hack up the values in the result after the fact. A few enginetests needed to be updated because they asserted the wrong values for InsertId coming back in the OkResult. It also makes RowUpdateAccumulator responsible for setting FoundRows, RowCount and LastInsertId. LastInsertId handling is pretty gnarly becuase we currently don't have it coming back from the insert iterator. It should be moved there.
• 685: server event listener
• 684: sql/plan: load_data: Move LoadInfile logic from server/handler to the plan node. This puts the LoadInfile functionality as a Service on the sql.Context and has the plan node call into it. This is a better place for it in general, rather than inspecting the resulting plan tree in server/handler and calling adhoc methods on the *mysql.Conn based on what we find. Also addresses a number of (potential) issues with the existing load data local functionality including:
  • Race-y file writes and reads from a single file (.LOADDATA) for concurrent requests.
  • Incorrect handling of scanner.Err() checking; we now correctly return the errors when they occur, instead of deferring them to the Close() call.
  • Avoids sending LoadInfilePacket for a non-LOCAL LOAD DATA call. The most complected part of the new implementation is the ordering constraint around RowUpdateAccumulator iterator calling Close() on the row source before it returns an Ok result from the Next() call. An alternative approach would be to copy the contents to a temporary file in LoadData.RowIter().
• 683: Changed Range overlap algorithm Implemented a tree structure for ranges based on https://en.wikipedia.org/wiki/Interval_tree#Augmented_tree. I tested this against a benchmark that replicated the slowdown seen in Dolt release 0.34.5 and saw linear scaling, just as we'd hope. There are some cases that are handled very inefficiently, and I left some strategies on how to solve them listed in the comments, which I will probably implement at a later time.
• 682: Alter table can now modify the auto increment attribute of a column As part of this work, moved (most of) the validation work out of the query nodes and into the analyzer. Added a bunch of new tests. Fixed some bugs in the memory table this uncovered.
• 681: Return Uint64 instead of int64 in charsets table
• 680: Window aggregation buffer sharing test Re: https://github.com/dolthub/dolt/issues/2448 Dolt diff table aggregation queries were bleeding results. We still use buffer sharing, but this test should prevent regressions for the same issue.
• 679: server/handler.go: Migrate doQuery rows loop to errgroup. Previously this used a |quit| signaling channel and an |errChan| to communicate any errors. We didn't get context cancelation in all cases and our call to |rows.Close()| raced with the background thread's call to |rows.Next()|.
• 678: server,sql/p{arse,lan}: Move handling of KILL statements to plan nodes instead of adhoc regex parsing in server handler.
• 676: Bug fix for multi-column index matching and test for same
• 675: Design for RowFrame
• 673: fix cross join panic The cross join expression check would panic for any Comparator outside of the whitelist, instead of returning false to indicate an unmatchable expression.

  > SELECT a6, x5, b2+a6, e4  FROM t6, t4, t5, t2 WHERE b4 in (434,267,535,407,331,2,921,469)   AND c5=819   AND d6 in (192,129,970,151)   AND (c2=728 OR 806=c2 OR 141=c2)
  panic: runtime error: invalid memory address or nil pointer dereference
  [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1e2c6d3]
  goroutine 1 [running]:
  github.com/dolthub/go-mysql-server/sql/expression.(*GetField).Name(...)
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/expression/get_field.go:82
  github.com/dolthub/go-mysql-server/sql/analyzer.comparisonSatisfiesJoinCondition({0x2d03fbe8, 0xc0013df5e0}, 0xc0013df580)
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/analyzer/replace_cross_joins.go:50 +0x2d3
  github.com/dolthub/go-mysql-server/sql/analyzer.expressionCoversJoin.func1({0x29e5070, 0xc0013df5e0})
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/analyzer/replace_cross_joins.go:61 +0x5b
  github.com/dolthub/go-mysql-server/sql/expression.InspectUp.func1({0x29e5070, 0xc0013df5e0})
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/expression/transform.go:61 +0x39
  github.com/dolthub/go-mysql-server/sql/expression.InspectUp.func2({0x29e5070, 0xc0013df5e0})
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/expression/transform.go:68 +0x27
  github.com/dolthub/go-mysql-server/sql/expression.TransformUp({0x29e5070, 0xc0013defc0}, 0xc00016dfc8)
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/expression/transform.go:53 +0x13d
  github.com/dolthub/go-mysql-server/sql/expression.InspectUp({0x29e5070, 0xc0013defc0}, 0xc00016e020)
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/expression/transform.go:67 +0xad
  github.com/dolthub/go-mysql-server/sql/analyzer.expressionCoversJoin({0x29e5070, 0xc0013defc0}, 0x3cd6108)
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/analyzer/replace_cross_joins.go:58 +0x3f
  github.com/dolthub/go-mysql-server/sql/analyzer.replaceCrossJoins.func1.1({0x29d3b00, 0xc0013df580})
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/analyzer/replace_cross_joins.go:93 +0x318
  github.com/dolthub/go-mysql-server/sql/plan.TransformUp.func1({{0x29d3b00, 0xc0013df580}, {0x29d3b00, 0xc0013df440}, 0x0, {0x37ff6f8, 0x0, 0x0}})
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/plan/transform.go:98 +0x4f
  github.com/dolthub/go-mysql-server/sql/plan.transformUpCtx({{0x29d3b00, 0xc0013df420}, {0x29d3b00, 0xc0013df440}, 0x0, {0x37ff6f8, 0x0, 0x0}}, 0x0, 0xc00016e778)
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/plan/transform.go:91 +0x6ba
  github.com/dolthub/go-mysql-server/sql/plan.transformUpCtx({{0x29d3b00, 0xc0013df440}, {0x29d3b00, 0xc0013df460}, 0x0, {0x37ff6f8, 0x0, 0x0}}, 0x0, 0xc00016e778)
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/plan/transform.go:72 +0x41f
  github.com/dolthub/go-mysql-server/sql/plan.transformUpCtx({{0x29d3b00, 0xc0013df460}, {0x29d3fc8, 0xc0013df480}, 0x0, {0x37ff6f8, 0x0, 0x0}}, 0x0, 0xc00016e778)
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/plan/transform.go:72 +0x41f
  github.com/dolthub/go-mysql-server/sql/plan.transformUpCtx({{0x29d3fc8, 0xc0013df480}, {0x0, 0x0}, 0xffffffffffffffff, {0x37ff6f8, 0x0, 0x0}}, 0x0, 0xc00016e778)
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/plan/transform.go:72 +0x41f
  github.com/dolthub/go-mysql-server/sql/plan.TransformUpCtx({0x29d3fc8, 0xc0013df480}, 0x100c00016e7e8, 0x100e674)
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/plan/transform.go:53 +0xbd
  github.com/dolthub/go-mysql-server/sql/plan.TransformUp({0x29d3fc8, 0xc0013df480}, 0x203000)
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/plan/transform.go:97 +0x45
  github.com/dolthub/go-mysql-server/sql/analyzer.replaceCrossJoins.func1({0x29d3fc8, 0xc0013df480})
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/analyzer/replace_cross_joins.go:85 +0x159
  github.com/dolthub/go-mysql-server/sql/plan.TransformUp.func1({{0x29d3fc8, 0xc0013df480}, {0x29d45f8, 0xc0013cd200}, 0x0, {0x37ff6f8, 0x0, 0x0}})
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/plan/transform.go:98 +0x4f
  github.com/dolthub/go-mysql-server/sql/plan.transformUpCtx({{0x29d3fc8, 0xc0013df2e0}, {0x29d45f8, 0xc0013cd200}, 0x0, {0x37ff6f8, 0x0, 0x0}}, 0x0, 0xc00016eca0)
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/plan/transform.go:91 +0x6ba
  github.com/dolthub/go-mysql-server/sql/plan.transformUpCtx({{0x29d45f8, 0xc0013cd200}, {0x0, 0x0}, 0xffffffffffffffff, {0x37ff6f8, 0x0, 0x0}}, 0x0, 0xc00016eca0)
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/plan/transform.go:72 +0x41f
  github.com/dolthub/go-mysql-server/sql/plan.TransformUpCtx({0x29d45f8, 0xc0013cd200}, 0x1d9c087, 0xc0013b6ec0)
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/plan/transform.go:53 +0xbd
  github.com/dolthub/go-mysql-server/sql/plan.TransformUp({0x29d45f8, 0xc0013cd200}, 0xc0007dbb01)
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/plan/transform.go:97 +0x45
  github.com/dolthub/go-mysql-server/sql/analyzer.replaceCrossJoins(0x26a3cc2, 0x26a0458, {0x29d45f8, 0xc0013cd200}, 0xc0013cd200)
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/analyzer/replace_cross_joins.go:78 +0x45
  github.com/dolthub/go-mysql-server/sql/analyzer.(*Batch).evalOnce(0x29d45f8, 0xc00096d740, 0xc0007dbb60, {0x29d45f8, 0xc0013cc5d0}, 0x104e534)
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/analyzer/batch.go:93 +0x1e7
  github.com/dolthub/go-mysql-server/sql/analyzer.(*Batch).Eval(0xc0006b9350, 0x37ff6f8, 0xc0007dbb60, {0x29d45f8, 0xc0013cc5d0}, 0x1)
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/analyzer/batch.go:54 +0x145
  github.com/dolthub/go-mysql-server/sql/analyzer.(*Analyzer).analyzeWithSelector(0xc0007dbb60, 0xc00054eb40, {0x29d45f8, 0xc00096d740}, 0x0, 0x271def0)
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/analyzer/analyzer.go:376 +0x314
  github.com/dolthub/go-mysql-server/sql/analyzer.(*Analyzer).Analyze(...)
  /Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/analyzer/analyzer.go:348
  ...
  

• 672: Automatically Generate README File Doesn't include argument types for functions. Janky solution to creating Instances of FunctionN. Might be difficult to edit README now.
• 668: fix _example/main.go sql.Schema -> sql.PrimaryKeySchema
• 667: sql/parse: Remove adhoc parsing of certain structures with regexes and custom lexers. Moves the following to vt/sqlparser:
  • SHOW VARIABLES
  • SHOW WARNINGS Gets rid of unnecessary special handling for SET and SHOW PROCESSLIST.
• 665: SqlEngine tracks BackgroundThreads SqlEngine manages the lifecycle of async background threads through a BackgroundThreads object. Integrators can add goroutines before or after SqlEngine initialization, but finalizing the engine will terminate the threads. Async goroutines are expected to terminate when the context provided by the thread manager is cancelled. The BackgroundThreads instance uses a wait group to allow threads to drain gracefully before exiting.
• 663: Restrict HASH IN conversion to supported operators. We were converting TupleIn to HashIn for expressions that HashIn could not evaluate. This change makes explicit which left and right children we permit for HashIn conversion, and expands HashIn.Eval to evaluate those expression classes. Added expression unit tests, analyzer unit tests, query enginetests, and query planner enginetests for more sets of expressions that we support and do not support. Additionally, this simplifies the expression evaluation hashing technique. The new behavior has not been verified with deeply nested tuple expressions.
• 661: server,sql/parse: ComQuery, Parse updates for better multi statement support.
• 657: Fix mysql commands in README
• 656: Removing unnecessary context creation
• 651: Fixed IS NOT NULL and IN not being recognized with composite indexes Fixes https://github.com/dolthub/go-mysql-server/issues/650
• 649: server: Add option to Config to DisableClientMultiStatements.
• 648: Added check for conflicting column definitions Checks for attempt to create primary key on nullable column Added tests in parse_test.go for new error.
• 646: faster integral type conversions we spend up to 10% of sysbench tests in this function
• 645: Fix Context Creation
• 642: Track PRIMARY KEY column ordering with a sql.PrimaryKeySchema struct. The parser exposes PRIMARY KEYs in the TableSpec for CREATE TABLE, which we pluck before planning. Integrators expected to use the new sql.PrimaryKeySchema even for keyless tables. CREATE TABLE LIKE and CREATE TABLE SELECT are handled separately and need more work to support the same order tracking. sql.PrimaryKeyAlterableTable now includes PrimaryKeySchema() and Pks() methods to access the new schema info.
• 641: Expanded index matches, added many index queries Besides the obvious addition of a ton of tests (ALL of which make use of indexes, which are enforced by the EXPLAIN statements), this also allows for another scenario to match indexes, although we're still missing a fairly important one. I'll list the scenarios below (assume an index over columns (v1, v2, v3). The new one we now match: previously expressions had to match the index prefix, but now it has been expanded to require only the first index expression be matched. Therefore, the filter WHERE v1 = 5 AND v3 = 2 is now used by indexes as we can still filter on v1 = 5 using the same principles that partial indexes use. v3 = 2 may not allow for rows to be skipped on the integrator's side when reading from their internal store, but may allow for them to be skipped when returning to the engine (removing the conversion to a sql.Row from the integrator's implementation which should be a fairly substantial speed increase). This has the added consequence that we can properly combine all ranges that make use of a specific index. That is,

  (v1 > 1 AND v3 > 1) OR (v1 < 4 AND v2 < 4)
  

  will now combine the two ranges (separated by OR). Reducing the prefix limitation has greatly increased the index match possibility. Just to note, the returned indexes are sorted by whether they're an exact match, then by how many expressions are a prefix, then by the index size, and lastly by the index name, so you should always get the "best" index for any given set of expressions. The above works in the case of a single index existing over the three columns. If (v1, v3, v2) is another index then the above example will match to two different indexes (both being a perfect prefix to an index). There is a way to get around this by taking note of which index has been used previously if multiple indexes share the same columns (or some indexes are subsets/supersets of some other indexes), however this was attempted and abandoned due to the added complexity and time it would take to implement. Also built off of this, we cannot match

  (v1 > 1 AND v3 > 1) OR (v2 < 4)
  

  As the (v2 < 4) is missing the first index expression (v1), therefore no index is used here (one is used by MySQL, which was tested to confirm that one should be in this case). Again this would be fixed by taking note of which indexes have already been used, as we only need the prefix to be matched one, and then all other sets of expressions just need to be subsets. Either way, we should now match many more queries than previously.
• 640: Add utilities needed by Dolt import path
• 638: Add support for inserting empty value tuples There was an edge case where if columns were not provided, it required there to be exactly num columns for value tuples, even if there are default values for those columns. Added a helper function existsNonZeroValues to avoid this.
• 637: crossjoins converted to inner join if parent filter has suitable join condition Filter predicates can be pushed into Join nodes under certain conditions:
  • Filter is the parent of a CrossJoin
  • Predicate expressions cover the two Join node dependency trees Filter predicates have to be divided specially to maintain logical equivalence. Only AND conjugates can be split and checked individually for join candidacy. If multiple conjugate trees match the join expression, a new conjugate AND of the subtrees are used as the join condition. Nested CrossJoins should be converted to InnerJoins if the parent filter has suitable join comparisons for each.
• 636: Fixed small issue with used expressions and partial indexes
• 635: plan,analyzer: Implement a TopRowsNode. For a query like: SELECT * FROM table ORDER BY id LIMIT 100 OFFSET 100 we do not need to materialize the entire sorted result. We can use a heap to compute a streaming TopN and output the results when the child iterator is EOF.
  • Add a simple implementation of TopN based on container/heap and sql/expression.Sorter.
  • Add a TopRowsNode and topRowIter.
  • Add a simple analyzer rule to replace a Sort node with a TopRows node when it sits directly underneath a Limit() or directly underneath a Limit(Offset()). Some care is required to keep sql_calc_found_rows operating as expected.
• 634: Fixed bug in string -> integer conversions for large integers (caught by dolt)
• 633: Implement REGEX_REPLACE Implementation for MySQL regexp_replace function Includes function tests and engine tests
• 632: Fixed some expressions not using the largest partial index
• 631: Fixed String() and WithChildren() I accidentally clicked merge on last PR. Fixed String() so that it won't panic on no args Fixed WithChildren() to compare against current length of args
• 630: Accept string float values for integer values
• 629: Added shim to run tests against actual MySQL instance This will allow GMS tests to be run against an actual local instance of MySQL. A few functions aren’t fully implemented (returning foreign keys for example) but pretty much everything else is. This will help with both verification of existing tests (as MySQL would be used as the source of truth) and also for writing tests (easy to see what MySQL returns). For the most part this is just the shim, but I added a single test to show how I’m approaching the GMS tests for mixed operators, so it should be pretty easy to see the utility of the shim. Full index tests will come in a separate PR.
• 628: Fix the global variable usage of load data. Re-add LOAD DATA tests
• 627: add FORMAT() locale Updated FORMAT() function handling locale with some locale test cases skipped.
• 626: Add ability to specify precision to current_timestamp function
• 625: Adding enginetest cases for IP functions Also change some functions to take interface pointer
• 622: implement FORMAT() function Added FORMAT() function for dolt sql
• 621: Implement IP Functions Functions Added: inet_aton inet_ntoa inet6_aton inet6_ntoa is_ipv4 is_ipv6 is_ipv4_compat is_ipv4_mapped Also added tests for each of these functions.
• 619: UNHEX now returns []byte Tests have been adjusted to reflect this change Added another case in function_test for []byte
• 617: hash in supports tuples
  • improve the hash function to handle tuples recursively
  • normalize the left side of the IN comparison so GetField is converted to Literal
  • analyzer will explicitly fail for HASH IN that should convert doesn't (should it still implicitly fall back?) questions:
  • what to do about tuple type conversion? should we store the explicit struct of types and validate as we walk the tree while hashing?
  • am i missing any edge cases i need to test?
  • add max_allowed_packet = 67108864 system variable to limit set size?
• 615: use hash map for IN equality operator between literal values
• 614: Fix for joins on columns that are not a prefix to an index As we now natively support partial indexes in GMS, we had a case where we'd match a single column to a partial index. We would then attempt to use the full index's column expressions which later failed as the columns were not a prefix to the index. Now we enforce that, when we start looking for single column matches, we only find indexes over a single column. Also, the index builder assumed it would be used if it was created, and therefore could return an range of length 0, so that has been fixed as well. It should only be hit in erroneous cases like the preceding one, but this should prevent panics in other circumstances as well.
• 612: Added ability to specify custom escape characters for LIKE There are new test cases for escape syntax. New parameter for Like Expression.
• 609: Reduced index range complexity, removed range overlapping All Ranges now only have a single column expression per range. Ranges no longer overlap. Additional comment changes and whatnot to facilitate both of these changes.
• 608: TRIM string functions + go.mod tidy
• 606: Changed some error names
• 605: Added Support for Full MySQL TRIM Syntax Parser recognizes keywords like: LEADING, TRAILING, and BOTH. TRIM function can now remove prefixes. Fixed LTRIM And RTRIM to not remove all whitespace, and just space characters. Also added functionality for RIGHT function.
• 604: Fix Panic in unique key error generation Poorly written for loop threw an error
• 602: Add case insensitivity to insert
• 601: Fix panic in json_unquote for null values This supercedes https://github.com/dolthub/go-mysql-server/pull/511 and adds tests
• 599: Remove the Index Editor Sorter and add alias support for Update Join
• 598: sql/analyzer: pushdown.go: Avoid pushing down projects across BEGIN blocks. When an UPDATE is within a begin/end block within a trigger, we should avoid projecting away the full table row.
• 597: sql/analyzer: pushdown_filters: Avoid pushing down filters below Window nodes. If a Filter is above a Window node, the Window node will be aggregating and computing projections based on the rows that it sees. It is not safe to push the filter below the Window node. The parser and analyzer currently work in concert (and must work in concert for Window results to be correct) to keep Filter nodes which do apply to the Window below the Window.
• 596: Add Right/left/Cross/Indexed Joins and Implement Table Edit Accumulator
  1. Add functionality for updates to work with Indexed Joins
  2. Implement a Table Edit Accumulator that writes changes at Close time
  3. Fix some undefined test cases
  4. Add functionality for Update Join w/ Right,Left,Cross joins
• 595: persistent session interface
• 594: sql/plan: Fix join iterators to always Close() their secondary RowIter. Fix Subquery to Dispose its subquery.
• 585: Changed index interface, support partial keys

vitess

• 168: Remove glog dependency, reduce log levels, remove all logutilpb usage and support.
• 167: go/vt/{vttls,tlstest}: Sync from upstream; supports CRL, minimum TLS version configuration.
• 166: proto: Move .proto files to dolthub/vt subdirectory and package name. Removes global proto registry conflict with vitessio/vitess, as per #155.
• 165: Change the prepared statements return value
• 164: add parsing for SRID Added parsing for SRID value SRID value is an integer value that can be upto 2^32-1. Other fixes:
  • Allowed .SQLType() function to evaluate types that are capitalized strings and added tests for it
  • Added NULL column option in formatting and added tests for it
• 162: Adding the SIGNED keyword for numeric type column definitions SIGNED is the default, so specifying it is allowed in SQL, but is a no-op. https://dev.mysql.com/doc/refman/8.0/en/numeric-type-syntax.html Found via: MySQL Grammar Crawler
• 161: support parsing analyze table t Adding support to parse analyze statements: https://dev.mysql.com/doc/refman/8.0/en/analyze-table.html
• 160: return Into value in GetInto() in SelectStatement This allows to get Into value without doing switch on its types.
• 159: support parsing of XOR logic operator Added support for parsing XOR logic operator
• 158: support SELECT ... INTO syntax Added supporting of SELECT ... INTO syntax Added tests for cases with different locations of INTO in SELECT statement INTO clause can only be used in one place, but it can come in different positions, before [FROM ...] or after the whole SELECT statement but beforing locking. (mysql recommends it after locking options) INTO OUTFILE ... and INTO DUMPFILE ... are not included yet.
• 157: add all charset introducers and parsing of charset introducers in LIKE function
  • Added all charset introducers supported by MySQL
  • Added parsing charset introducer in front of string literal of LIKE function
• 154: move CURRENT to non-reserved-keyword-list CURRENT is non reserved keyword, it fixes issue of SELECT * FROM current query failing Fixes https://github.com/dolthub/dolt/issues/3224
• 153: Add additional syntax support for CREATE VIEW statement Added ALGORITHM=..., DEFINER=..., SQL SECURITY DEFINER|INVOKER syntaxes for CREATE VIEW statement. Fixed definer_opt to parse account_name instead of ID to support parsing of user@locahost syntax. Moved VIEW (non-reserved keyword) from non-reserved to reserved as non-reserved keywords are used for account_name parsing, so it creates shift/reduce conflict for view.
• 152: support generated columns There is no functionality besides parsing. Fix for: https://github.com/dolthub/dolt/issues/3089
• 151: Parser support for show create table <table> as of <asof> Adding support for parsing show create table <table> as of <asof>. Removing the OnTable field in Show struct that was duplicating the Table field. Two minor formatting fixes.
• 150: Parser support for describe table as of and show columns from table as of Adding parser support for describe <table> as of <asof> and show columns from <table> as of <asof>
• 149: add db name specification to create/drop TRIGGER statements
• 148: add db name specification to create/drop procedure statements Database name is case sensitive but procedure name is not, so TableIdent and ColIdent structs are used for defining db and procedure names, respectively.
• 147: Have SHOW CREATE PROCEDURE <procedure_name> parse TODO: need changes in GMS to handle show create procedure logic.
• 146: /go.mod: require go 1.18
• 144: Fix of View, Trigger, Procedure position params when used in MySQL special comments
• 143: Allow Columns to Start with Numbers...kinda Recognizes identifiers that start with numbers, unless they are immediately after a . TODO: table names that start with numbers do parse, but there is a regex blocks these from parsing in dolt, so will fix in future PR.
• 142: Add DISABLE | ENABLE KEYS syntax for ALTER TABLE statement Added parsing of DISABLE KEYS and ENABLE KEYS options for ALTER TABLE statement Updated checking Index action keyword to be case insensitive
• 140: support CHARSET, COLLATE and ENCRYPTION syntax in CREATE DATABASE statement added parsing of CHARACTER SET, COLLATE and ENCRYPTION options for CREATE DATABASE statement
• 139: Revert "Merge pull request #137 from dolthub/james/special-comments" This reverts commit 19d17da876ea71be47f41092d8ab420885b9ba18, reversing changes made to 0bb5262822024a793f65040a77430ebe0624058e.
• 137: Handle subquery indexes correctly for special MySQL comments Fix for: https://github.com/dolthub/dolt/issues/2980 When Tokenizer detects a special MySQL comment, it just creates a new Tokenizer and embeds it in the old one under the specialComment member variable. This "inner" Tokenizer is then used to handle all the parsing, and its results are just passed to the outer Tokenizer. However, the issue is that the lexer or whatever only reads the Position member variable from the outermost Tokenizer. So my proposed fix is to just copy over the Position of the inner Tokenizer to the outer one (with an offset to handle the leading /*![12345]. I think a better fix might be to just replace the old Tokenizer with the one we create for specialComment?
• 136: Initial implementation of fulltext with indexes and DDL This pr
  1. Adds fulltext parsing support for indexes and DDL.
  2. Formats the repo
• 135: Allowing Event Keywords to be used as column name, table name, variable name, etc
• 134: Get Closer to MySQL's Behavior for Keywords (Reserved and Non-Reserved) I doubt I'll be able to get all of the keywords working immediately. This PR focuses on getting as many of the keywords in and working, and documenting what doesn't work yet.
• 133: reserve found and allow as column safe
• 132: Add FLUSH as a command and its options
• 131: Table function support Add support for table_functions as a new type of table factor in the dolthub/vitess SQL grammar.
• 130: add PROCESSLIST to non-reserved-keyword list
• 129: Adding scripts to test parsing behavior of MySQL
• 128: /go/bucketpool: Optimized findBucket
• 127: Add testcases for exercising keywords and remove some create user tests This PR:
  1. Allows the COMMENT_KEYWORD to be alias by selects/
  2. Removes some usages of aliased keywords in the select queries
  3. Adds skipped tests that exercises the validity of reserved and non reserved keywords through a series of tests
• 126: go/mysql: Add buffering to prepared statement result sets
• 121: Add show like status
• 120: Named Windows Add yacc syntax for window names. WindowDef is used with either its Name field in a WINDOW clause, or NameRef in an OVER clause. NameRef is OVER is not mandatory, but Name in WINDOW is. This PR adds precedence rules to avoid shift/reduce conflicts between the WINDOW name field, which can be any string, and RANGE, PARTITION, ROWS, and RANGE should be prioritized for other rules if they appear as the first token in a window_spec.
• 119: partitionBy and overBy optional, one mandatory with frame def
• 118: Added remaining privileges This adds the remaining privileges.
• 117: recursive CTEs
• 116: tools.go hack.
• 115: go.mod: Bump to go 1.17.
• 114: Remove a bunch of unused stuff.
• 113: .github,go: Cleanup workflows. Fix all tests to work under -race. Fix fragile prototext assertions in tests.
• 112: go,go/vt/proto: Regenerate pb message implementations, upgrade to APIv2.
• 111: go/vt/sqlparser: Get rid of partially parsed statements and {,ddl_}skip_to_end.
• 110: add serial supoort add serial support https://dev.mysql.com/doc/refman/8.0/en/numeric-type-syntax.html#:~:text=SERIAL
• 109: go/mysql: Make all tests pass on macOS. Bumps expected TLS version from 1.2 to 1.3. Adds some string matching for strerror strings that are different between Linux and macOS.
• 108: add support for constraint named unique This is an enhancement for #95 According to https://dev.mysql.com/doc/refman/8.0/en/create-table.html, there will be a format like constraint namedx unique (full_name) in create table statement.
• 107: go/vt/sqlparser: Remove PartialDDL support. There's no need to support partial DDL in dolt.
• 104: Window function frames, extent, and bounds Adds parsing capabilities for window frames, including one sided frames, two sided frames, RANGE value frames, integral and interval based frames, and errors for semantically incorect combinations of frame options. Only value based intervals are currently supported. No prepared statement support for expression bounds.
• 103: go/mysql: conn: Add LoadInfile method which returns a ReadCloser for reading the remote file from a client. We will migrate the implementation in go-mysql-server from HandleLoadDataLocalQuery to this interface. This keeps the responsibility for writing the file and the likes out of the vitess/conn layer.
• 102: Added GRANT, REVOKE, and the remainder of CREATE USER
• 101: go/vt/sqlparser: Parse KILL {CONNECTION,QUERY} statements.
• 100: go/vt/sqlparser: Extend the parser for correctly parsing some extended SHOW syntax.
  • SHOW COUNT(*) {WARNINGS, ERRORS}
  • SHOW {WARNINGS,ERRORS} LIMIT ...[, ...]
  • SHOW {,SESSION,GLOBAL} VARIABLES {LIKE,WHERE} ...
  • SHOW FULL PROCESSLIST
• 99: Added more statements for user handling Added RENAME USER, CREATE ROLE, DROP ROLE, SHOW GRANTS, and a portion of CREATE USER. The rest of CREATE USER, along with GRANT and REVOKE will come in the next PR.
• 97: Added proper syntax parsing for account names, along with DROP USER This PR primarily implements the changes necessary for account names, which required edits in the parser itself. DROP USER was implemented to test the name variations, so refer to the tests in parse_test.go for how account names work (and to see why the changes were necessary). All parser tests were validated against a MySQL 8.0 instance. In a nutshell, account names take the form user@host, and both user and host must be separate strings. The parser treated @ as a standard letter, and we used regex matching in GMS and Dolt to catch "invalid" names (@@system_var is valid but what@@ is invalid). My first pass was to treat the entire account name as a string and split it using a regex, however I could not get it to pass all of the parser tests. This was the only way I could make it work with all tests passing. Notes:
  • The other statements will come in a different PR, which will be straightforward in comparison.
  • DROP USER ''; looks like weird syntax, but an empty name means that all names match. In this case, you can assign some dbs/tables to be available to everyone.
  • Following the above, DROP USER ``; is valid, but failed as we returned an error on empty quoted identifiers. MySQL allows empty quoted identifiers to parse, but rejects them as invalid names depending on the query. This has been fixed.
  • Added ast_permissions.go to put all of the statements dealing with users, grants, etc. For me, ast.go operates very slowly and occasionally hangs for a few minutes, so I'm putting my new statements in a new file for now. I doubt you'll have pushback, but if so I can merge the file back into ast.go when I'm done.
• 96: Better support for MySQL MULTI_STATEMENTS.
• 95: Added support for CONSTRAINT UNIQUE KEY syntax in CREATE TABLE
• 94: add Null to column definition create table t (a int primary key) succeeds create table t (a int not null primary key) succeeds create table t (a int null primary key) fails Create new BoolVal field for ColumnDefinition to more easily check for attempts to create nullable columns with primary key. Also allows people to do create table t(a int null not null ...) if they want.
• 93: go/mysql/conn.go: Add a way to disable client_multi_statements handling on a per-connection basis. Conn's handling of CLIENT_MULTI_STATEMENTS is currently wrong for packets that contain statements which can include other statements (e.g., CREATE TRIGGER). Add a way for an integrator to disable it for now.
• 92: parser can detect insert into () values <tuple_list> added new case for insert_data should be able to correctly parse insert into test_tb values ()insert into test_tb () values ()insert into test_tb values (), (), ...insert into test_tb () values (), (), ...
  91: added FORMAT keyword added format expression keyword
  90: Noop behavior for DROP TABLE CASCASE/RESTRICT
  89: Added Test Cases for LIKE ... ESCAPE ...
  88: Add TRIM functionality Changed sql.y to recognize keywords TRIM, LEADING, TRAILING, and BOTH
  85: can use status w/o quotes for insert statement re: https://github.com/dolthub/dolt/issues/2195

  Closed Issues

  • 1050: Complex query with ABS, !=, REGEXP, and CONVERT fails.
  • 950: README suggests installing via now depreciated "go get"
  • 989: willing to maintaion sql standard schema support?
  • 973: no index record in information_schema.statistics
  • 808: FilteredTable uses wrong schema with alias
  • 911: _example Failure to output
  • 916: RightIndexedJoin commutativity correctness
  • 894: when is version 1.0 api standard set?
  • 900: Does go-mysql-server support create database and table from SQL file?
  • 901: ERROR 1105 (HY000): Error: index already exists
  • 888: The _example/main.go compile error.
  • 815: Index error lost in parent call
  • 790: String variables aren't evaluated correctly in where clauses
  • 799: Aggregate Partition Window Rows beyond 127
  • 146: regex-metrics can't be initialized
  • 171: Two foreign keys in same CREATE TABLE statements triggers error
  • 355: Question/Feature Request: How can I increase the parallelism of expression evaluation?
  • 654: fatal error: concurrent map write read
  • 705: error: can't modify database databaseProvider
  • 709: create table with unnamed key got error ERROR 1105 (HY000): Error: index already exists
  • 191: Support for VALUES in INSERT ... ON DUPLICATE KEY UPDATE
  • 250: unsupported syntax: create database mapping
  • 313: How can I custom my log.
  • 481: Anyone working on the replication?
  • 703: undefined: sql.NewPrimaryKeySchema
  • 662: Automatically generate function docs
  • 664: _example/main.go unable to compile
  • 653: HASH IN fails on arithmetic, function nodes
  • 618: Unhex function returns a string, when it should return []byte
  • 650: IS NOT NULL and IN are causing composite index to be ignored
  • 643: pls dont rush the API 1.0 but do think properly and optimize for performance
  • 172: Support for prepared statements
  • 179: Combine index lookups on different columns of multi-column indexes with different operators
  • 603: Panic when using memory implemenation
  • 584: Proxy support?
  • 155: Vitess protocol buffer namespace conflict