Gravitino Release 0.5.1

Gravitino 0.5.1 is a stable release based on branch-0.5 that contains bug fixes and introduces some core features for Trino and Spark connectors and security. Additionally, the Python client is ready to use in version 0.5.1.

Core Features

1. Apache Spark Connector Enhancement: Supports multiple Spark (3.3, 3.4, 3.5) and Scala (2.12, 2.13) versions. Introduces several advanced features for Iceberg support like row-level operations, time travel query, procedures, etc.

2. Trino Connector Improvement: Improve the Gravitino trino connector to use Trino's dynamic catalog framework to better support queries in the distributed mode.

3. Security Improvement Supports the condition of privileges, support authentication between Gravitino and Hive, Hadoop catalog.

4. Python client Fixed several critical bugs, enhancing the usability of the Gravitino Python client for real environments.

Gravitino Core

• Supports time sliding window to record metrics #3152
• The semantics of the delete operation for relational storage should be aligned with kv storage #3092
• Provide specific classLoader for Partition operation #2997
• Support preassigned partitions when creating partitioned table #2266
• Remove log4j from client and client java runtime module #3649
• Improve compatibility issues with data type serde #3598

Security

• Privilege should have deny mechanism #3342
• Authentication between the Hadoop catalog and Gravitino. #3337
• Support Kerberos client authentication in gvfs #2711
• Add Hive user authentication end-to-end test #3489
• Add IT to test kerberosized HDFS #3432
• Fix the Hive catalog bug In multiple Kerberized HMS #3295

Catalogs

Iceberg

• Support Iceberg RestCatalog in Gravitino #3149
• Add reserved properties to Table Properties when load an Iceberg table #2962
• Bucket and truncate are missing width params in Gravitino SortOrder #2921

Doris

• Support creating Doris table with partition #3347
• Filter system database when list database for Doris catalogs #3275

MySQL

• jdbc-mysql catalogs failed on mysql 8.x driver #3269
• MysqlTypeConverter cannot handle the datetime data type #1761

Kafka

• Adding a new Apache Kafka catalog works but times out listing topics #3249
• Drop kafka catalog maybe failed after altering #3673

Trino Connector

• Gravitino Trino connector loads the catalog using Trino's dynamic catalog feature #3524
• Support automatically loading catalog to Trino by using CREATE CATALOG command #3601
• Support create Internal connector like hive/iceberg/mysql/pg/memory #3587
• [trino-connector] Support create Gravitino catalog by using Trino CREATE CATALOG command #3526

Spark Connector

• Support multiple Spark and Scala versions #1574
• Remove iceberg runtime from spark connector jar #3396
• Support passing catalog properties to spark connector #3292
• Support iceberg metadataColumns #2587
• Support row-level operations to iceberg Table #2543
• Support Iceberg time travel in SQL queries #3264
• Support Iceberg RestCatalog with HiveCatalog backend in spark-connector #3193
• Support Iceberg Spark Procedure #3186
• Alter table comment though table properties failed #3137
• Spark connector create hive table failed for using hive #3129

Python Client

• Correct type specification of create_fileset function in fileset_catalog.py #3508
• When calling hash() function in name_identifier with python client throw an exception #3521
• Add test coverage tool for client-python #3517
• Add Pylint Rules for client-python naming rules #3205
• Add auto formatter for client-python #3206
• Add Pylint Rules for client-python #3203
• Unspecified pylint version in client-python #3558

UI

• Gravitino icon doesn't show on UI #3413
• UI: the font family of gravitino title text in production mode does not match with dev mode #3386
• With only one metalake UI presents user with a choice #3285
• Gravitino UI when creating a new fileset catalog it seems to be missing location property #3257

Build and Others

• Unable to build Gravitino in Linux environment due to JAVA problem #3336
• ./gradlew test -PskipITs failed in python-client test #3319
• TrinoContainer#checkSyncCatalogFromGravitino will always return false #3237

Limitation and known issues

• The query in Trino connector for 0.5.1 has slower performance than version 0.5.0 #3738

Credits

This release acknowledges the hard work and dedication of all contributors who have helped make this release possible.

@caican00 @ch3yne @danhuawang @diqiu50 @FANNG1 @featherchen @jerqi @kalencaya @LauraXia123 @lw-yang @MukarramHaq @mchades @noidname01 @qqqttt123 @SteNicholas @TEOTEO520 @theoryxu @unknowntpo @xloya @xiaozcy @xunliu @yijhenlin @yuqi1129 @zhoukangcn