MyGit

v4.0.0-preview-rc1

delta-io/delta

版本发布时间: 2024-06-06 09:34:57

delta-io/delta最新发布版本:v4.0.0-preview-rc2(2024-06-08 07:39:48)

This is a preview of Delta 4.0 Preview. The final release notes are still being generated.


We are excited to announce the preview release of Delta Lake 4.0.0 RC1! Instructions for how to use this release candidate are at the end of these notes. To give feedback on this release candidate, please create issues in our Delta repository.

Delta Spark

Delta Spark 4.0 preview is built on Apache Spark™ 4.0.0-preview1. Similar to Apache Spark, we have released Maven artifacts for Scala 2.13.

The key features of this release are:

More detailed release notes coming soon!

Delta Kernel Java

The Delta Kernel project is a set of Java and Rust libraries for building Delta connectors that can read and write to Delta tables without the need to understand the Delta protocol details). Release notes for this component to come later.

How to use this Release Candidate

Download Spark 4.0.0-preview1 from https://archive.apache.org/dist/spark/spark-4.0.0-preview1.

For this release candidate, we have published the artifacts to a staging repository. Here’s how you can use them:

Spark Submit

Add --repositories https://oss.sonatype.org/content/repositories/iodelta-1147 to the command line arguments.

spark-submit --packages io.delta:delta-spark_2.13:4.0.0rc1 \
--repositories https://oss.sonatype.org/content/repositories/iodelta-1147 examples/examples.py

Currently Spark shells (PySpark and Scala) do not accept the external repositories option. However, once the artifacts have been downloaded to the local cache, the shells can be run with Delta 4.0.0rc1 by just providing the --packages io.delta:delta-spark_2.13:4.0.0rc1 argument.

Spark Shell

bin/spark-shell --packages io.delta:delta-spark_2.13:4.0.0rc1 \
--repositories https://oss.sonatype.org/content/repositories/iodelta-1147 \
--conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
--conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog

Spark SQL

bin/spark-sql --packages io.delta:delta-spark_2.13:4.0.0rc1 \
--repositories https://oss.sonatype.org/content/repositories/iodelta-1147 \
--conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
--conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog

Maven

<repositories>
  <repository>
    <id>staging-repo</id>
    <url>https://oss.sonatype.org/content/repositories/iodelta-1147</url>
  </repository>
</repositories>
<dependency>
  <groupId>io.delta</groupId>
  <artifactId>delta-spark_2.13</artifactId>
  <version>4.0.0rc1</version>
</dependency>

SBT Project

libraryDependencies += "io.delta" %% "delta-spark" % "4.0.0rc1"
resolvers += "Delta" at https://oss.sonatype.org/content/repositories/iodelta-1147

(PySpark) Delta-Spark

Name: delta-spark
Version: 4.0.0rc1
Summary: Python APIs for using Delta Lake with Apache Spark
Home-page: https://github.com/delta-io/delta/
Author: The Delta Lake Project Authors
Author-email: delta-users@googlegroups.com
License: Apache-2.0
Location: /Users/allison.portis/opt/anaconda3/envs/delta-release-4-0/lib/python3.9/site-packages
Requires: importlib-metadata, pyspark
Required-by: 

Note: artifacts delta-flink, delta-iceberg, delta-hudi, delta-hive and delta-standalone are not included in this preview but will be available in a future release

相关地址:原始地址 下载(tar) 下载(zip)

1、 delta-spark-4.0.0rc1.tar.gz 21.31KB

2、 delta_spark-4.0.0rc1-py3-none-any.whl 20.81KB

查看:2024-06-06发行的版本