Update Spark 3.5 from 3.5.4 to 3.5.5 and Drop Spark 3.4 #18576
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Diff for 97e3b6a:diff --git a/_bashbrew-cat b/_bashbrew-cat
index 635cae1..58dd927 100644
--- a/_bashbrew-cat
+++ b/_bashbrew-cat
@@ -1,65 +1,45 @@
Maintainers: Apache Spark Developers <dev@spark.apache.org> (@ApacheSpark)
GitRepo: https://github.com/apache/spark-docker.git
-Tags: 3.4.4-scala2.12-java11-python3-r-ubuntu
+Tags: 3.5.5-scala2.12-java11-python3-r-ubuntu
Architectures: amd64, arm64v8
-GitCommit: 18c599ec44230c48fc982eb52d6cdf069883a57d
-Directory: 3.4.4/scala2.12-java11-python3-r-ubuntu
+GitCommit: 86f243f6df77cf247f8eb9931558d719e0c56144
+Directory: 3.5.5/scala2.12-java11-python3-r-ubuntu
-Tags: 3.4.4-scala2.12-java11-python3-ubuntu, 3.4.4-python3, 3.4.4
+Tags: 3.5.5-scala2.12-java11-python3-ubuntu, 3.5.5-python3, 3.5.5, python3, latest
Architectures: amd64, arm64v8
-GitCommit: 18c599ec44230c48fc982eb52d6cdf069883a57d
-Directory: 3.4.4/scala2.12-java11-python3-ubuntu
+GitCommit: 86f243f6df77cf247f8eb9931558d719e0c56144
+Directory: 3.5.5/scala2.12-java11-python3-ubuntu
-Tags: 3.4.4-scala2.12-java11-r-ubuntu, 3.4.4-r
+Tags: 3.5.5-scala2.12-java11-r-ubuntu, 3.5.5-r, r
Architectures: amd64, arm64v8
-GitCommit: 18c599ec44230c48fc982eb52d6cdf069883a57d
-Directory: 3.4.4/scala2.12-java11-r-ubuntu
+GitCommit: 86f243f6df77cf247f8eb9931558d719e0c56144
+Directory: 3.5.5/scala2.12-java11-r-ubuntu
-Tags: 3.4.4-scala2.12-java11-ubuntu, 3.4.4-scala
+Tags: 3.5.5-scala2.12-java11-ubuntu, 3.5.5-scala, scala
Architectures: amd64, arm64v8
-GitCommit: 18c599ec44230c48fc982eb52d6cdf069883a57d
-Directory: 3.4.4/scala2.12-java11-ubuntu
+GitCommit: 86f243f6df77cf247f8eb9931558d719e0c56144
+Directory: 3.5.5/scala2.12-java11-ubuntu
-Tags: 3.5.4-scala2.12-java11-python3-r-ubuntu
+Tags: 3.5.5-scala2.12-java17-python3-r-ubuntu
Architectures: amd64, arm64v8
-GitCommit: 6b917ced4279dd7b3a33a81a08db37b3f27e037b
-Directory: 3.5.4/scala2.12-java11-python3-r-ubuntu
+GitCommit: 86f243f6df77cf247f8eb9931558d719e0c56144
+Directory: 3.5.5/scala2.12-java17-python3-r-ubuntu
-Tags: 3.5.4-scala2.12-java11-python3-ubuntu, 3.5.4-python3, 3.5.4, python3, latest
+Tags: 3.5.5-scala2.12-java17-python3-ubuntu, 3.5.5-java17-python3, 3.5.5-java17, python3-java17
Architectures: amd64, arm64v8
-GitCommit: 6b917ced4279dd7b3a33a81a08db37b3f27e037b
-Directory: 3.5.4/scala2.12-java11-python3-ubuntu
+GitCommit: 86f243f6df77cf247f8eb9931558d719e0c56144
+Directory: 3.5.5/scala2.12-java17-python3-ubuntu
-Tags: 3.5.4-scala2.12-java11-r-ubuntu, 3.5.4-r, r
+Tags: 3.5.5-scala2.12-java17-r-ubuntu, 3.5.5-java17-r
Architectures: amd64, arm64v8
-GitCommit: 6b917ced4279dd7b3a33a81a08db37b3f27e037b
-Directory: 3.5.4/scala2.12-java11-r-ubuntu
+GitCommit: 86f243f6df77cf247f8eb9931558d719e0c56144
+Directory: 3.5.5/scala2.12-java17-r-ubuntu
-Tags: 3.5.4-scala2.12-java11-ubuntu, 3.5.4-scala, scala
+Tags: 3.5.5-scala2.12-java17-ubuntu, 3.5.5-java17-scala
Architectures: amd64, arm64v8
-GitCommit: 6b917ced4279dd7b3a33a81a08db37b3f27e037b
-Directory: 3.5.4/scala2.12-java11-ubuntu
-
-Tags: 3.5.4-scala2.12-java17-python3-r-ubuntu
-Architectures: amd64, arm64v8
-GitCommit: 6b917ced4279dd7b3a33a81a08db37b3f27e037b
-Directory: 3.5.4/scala2.12-java17-python3-r-ubuntu
-
-Tags: 3.5.4-scala2.12-java17-python3-ubuntu, 3.5.4-java17-python3, 3.5.4-java17, python3-java17
-Architectures: amd64, arm64v8
-GitCommit: 6b917ced4279dd7b3a33a81a08db37b3f27e037b
-Directory: 3.5.4/scala2.12-java17-python3-ubuntu
-
-Tags: 3.5.4-scala2.12-java17-r-ubuntu, 3.5.4-java17-r
-Architectures: amd64, arm64v8
-GitCommit: 6b917ced4279dd7b3a33a81a08db37b3f27e037b
-Directory: 3.5.4/scala2.12-java17-r-ubuntu
-
-Tags: 3.5.4-scala2.12-java17-ubuntu, 3.5.4-java17-scala
-Architectures: amd64, arm64v8
-GitCommit: 6b917ced4279dd7b3a33a81a08db37b3f27e037b
-Directory: 3.5.4/scala2.12-java17-ubuntu
+GitCommit: 86f243f6df77cf247f8eb9931558d719e0c56144
+Directory: 3.5.5/scala2.12-java17-ubuntu
Tags: 4.0.0-preview2-scala2.13-java17-python3-r-ubuntu
Architectures: amd64, arm64v8
diff --git a/_bashbrew-list b/_bashbrew-list
index bfb3095..99b65f2 100644
--- a/_bashbrew-list
+++ b/_bashbrew-list
@@ -1,27 +1,19 @@
-spark:3.4.4
-spark:3.4.4-python3
-spark:3.4.4-r
-spark:3.4.4-scala
-spark:3.4.4-scala2.12-java11-python3-r-ubuntu
-spark:3.4.4-scala2.12-java11-python3-ubuntu
-spark:3.4.4-scala2.12-java11-r-ubuntu
-spark:3.4.4-scala2.12-java11-ubuntu
-spark:3.5.4
-spark:3.5.4-java17
-spark:3.5.4-java17-python3
-spark:3.5.4-java17-r
-spark:3.5.4-java17-scala
-spark:3.5.4-python3
-spark:3.5.4-r
-spark:3.5.4-scala
-spark:3.5.4-scala2.12-java11-python3-r-ubuntu
-spark:3.5.4-scala2.12-java11-python3-ubuntu
-spark:3.5.4-scala2.12-java11-r-ubuntu
-spark:3.5.4-scala2.12-java11-ubuntu
-spark:3.5.4-scala2.12-java17-python3-r-ubuntu
-spark:3.5.4-scala2.12-java17-python3-ubuntu
-spark:3.5.4-scala2.12-java17-r-ubuntu
-spark:3.5.4-scala2.12-java17-ubuntu
+spark:3.5.5
+spark:3.5.5-java17
+spark:3.5.5-java17-python3
+spark:3.5.5-java17-r
+spark:3.5.5-java17-scala
+spark:3.5.5-python3
+spark:3.5.5-r
+spark:3.5.5-scala
+spark:3.5.5-scala2.12-java11-python3-r-ubuntu
+spark:3.5.5-scala2.12-java11-python3-ubuntu
+spark:3.5.5-scala2.12-java11-r-ubuntu
+spark:3.5.5-scala2.12-java11-ubuntu
+spark:3.5.5-scala2.12-java17-python3-r-ubuntu
+spark:3.5.5-scala2.12-java17-python3-ubuntu
+spark:3.5.5-scala2.12-java17-r-ubuntu
+spark:3.5.5-scala2.12-java17-ubuntu
spark:4.0.0-preview2
spark:4.0.0-preview2-java21
spark:4.0.0-preview2-java21-python3
diff --git a/_bashbrew-list-build-order b/_bashbrew-list-build-order
index d99b479..65155be 100644
--- a/_bashbrew-list-build-order
+++ b/_bashbrew-list-build-order
@@ -1,17 +1,13 @@
-spark:3.4.4-scala
-spark:3.4.4-scala2.12-java11-python3-r-ubuntu
-spark:3.5.4-java17-scala
-spark:3.5.4-scala2.12-java17-python3-r-ubuntu
+spark:3.5.5-java17-scala
+spark:3.5.5-scala2.12-java17-python3-r-ubuntu
spark:4.0.0-preview2-java21-scala
spark:4.0.0-preview2-scala
spark:4.0.0-preview2-scala2.13-java17-python3-r-ubuntu
spark:4.0.0-preview2-scala2.13-java21-python3-r-ubuntu
spark:python3-java17
spark:scala
-spark:3.4.4
-spark:3.4.4-r
-spark:3.5.4-java17-r
-spark:3.5.4-scala2.12-java11-python3-r-ubuntu
+spark:3.5.5-java17-r
+spark:3.5.5-scala2.12-java11-python3-r-ubuntu
spark:4.0.0-preview2
spark:4.0.0-preview2-java21
spark:4.0.0-preview2-java21-r
diff --git a/spark_3.4.4/Dockerfile b/spark_3.4.4/Dockerfile
deleted file mode 100644
index d450ca5..0000000
diff --git a/spark_3.5.4-java17-r/Dockerfile b/spark_3.5.4-java17-r/Dockerfile
deleted file mode 100644
index 0d42b81..0000000
diff --git a/spark_3.5.4-java17-scala/Dockerfile b/spark_3.5.4-java17-scala/Dockerfile
deleted file mode 100644
index 4125654..0000000
diff --git a/spark_3.5.4-java17-scala/entrypoint.sh b/spark_3.5.4-java17-scala/entrypoint.sh
deleted file mode 100755
index c576d8f..0000000
diff --git a/spark_3.5.4-scala2.12-java17-python3-r-ubuntu/Dockerfile b/spark_3.5.4-scala2.12-java17-python3-r-ubuntu/Dockerfile
deleted file mode 100644
index 5113809..0000000
diff --git a/spark_3.4.4-r/Dockerfile b/spark_3.5.5-java17-r/Dockerfile
similarity index 93%
rename from spark_3.4.4-r/Dockerfile
rename to spark_3.5.5-java17-r/Dockerfile
index 1625536..1fdf9b9 100644
--- a/spark_3.4.4-r/Dockerfile
+++ b/spark_3.5.5-java17-r/Dockerfile
@@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-FROM spark:3.4.4-scala2.12-java11-ubuntu
+FROM spark:3.5.5-scala2.12-java17-ubuntu
USER root
@@ -23,6 +23,6 @@ RUN set -ex; \
apt-get install -y r-base r-base-dev; \
rm -rf /var/lib/apt/lists/*
-ENV R_HOME /usr/lib/R
+ENV R_HOME=/usr/lib/R
USER spark
diff --git a/spark_3.4.4-scala/Dockerfile b/spark_3.5.5-java17-scala/Dockerfile
similarity index 95%
rename from spark_3.4.4-scala/Dockerfile
rename to spark_3.5.5-java17-scala/Dockerfile
index 39cb49a..fc83654 100644
--- a/spark_3.4.4-scala/Dockerfile
+++ b/spark_3.5.5-java17-scala/Dockerfile
@@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-FROM eclipse-temurin:11-jre-focal
+FROM eclipse-temurin:17-jammy
ARG spark_uid=185
@@ -36,8 +36,8 @@ RUN set -ex; \
# Install Apache Spark
# https://downloads.apache.org/spark/KEYS
-ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.4.4/spark-3.4.4-bin-hadoop3.tgz \
- SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.4.4/spark-3.4.4-bin-hadoop3.tgz.asc \
+ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.5.5/spark-3.5.5-bin-hadoop3.tgz \
+ SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.5.5/spark-3.5.5-bin-hadoop3.tgz.asc \
GPG_KEY=F28C9C925C188C35E345614DEDA00CE834F0FC5C
RUN set -ex; \
@@ -72,7 +72,7 @@ RUN set -ex; \
COPY entrypoint.sh /opt/
-ENV SPARK_HOME /opt/spark
+ENV SPARK_HOME=/opt/spark
WORKDIR /opt/spark/work-dir
diff --git a/spark_3.4.4-scala/entrypoint.sh b/spark_3.5.5-java17-scala/entrypoint.sh
similarity index 100%
rename from spark_3.4.4-scala/entrypoint.sh
rename to spark_3.5.5-java17-scala/entrypoint.sh
diff --git a/spark_3.5.4-scala2.12-java11-python3-r-ubuntu/Dockerfile b/spark_3.5.5-scala2.12-java11-python3-r-ubuntu/Dockerfile
similarity index 93%
rename from spark_3.5.4-scala2.12-java11-python3-r-ubuntu/Dockerfile
rename to spark_3.5.5-scala2.12-java11-python3-r-ubuntu/Dockerfile
index 0778a40..b5127ed 100644
--- a/spark_3.5.4-scala2.12-java11-python3-r-ubuntu/Dockerfile
+++ b/spark_3.5.5-scala2.12-java11-python3-r-ubuntu/Dockerfile
@@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-FROM spark:3.5.4-scala2.12-java11-ubuntu
+FROM spark:3.5.5-scala2.12-java11-ubuntu
USER root
@@ -24,6 +24,6 @@ RUN set -ex; \
apt-get install -y r-base r-base-dev; \
rm -rf /var/lib/apt/lists/*
-ENV R_HOME /usr/lib/R
+ENV R_HOME=/usr/lib/R
USER spark
diff --git a/spark_3.4.4-scala2.12-java11-python3-r-ubuntu/Dockerfile b/spark_3.5.5-scala2.12-java17-python3-r-ubuntu/Dockerfile
similarity index 93%
rename from spark_3.4.4-scala2.12-java11-python3-r-ubuntu/Dockerfile
rename to spark_3.5.5-scala2.12-java17-python3-r-ubuntu/Dockerfile
index 9b0f38a..a900883 100644
--- a/spark_3.4.4-scala2.12-java11-python3-r-ubuntu/Dockerfile
+++ b/spark_3.5.5-scala2.12-java17-python3-r-ubuntu/Dockerfile
@@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-FROM spark:3.4.4-scala2.12-java11-ubuntu
+FROM spark:3.5.5-scala2.12-java17-ubuntu
USER root
@@ -24,6 +24,6 @@ RUN set -ex; \
apt-get install -y r-base r-base-dev; \
rm -rf /var/lib/apt/lists/*
-ENV R_HOME /usr/lib/R
+ENV R_HOME=/usr/lib/R
USER spark
diff --git a/spark_latest/Dockerfile b/spark_latest/Dockerfile
index ddb7c57..5c3033d 100644
--- a/spark_latest/Dockerfile
+++ b/spark_latest/Dockerfile
@@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-FROM spark:3.5.4-scala2.12-java11-ubuntu
+FROM spark:3.5.5-scala2.12-java11-ubuntu
USER root
diff --git a/spark_python3-java17/Dockerfile b/spark_python3-java17/Dockerfile
index 8a2cd5d..d9fb2f4 100644
--- a/spark_python3-java17/Dockerfile
+++ b/spark_python3-java17/Dockerfile
@@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-FROM spark:3.5.4-scala2.12-java17-ubuntu
+FROM spark:3.5.5-scala2.12-java17-ubuntu
USER root
diff --git a/spark_r/Dockerfile b/spark_r/Dockerfile
index c5da428..7beaf3f 100644
--- a/spark_r/Dockerfile
+++ b/spark_r/Dockerfile
@@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-FROM spark:3.5.4-scala2.12-java11-ubuntu
+FROM spark:3.5.5-scala2.12-java11-ubuntu
USER root
@@ -23,6 +23,6 @@ RUN set -ex; \
apt-get install -y r-base r-base-dev; \
rm -rf /var/lib/apt/lists/*
-ENV R_HOME /usr/lib/R
+ENV R_HOME=/usr/lib/R
USER spark
diff --git a/spark_scala/Dockerfile b/spark_scala/Dockerfile
index b5d6a4a..2adc6a6 100644
--- a/spark_scala/Dockerfile
+++ b/spark_scala/Dockerfile
@@ -36,9 +36,9 @@ RUN set -ex; \
# Install Apache Spark
# https://downloads.apache.org/spark/KEYS
-ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.5.4/spark-3.5.4-bin-hadoop3.tgz \
- SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.5.4/spark-3.5.4-bin-hadoop3.tgz.asc \
- GPG_KEY=19F745C40A0E550420BB2C522541488DA93FE4B4
+ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.5.5/spark-3.5.5-bin-hadoop3.tgz \
+ SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.5.5/spark-3.5.5-bin-hadoop3.tgz.asc \
+ GPG_KEY=F28C9C925C188C35E345614DEDA00CE834F0FC5C
RUN set -ex; \
export SPARK_TMP="$(mktemp -d)"; \
@@ -72,7 +72,7 @@ RUN set -ex; \
COPY entrypoint.sh /opt/
-ENV SPARK_HOME /opt/spark
+ENV SPARK_HOME=/opt/spark
WORKDIR /opt/spark/work-dirRelevant Maintainers:
|
yosifkit
approved these changes
Mar 6, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.