Skip to content

Commit e03afc9

Browse files
LucaCanalidongjoon-hyun
LucaCanali
authored and
dongjoon-hyun
committed
[SPARK-36573][BUILD][TEST] Add a default value to ORACLE_DOCKER_IMAGE
### What changes were proposed in this pull request? Currently, the procedure to run the Oracle Integration Suite is based on building the Oracle RDBMS image from the Dockerfiles provided by Oracle. Recently, Oracle has started providing database images, see https://container-registry.oracle.com Moreover an Oracle employee is maintaining Oracle XE images that are streamlined for testing at https://hub.docker.com/r/gvenzl/oracle-xe and https://github.com/gvenzl/oci-oracle-xe This solves the issue that official images are quite large and make testing resource-intensive and slow. This proposes to document the available options and to introduce a default value for ORACLE_DOCKER_IMAGE ### Why are the changes needed? This change will make it easier and faster to run the Oracle Integration Suite, removing the need to manually build an Oracle DB image. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manually tested: ``` export ENABLE_DOCKER_INTEGRATION_TESTS=1 ./build/sbt -Pdocker-integration-tests "testOnly org.apache.spark.sql.jdbc.OracleIntegrationSuite" ./build/sbt -Pdocker-integration-tests "testOnly org.apache.spark.sql.jdbc.v2.OracleIntegrationSuite" ``` Closes #33821 from LucaCanali/oracleDockerIntegration. Authored-by: Luca Canali <luca.canali@cern.ch> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 5b4c216 commit e03afc9

File tree

3 files changed

+45
-55
lines changed

3 files changed

+45
-55
lines changed

‎.github/workflows/build_and_test.yml

+1-19
Original file line numberDiff line numberDiff line change
@@ -686,7 +686,7 @@ jobs:
686686
HIVE_PROFILE: hive2.3
687687
GITHUB_PREV_SHA: ${{ github.event.before }}
688688
SPARK_LOCAL_IP: localhost
689-
ORACLE_DOCKER_IMAGE_NAME: oracle/database:18.4.0-xe
689+
ORACLE_DOCKER_IMAGE_NAME: gvenzl/oracle-xe:18.4.0
690690
SKIP_MIMA: true
691691
steps:
692692
- name: Checkout Spark repository
@@ -724,24 +724,6 @@ jobs:
724724
uses: actions/setup-java@v1
725725
with:
726726
java-version: 8
727-
- name: Cache Oracle docker-images repository
728-
id: cache-oracle-docker-images
729-
uses: actions/cache@v2
730-
with:
731-
path: ./oracle/docker-images
732-
# key should contains the commit hash of the Oracle docker images to be checkout.
733-
key: oracle-docker-images-3f422c4a35b423dfcdbcc57a84f01db6c82eb6c1
734-
- name: Checkout Oracle docker-images repository
735-
uses: actions/checkout@v2
736-
with:
737-
fetch-depth: 0
738-
repository: oracle/docker-images
739-
ref: 3f422c4a35b423dfcdbcc57a84f01db6c82eb6c1
740-
path: ./oracle/docker-images
741-
- name: Install Oracle Docker image
742-
run: |
743-
cd oracle/docker-images/OracleDatabase/SingleInstance/dockerfiles
744-
./buildContainerImage.sh -v 18.4.0 -x
745727
- name: Run tests
746728
run: |
747729
./dev/run-tests --parallelism 1 --modules docker-integration-tests --included-tags org.apache.spark.tags.DockerTest

‎external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleIntegrationSuite.scala

+22-18
Original file line numberDiff line numberDiff line change
@@ -34,42 +34,46 @@ import org.apache.spark.sql.types._
3434
import org.apache.spark.tags.DockerTest
3535

3636
/**
37-
* The following would be the steps to test this
38-
* 1. Build Oracle database in Docker, please refer below link about how to.
39-
* https://github.com/oracle/docker-images/blob/master/OracleDatabase/SingleInstance/README.md
40-
* 2. export ORACLE_DOCKER_IMAGE_NAME=$ORACLE_DOCKER_IMAGE_NAME
41-
* Pull oracle $ORACLE_DOCKER_IMAGE_NAME image - docker pull $ORACLE_DOCKER_IMAGE_NAME
42-
* 3. Start docker - sudo service docker start
43-
* 4. Run spark test - ./build/sbt -Pdocker-integration-tests
44-
* "testOnly org.apache.spark.sql.jdbc.OracleIntegrationSuite"
37+
* The following are the steps to test this:
4538
*
46-
* An actual sequence of commands to run the test is as follows
39+
* 1. Choose to use a prebuilt image or build Oracle database in a container
40+
* - The documentation on how to build Oracle RDBMS in a container is at
41+
* https://github.com/oracle/docker-images/blob/master/OracleDatabase/SingleInstance/README.md
42+
* - Official Oracle container images can be found at https://container-registry.oracle.com
43+
* - A trustable and streamlined Oracle XE database image can be found on Docker Hub at
44+
* https://hub.docker.com/r/gvenzl/oracle-xe see also https://github.com/gvenzl/oci-oracle-xe
45+
* 2. Run: export ORACLE_DOCKER_IMAGE_NAME=image_you_want_to_use_for_testing
46+
* - Example: export ORACLE_DOCKER_IMAGE_NAME=gvenzl/oracle-xe:latest
47+
* 3. Run: export ENABLE_DOCKER_INTEGRATION_TESTS=1
48+
* 4. Start docker: sudo service docker start
49+
* - Optionally, docker pull $ORACLE_DOCKER_IMAGE_NAME
50+
* 5. Run Spark integration tests for Oracle with: ./build/sbt -Pdocker-integration-tests
51+
* "testOnly org.apache.spark.sql.jdbc.OracleIntegrationSuite"
4752
*
53+
* A sequence of commands to build the Oracle XE database container image:
4854
* $ git clone https://github.com/oracle/docker-images.git
49-
* // Head SHA: 3f422c4a35b423dfcdbcc57a84f01db6c82eb6c1
5055
* $ cd docker-images/OracleDatabase/SingleInstance/dockerfiles
5156
* $ ./buildContainerImage.sh -v 18.4.0 -x
5257
* $ export ORACLE_DOCKER_IMAGE_NAME=oracle/database:18.4.0-xe
53-
* $ export ENABLE_DOCKER_INTEGRATION_TESTS=1
54-
* $ cd $SPARK_HOME
55-
* $ ./build/sbt -Pdocker-integration-tests
56-
* "testOnly org.apache.spark.sql.jdbc.OracleIntegrationSuite"
5758
*
58-
* It has been validated with 18.4.0 Express Edition.
59+
* This procedure has been validated with Oracle 18.4.0 Express Edition.
5960
*/
6061
@DockerTest
6162
class OracleIntegrationSuite extends DockerJDBCIntegrationSuite with SharedSparkSession {
6263
import testImplicits._
6364

6465
override val db = new DatabaseOnDocker {
65-
lazy override val imageName = sys.env("ORACLE_DOCKER_IMAGE_NAME")
66+
lazy override val imageName =
67+
sys.env.getOrElse("ORACLE_DOCKER_IMAGE_NAME", "gvenzl/oracle-xe:18.4.0")
68+
val oracle_password = "Th1s1sThe0racle#Pass"
6669
override val env = Map(
67-
"ORACLE_PWD" -> "oracle"
70+
"ORACLE_PWD" -> oracle_password, // oracle images uses this
71+
"ORACLE_PASSWORD" -> oracle_password // gvenzl/oracle-xe uses this
6872
)
6973
override val usesIpc = false
7074
override val jdbcPort: Int = 1521
7175
override def getJdbcUrl(ip: String, port: Int): String =
72-
s"jdbc:oracle:thin:system/oracle@//$ip:$port/xe"
76+
s"jdbc:oracle:thin:system/$oracle_password@//$ip:$port/xe"
7377
}
7478

7579
override val connectionTimeout = timeout(7.minutes)

‎external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/OracleIntegrationSuite.scala

+22-18
Original file line numberDiff line numberDiff line change
@@ -29,41 +29,45 @@ import org.apache.spark.sql.types._
2929
import org.apache.spark.tags.DockerTest
3030

3131
/**
32-
* The following would be the steps to test this
33-
* 1. Build Oracle database in Docker, please refer below link about how to.
34-
* https://github.com/oracle/docker-images/blob/master/OracleDatabase/SingleInstance/README.md
35-
* 2. export ORACLE_DOCKER_IMAGE_NAME=$ORACLE_DOCKER_IMAGE_NAME
36-
* Pull oracle $ORACLE_DOCKER_IMAGE_NAME image - docker pull $ORACLE_DOCKER_IMAGE_NAME
37-
* 3. Start docker - sudo service docker start
38-
* 4. Run spark test - ./build/sbt -Pdocker-integration-tests
39-
* "testOnly org.apache.spark.sql.jdbc.v2.OracleIntegrationSuite"
32+
* The following are the steps to test this:
4033
*
41-
* An actual sequence of commands to run the test is as follows
34+
* 1. Choose to use a prebuilt image or build Oracle database in a container
35+
* - The documentation on how to build Oracle RDBMS in a container is at
36+
* https://github.com/oracle/docker-images/blob/master/OracleDatabase/SingleInstance/README.md
37+
* - Official Oracle container images can be found at https://container-registry.oracle.com
38+
* - A trustable and streamlined Oracle XE database image can be found on Docker Hub at
39+
* https://hub.docker.com/r/gvenzl/oracle-xe see also https://github.com/gvenzl/oci-oracle-xe
40+
* 2. Run: export ORACLE_DOCKER_IMAGE_NAME=image_you_want_to_use_for_testing
41+
* - Example: export ORACLE_DOCKER_IMAGE_NAME=gvenzl/oracle-xe:latest
42+
* 3. Run: export ENABLE_DOCKER_INTEGRATION_TESTS=1
43+
* 4. Start docker: sudo service docker start
44+
* - Optionally, docker pull $ORACLE_DOCKER_IMAGE_NAME
45+
* 5. Run Spark integration tests for Oracle with: ./build/sbt -Pdocker-integration-tests
46+
* "testOnly org.apache.spark.sql.jdbc.v2.OracleIntegrationSuite"
4247
*
48+
* A sequence of commands to build the Oracle XE database container image:
4349
* $ git clone https://github.com/oracle/docker-images.git
44-
* // Head SHA: 3f422c4a35b423dfcdbcc57a84f01db6c82eb6c1
4550
* $ cd docker-images/OracleDatabase/SingleInstance/dockerfiles
4651
* $ ./buildContainerImage.sh -v 18.4.0 -x
4752
* $ export ORACLE_DOCKER_IMAGE_NAME=oracle/database:18.4.0-xe
48-
* $ export ENABLE_DOCKER_INTEGRATION_TESTS=1
49-
* $ cd $SPARK_HOME
50-
* $ ./build/sbt -Pdocker-integration-tests
51-
* "testOnly org.apache.spark.sql.jdbc.v2.OracleIntegrationSuite"
5253
*
53-
* It has been validated with 18.4.0 Express Edition.
54+
* This procedure has been validated with Oracle 18.4.0 Express Edition.
5455
*/
5556
@DockerTest
5657
class OracleIntegrationSuite extends DockerJDBCIntegrationSuite with V2JDBCTest {
5758
override val catalogName: String = "oracle"
5859
override val db = new DatabaseOnDocker {
59-
lazy override val imageName = sys.env("ORACLE_DOCKER_IMAGE_NAME")
60+
lazy override val imageName =
61+
sys.env.getOrElse("ORACLE_DOCKER_IMAGE_NAME", "gvenzl/oracle-xe:18.4.0")
62+
val oracle_password = "Th1s1sThe0racle#Pass"
6063
override val env = Map(
61-
"ORACLE_PWD" -> "oracle"
64+
"ORACLE_PWD" -> oracle_password, // oracle images uses this
65+
"ORACLE_PASSWORD" -> oracle_password // gvenzl/oracle-xe uses this
6266
)
6367
override val usesIpc = false
6468
override val jdbcPort: Int = 1521
6569
override def getJdbcUrl(ip: String, port: Int): String =
66-
s"jdbc:oracle:thin:system/oracle@//$ip:$port/xe"
70+
s"jdbc:oracle:thin:system/$oracle_password@//$ip:$port/xe"
6771
}
6872

6973
override def sparkConf: SparkConf = super.sparkConf

0 commit comments

Comments
 (0)