Skip to content

Latest commit

 

History

History
255 lines (177 loc) · 8.75 KB

developer_guide.md

File metadata and controls

255 lines (177 loc) · 8.75 KB

Developer Guide

Required Build Tools

For building and testing the driver you will need

To build Metabase itself you will need

On Ubuntu you can install the dependencies by running

sudo apt install nodejs yarnpkg

Fedora:

yum install perl-Digest-SHA nodejs yarnpkg

On macOS you additionally need gnu-sed:

brew install nodejs yarnpkg clojure gnu-sed
# Then add gnubin to your PATH:
# export PATH="/usr/local/opt/gnu-sed/libexec/gnubin:$PATH"

Run the following commands to check the current versions:

clojure -M --eval "(clojure-version)"
clojure --version

Setup Development Environment

  1. Checkout Metabase at $HOME/git/metabase (= $METABASE_DIR) and build it:

    cd $HOME/git
git clone https://github.com/metabase/metabase.git
cd metabase
git fetch --all --tags
export METABASE_VERSION=v0.48.0
git reset --hard
rm -vf target/patch_excluded_test_applied
git checkout "tags/${METABASE_VERSION}" -b "${METABASE_VERSION}-branch"
# Build (this will take ~15min)
./bin/build.sh

  2. Download the Exasol JDBC driver from the Download Portal and install it:

    cp exajdbc.jar "$METABASE_DIR/plugins"

  3. Checkout the Exasol Metabase driver at $HOME/git/metabase (= $METABASE_EXASOL_DRIVER)

    git clone https://github.com/exasol/metabase-driver.git
cd metabase-driver

Upgrading Metabase

To ensure compatibility we need to regularly update to the latest Metabase version. You can find the latest Metabase version on the GitHub release page.

Metabase publishes two variants:

  • OSS: version numbers v0.x.y
  • Enterprise: version numbers v1.x.y

We only use the OSS variant with version numbers v0.x.y.

To upgrade Metabase follow these steps:

  1. Check the Metabase driver changelog for breaking changes
  2. Replace the previous version in all files with the new version by searching for v0.x.y
  3. Run unit tests
  4. Run integration tests

The following things can go wrong:

  • The patch excluding inapproriate tests cannot be applied. See excluded tests for details.
  • Tests fail or abort
    • If possible fix the problem in the driver
    • If failures are related to Exasol specifics (e.g. missing TIME data type etc.) modify the test in Metabase and update the patch file, see excluded tests for details.
    • If failures are unrelated to Exasol or the driver, you might delete it and update the patch file, see excluded tests for details.

Running Driver Unit Tests

./scripts/run-unit-tests.sh

Start Metabase With the Exasol Driver

To start Metabase with the Exasol driver from source:

export METABASE_DIR="$HOME/git/metabase"
export METABASE_EXASOL_DRIVER="$HOME/git/metabase-driver"
cd $METABASE_DIR
clojure -Sdeps '{:deps {metabase/exasol-driver {:local/root "'"$METABASE_EXASOL_DRIVER"'"}}}' -J-Dmb.dev.additional.driver.manifest.paths=$METABASE_EXASOL_DRIVER/resources/metabase-plugin.yaml -M:run

After startup is complete (log message: Metabase Initialization COMPLETE) you can access Metabase at http://localhost:3000/.

Build and Install Exasol Driver

To start Metabase with the built Exasol driver:

export METABASE_DIR="$HOME/git/metabase"
export METABASE_EXASOL_DRIVER="$HOME/git/metabase-driver"
cd $METABASE_EXASOL_DRIVER

# Build driver
clojure -X:build :project-dir "\"$(pwd)\""

# Install driver
cp -v "$METABASE_EXASOL_DRIVER/target/exasol.metabase-driver.jar" "$METABASE_DIR/plugins/"

# Run Metabase
cd $METABASE_DIR
clojure -M:run

Running Integration Tests

You need to have metabase checked out next to this repository.

Start Exasol docker container:

docker run --publish 8563:8563 --publish 2580:2580 --publish 443:443 --detach --privileged --stop-timeout 120 exasol/docker-db:7.1.23

Start integration tests:

EXASOL_HOST=<hostname> EXASOL_PORT=8563 EXASOL_USER=sys EXASOL_PASSWORD=exasol ./scripts/run-integration-tests.sh

This script builds and installs the driver before running the integration tests. The driver must be installed to $METABASE_DIR/plugins/ for running the integration tests.

To run only a single tests or only tests in a namespace add arguments:

./scripts/run-integration-tests.sh :only name.space/single-test
./scripts/run-integration-tests.sh :only name.space

Using the REPL

export MB_EXASOL_TEST_HOST=<hostname>
export MB_EXASOL_TEST_PORT=8563
export MB_EXASOL_TEST_USER=sys
export MB_EXASOL_TEST_PASSWORD=exasol
export MB_EXASOL_TEST_CERTIFICATE_FINGERPRINT=<fingerprint>
export DRIVER=exasol

# Command line REPL:
clojure -A:dev:drivers:drivers-dev:test

# Network REPL:
clojure -M:dev:drivers:drivers-dev:test:nrepl # -> Connect with editor

In the REPL you can use the following commands:

(require 'dev) ; this will take some seconds
(dev/start!)

; Run SQL query:
(dev/query-jdbc-db :exasol "SELECT 1")
(dev/query-jdbc-db [:exasol 'test-data] "SELECT * from CAM_179.\"test_data_users\"")

Helpful Files in Metabase

Configure Logging

To increase the log level for integration tests, edit file $METABASE_DIR/test_config/log4j2-test.xml.

Excluded Tests

Exasol does not support the TIME data type. That is why we can't load the following datasets from the Metabase integration tests:

  • test-data-with-time
  • attempted-murders

We exclude TIME related and other broken tests by patching the metabase sources with patch scripts/exclude_tests.diff.

Script run-integration-tests.sh automatically applies this patch when file $METABASE_DIR/target/patch_excluded_test_applied does not exist.

When the patch file has changed or you updated to a new Metabase release, do the following and re-run the integration tests with run-integration-tests.sh.

cd $METABASE_DIR
git reset --hard && rm -vf target/patch_excluded_test_applied

Applying Patch Fails

If applying the patch fails after upgrading to a new Metabase version, follow these steps:

  1. Run cd $METABASE_DIR && git reset --hard && rm -vf target/patch_excluded_test_applied
  2. Remove the failed part from exclude_tests.diff
  3. Run integration tests run-integration-tests.sh. This will apply the patch.
  4. Modify Metabase tests to adapt them to Exasol
  5. Update patch by running cd $METABASE_DIR && git diff > $METABASE_EXASOL_DRIVER/scripts/exclude_tests.diff

Linting

clojure -M:clj-kondo --lint src test --debug

Troubleshooting

FileNotFoundException: Could not locate metabase/test/data/exasol__init.class, metabase/test/data/exasol.clj or metabase/test/data/exasol.cljc on classpath.

Verify that $METABASE_DIR/modules/drivers/exasol is a symlink to the metabase-driver directory.

Failing Integration Tests

Different Decimal Point

Tests fail on macOS because they expect numbers with a . as decimal point (e.g. 1000.0 µs) but get a , (e.g. 1000,0 µs), e.g.:

expected: (thrown-with-msg?
           clojure.lang.ExceptionInfo
           #"Timed out after 1000\.0 µs\."
           (if (instance? Throwable result) (throw result) result))
  actual: #<clojure.lang.ExceptionInfo@141971c7 clojure.lang.ExceptionInfo: Timed out after 1000,0 µs. {:status :timed-out, :type :timed-out}>

Solution: run tests under Linux with English locale or pass arguments -J-Duser.country=US -J-Duser.language=en to clojure when starting the tests (default in run-integration-tests.sh).

Time Dependent Tests

Some Metabase integration tests depend on the current timestamp and will fail when the year changes. See issue #14 for details.