Skip to content

Latest commit

 

History

History
335 lines (262 loc) · 9.42 KB

INSTALL.org

File metadata and controls

335 lines (262 loc) · 9.42 KB

Installation Instructions

We assume the framework is installed in a recent Linux-based system (Debian, for instance). Since most of the framework is written in R, we expect a modern version is already installed in the system. The last version for which we have tested, is as follows.

sessionInfo()
R version 3.4.2 (2017-09-28)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux buster/sid

Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

loaded via a namespace (and not attached):
[1] compiler_3.4.2

Installing only the R package

To install only the R package, the devtools package is required. It can be installed with:

install.packages("devtools")

Then, the StarVZ package can be installed via github with:

library(devtools)
devtools::install_github("schnorr/starvz")

Also, it is possible to install the package with the local files:

library(devtools)
devtools::install_local(path="/starvz_git_path/")

Now all StarVZ R functions can be accessible with:

library(starvz)

Non-R Dependencies

You need to install these tools:

Below, we provide customized installations tips for the non-R dependencies above. Please, make sure they are still updated against the official instructions available for each tool.

StarPU, fxt, and poti

  1. Install the latest version of FXT 
    wget http://download.savannah.nongnu.org/releases/fkt/fxt-0.3.7.tar.gz
tar xfz fxt-0.3.7.tar.gz; cd fxt-0.3.7/;
./configure --prefix=$HOME/install/fxt-0.3.7/
make install
  2. Install poti 
    git clone http://github.com/schnorr/poti
mkdir -p poti/build; cd poti/build
cmake -DCMAKE_INSTALL_PREFIX=$HOME/install/poti-git/ ..; make ; make install
  3. Install StarPU from the SVN 
    svn checkout svn+ssh://scm.gforge.inria.fr/svnroot/starpu/trunk starpu-trunk
mkdir -p starpu-trunk/build; cd starpu-trunk/; ./autogen.sh; cd build
PKG_CONFIG_PATH=$HOME/install/poti-git/lib/pkgconfig/ \
 ../configure \
   --with-fxt=$HOME/install/fxt-0.3.7/ \
   --enable-debug \
   --disable-build-examples \
   --disable-build-tests \
   --enable-verbose \
   --disable-silent-rules \
   --enable-poti \
   --disable-hdf5 \
   --prefix=$HOME/install/starpu-trunk/
make -j 4
make install

PajeNG

One step does everything.

git clone http://github.com/schnorr/pajeng
mkdir -p pajeng/build ; cd pajeng/build
cmake -DCMAKE_INSTALL_PREFIX=$HOME/install/pajeng-git/ ..; make -j 4 ; make install

Pmtool

There is a required tool called starpu_perfmodel_recdump whose source code is available with pmtool. To be compiled, it requires to be included in the StarPU tools directory and be compiled within a StarPU-simgrid configuration. So, the detailed steps below reflect this necessity:

  1. Install Simgrid 
    git clone https://github.com/simgrid/simgrid simgrid-git
mkdir simgrid-git/build ; cd simgrid-git/build
cmake -Denable_documentation=OFF -DCMAKE_INSTALL_PREFIX=$HOME/install/simgrid-git/ ..; make -j 4 ; make install
  2. Clone pmtool to get the auxiliary tool 
    git clone http://gitlab.inria.fr/eyrauddu/pmtool
  3. Checkout starpu and copy the auxiliary tool 
    svn checkout svn+ssh://scm.gforge.inria.fr/svnroot/starpu/trunk starpu-trunk-simgrid
cp pmtool/starpu_export/starpu_perfmodel_recdump.c ./starpu-trunk-simgrid/tools/
echo "STARPU_TOOLS += starpu_perfmodel_recdump" >> ./starpu-trunk-simgrid/tools/Makefile.am
echo "bin_PROGRAMS += starpu_perfmodel_recdump" >> ./starpu-trunk-simgrid/tools/Makefile.am
  4. Configure StarPU with SimGrid 
    mkdir -p starpu-trunk-simgrid/build; cd starpu-trunk-simgrid/; ./autogen.sh; cd build
PKG_CONFIG_PATH=$HOME/install/poti-git/lib/pkgconfig/  \
   ../configure \
      --with-simgrid-dir=$HOME/install/simgrid-git/ \
      --enable-simgrid \
      --enable-debug \
      --disable-build-examples \
      --disable-build-tests \
      --enable-verbose \
      --disable-silent-rules \
      --enable-poti \
      --disable-hdf5 \
      --prefix=$HOME/install/starpu-trunk-simgrid/
make -j 4
make
  5. Install CPLEX to have richer pmtool results

    Download the appropriate version from here.

    Make sure you install CPLEX at $HOME/install/cplex-12.7.0/.

    ./cplex_studio127.linux-x86-64.bin
  6. Compile pmtool (already cloned in Step 2) 
    cd pmtool
export CPLEX_DIR=$HOME/install/cplex-12.7.0/

Uninstall R package

remove.packages("starvz")

Development Tips

If you are involved in the development, this might be of your interest:

Reinstall from directory

library(devtools)
devtools::install_local(path="/starvz_git_path/", force=TRUE)

Reload after reinstall

detach("package:starvz", unload=TRUE)
library(starvz)

Arrow installation on Debian

StarVZ now depends on the arrow R package to work with parquet files, so before installing starvz, install arrow with parquet support.

While the arrow installation is straightforward in many systems (pre-compiled binaries), it may be hard to obtain an installation with compression support in Debian systems. See this link for more information. A possible fix is do the following (install everything by hand following a combination of these instructions). See below for details.

Install the Apache Arrow C++

You may need to install bz2, lz4, snappy, zlib, and zstd libraries.

git clone https://github.com/apache/arrow
cd arrow/cpp
mkdir build
cd build
cmake -DARROW_CSV=ON \
      -DARROW_JSON=ON \
      -DARROW_PARQUET=ON \
      -DARROW_DATASET=ON \
      -DARROW_WITH_BZ2=ON \
      -DARROW_WITH_LZ4=ON \
      -DARROW_WITH_SNAPPY=ON \
      -DARROW_WITH_ZLIB=ON \
      -DARROW_WITH_ZSTD=ON \
      ..
make
make install

Install the R API

In the same cloned repository.

export ARROW_USE_PKG_CONFIG=true
export LIBARROW_BINARY=false
export LIBARROW_MINIMAL=false
export PKG_CONFIG_PATH=/home/schnorr/install/arrow/lib/pkgconfig/
cd arrow/r
make clean
R CMD INSTALL .

Verify that everything went smoothly by issuing these commands in your R session:

library(arrow)
arrow_available()
arrow::CompressionType

Confirm the output is something like this:

> arrow_available()
[1] TRUE

> arrow::CompressionType
$UNCOMPRESSED
[1] 0

$SNAPPY
[1] 1

$GZIP
[1] 2

$BROTLI
[1] 3

$ZSTD
[1] 4

$LZ4
[1] 5

$LZ4_FRAME
[1] 6

$LZO
[1] 7

$BZ2
[1] 8

It may be required to set the LD_LIBRARY_PATH to point to your arrow libs. If you put that in your $HOME/.bashrc or similar, it may be easy to live with.