Merge pull request #16 from cirlabs/create-virtual-machine

Create virtual machine

HOWTO.md

@@ -0,0 +1,60 @@
+How-to
+------
+This guide will show you how to create a virtualbox virtual machine from scratch, install Xubuntu on it and bootstrap it with all the latest and greatest software.
+
+## Setup
+- Download and install VirtualBox and the Extensions
+    - https://www.virtualbox.org/wiki/Downloads
+
+- Download the Xubuntu 14.04 .iso
+    - http://xubuntu.org/getxubuntu/
+    - Choose 32-bit as some folks may have old machines
+
+- Follow VirtualBox instructions for installing virtualbox on "Creating your first virtual machine" Section 1.7
+    - http://www.virtualbox.org/manual/ch01.html
+    - When asked about which virtual image type to use, choose VMDK
+        - http://www.virtualbox.org/manual/ch05.html#vdidetails
+    - Set storage to whatever you'd like. I recommend 30.00 GB to 2.00 T DYNAMICALLY ALLOCATED !important
+
+    - Set the RAM (We suggest 2048 MB or 2GB)
+
+
+- Start the virtual machine and navigate your file system to the path of the iso
+
+- Follow the Xubuntu prompts to install the operating system on the file system
+    - set the username and password
+    - For techraking: `nicar` with the password `nicar`
+
+- Setup Guest Additions
+    - Guest Additions will give you full resolution and filesystem sharing, which are quite nice in VMs
+    - Log into the Xubuntu VM, open up the terminal and run this command:
+        - `sudo apt-get install virtualbox-guest-dkms virtualbox-guest-utils virtualbox-guest-x11`
+    - Restart the VM to have a theorically native resolution supported virtual machine
+
+## Bootstrap the box
+- Fetch the bootstrap script by downloading it with `wget`. Open up terminal and type:
+    - `$ wget https://raw.githubusercontent.com/cirlabs/vm/master/bootstrap.sh`
+- make the script executable
+    - `$ chmod +x bootstrap.sh`
+- run the script WITHOUT a subprocess
+    - `$ . ./bootstrap.sh`
+        - See: http://stackoverflow.com/a/16011496/868724
+    - NOTE: This'll take some time. The first commands to run are `apt-get update` and `apt-get upgrade`. These two commands will make sure the the operating systems has the latest and greatest security patches and features. These are key before installing the rest of the software.
+
+## Extra changes
+Here are some last configuration changes to do after `bootstrap.sh` finishes executing.
+
+- Configure PostgreSQL
+
+```bash
+$ sudo su - postgres
+$ "CREATE USER nicar SUPERUSER;" | psql -d postgres
+$ exit
+```
+
+## Export
+1. Open VirtualBox
+2. Navigate to *File > Export Appliance*
+3. Select the Xubuntu virtual machine
+4. Choose OVF 1.0 (2.0 is for cloud computing compatibility. While the latest standard, it's still buggy in other virtualization software like VMware. See: https://www.virtualbox.org/ticket/11160)
+5. Press *Export*

INSTALL.md

@@ -0,0 +1,19 @@
+Manual installation instructions
+---------------------------------
+A step-by-step walkthrough for manually installing the VirtualBox and CIR's data journalism VM. Only go about it this way if you're on Windows or if you don't trust the bash script in README. Note: Manual
+
+## Setup VirtualBox
+
+1. Download VirtualBox for your operating system: https://www.virtualbox.org/wiki/Downloads
+2. Double click on the file (.exe for windows, .dmg for Mac OS X) and use the setup wizard to install VirtualBox.
+3. Download the latest VirtualBox extension pack (same file for all operating systems) [http://download.virtualbox.org/virtualbox/4.3.20/Oracle_VM_VirtualBox_Extension_Pack-4.3.20-96996.vbox-extpack]
+4. Double click on the .vbox-extpack file to install the extensions
+
+If you get lost at all during this process, refer to the VirtualBox installation manual: https://www.virtualbox.org/manual/ch01.html#intro-installing
+
+## Download and install virtual machine
+
+1. Once VirtualBox is installed, download the virtual machine. It's a 3GB file so make sure you have space on your hard drive: https://s3-us-west-1.amazonaws.com/vms/nicar-pre-k-2015.ova
+2. Doubleclick file to install the virtual machine | *If you get stuck*: https://www.virtualbox.org/manual/ch01.html#ovf
+3. During installation, click the __Reinitialize MAC Address__ checkbox
+4. Start the virtual machine | *If you get stuck*: https://www.virtualbox.org/manual/ch01.html#idp51818064

PACKAGE

@@ -3,9 +3,9 @@ User: nicar
 Password: nicar
 
 # VirtualBox Image Notes
-- OS: Xubuntu Linux 13.10
-- RAM: 1GB
-- HDD: 30GB Total Capacity / (7.42 GB when first installed) Dynamically allocated
+- OS: Xubuntu Linux 14.04
+- RAM: 2GB
+- HDD: 2 TB Total Capacity / (7.42 GB when first installed) Dynamically allocated
 
 # installed command line programs
 csvkit
@@ -26,7 +26,7 @@ Node.js
 NPM
 Ruby
 Rails
-RVM
+Rbenv
 Bower
 Grunt
 Fabric

README.md

@@ -1,15 +1,16 @@
 vm
-==
-A Linux-powered data journalism virtual machine image for VirtualBox. Meant for beginners and/or people interested in learning the data journalism stack. Direct link to the image: [Data Journalism VM .ova](https://s3-us-west-1.amazonaws.com/vms/nicar-pre-k-2014.ova) (3GB)
+--
+A Linux-powered data journalism virtual machine image for VirtualBox. Meant for beginners and/or people interested in learning the data journalism stack. Direct link to the image: [Data Journalism VM .ova](https://s3-us-west-1.amazonaws.com/vms/nicar-pre-k-2015.ova) (3GB)
 
 ## Quickstart
 With VirtualBox installed, kick open Terminal, paste the below command and press enter:
+
 ```bash
 $ wget https://raw.githubusercontent.com/cirlabs/vm/master/install-vm.sh && bash install-vm.sh
 ```
 
 ## Features
-The [Xubuntu 13.10](http://xubuntu.org/) operating system and a bunch of libraries comingly used by data journalist. This includes (but is not limited to):
+The [Xubuntu 14.04](http://xubuntu.org/) operating system and a bunch of libraries comingly used by data journalist like:
 - IPython
 - Django
 - SQLite, MySQL, PostgreSQL/PostGIS
@@ -27,10 +28,10 @@ See [PACKAGE](https://github.com/cirlabs/vm/blob/master/PACKAGE) for full list
 ## Installing the Virtual Machine
 
 ### Mac OS X / Linux
-1. Download the script: [VM-1.0.zip](https://github.com/cirlabs/vm/archive/1.0.zip)
+1. Download the script: [VM-2.0.zip](https://github.com/cirlabs/vm/archive/2.0.zip)
 2. Extract the .zip file. You should see a file called `install-vm.sh` inside the folder.
 3. Open Terminal. For Mac OS X, go to `Applications/Utilities/Terminal.app`. For Linux, press __CTRL+ALT+T__ to open it.
-4. In Terminal, type `bash` and the path to `install-vm.sh`. The easiest way to do this is to type `bash` and then *drag-and-drop* `install-vm.sh` into the terminal prompt. You should have something like __`bash /Users/username/Downloads/VM-1.0/install-vm-sh`__.
+4. In Terminal, type `bash` and the path to `install-vm.sh`. The easiest way to do this is to type `bash` and then *drag-and-drop* `install-vm.sh` into the terminal prompt. You should have something like __`bash /Users/username/Downloads/VM-2.0/install-vm-sh`__.
 5. Press [ENTER] and follow the instructions
 
 
@@ -42,17 +43,9 @@ See [PACKAGE](https://github.com/cirlabs/vm/blob/master/PACKAGE) for full list
 4. *(Optional)* Prompts user to select path to install virtual machine, e.g., `/Volumes/my-external-harddive/vms/`
 5. Starts the virtual machine
 
-### Windows
-Windows isn't UNIX based so this script won't work. You can install [Cygwin](http://www.cygwin.com/) and try it there but I have not tested it. You're probably better off downloading the VirtualBox image directly from our AWS S3 bucket and importing it that way. 
-
-Before you download and install the VM, make sure you have the VirtualBox extension pack installed for your version of VirtualBox (currently 4.3.10).
+### Windows users
+Follow the instructions in [INSTALL](https://github.com/cirlabs/vm/blob/master/INSTALL.md)
 
-Do the following:
-
-1. Download and install [VirtualBox Extension Pack 4.3.10](http://download.virtualbox.org/virtualbox/4.3.10/Oracle_VM_VirtualBox_Extension_Pack-4.3.10-93012.vbox-extpack) (10.4 MB) | [Instructions](https://www.virtualbox.org/manual/ch01.html#intro-installing)
-2. Download and install  the [Data Journalism VM file](https://s3-us-west-1.amazonaws.com/vms/nicar-pre-k-2014.ova) (3GB) | [Instructions](https://www.virtualbox.org/manual/ch01.html#ovf)
-3. During installation, click the __Reinitialize MAC Address__ checkbox
-4. Start the virtual machine | [Instructions](https://www.virtualbox.org/manual/ch01.html#idp51818064)
 
 ## Roll your own
 Not enterested in using VirtualBox but want to bootstrap a Linux box for data journalism? Checkout [bootstrap.sh](https://github.com/cirlabs/vm/blob/master/bootstrap.sh).
@@ -62,6 +55,7 @@ The 3 GB .ova file should be saved to `~/.vms`. You can remove it, as well as an
 
 ## Limitations / Notes
 - The VM will have a 800x600 screen resolution on startup. There are ways to make your VM fit the size of your host display but that goes on beyond the scope of this project. See [VirtualBox Manual 1.8.5. Resizing the machine's window](https://www.virtualbox.org/manual/ch01.html#intro-resize-window) for more details. Do note that installing VirtualBox Guest Additions only works for Windows and Linux users. Sorry Mac folks.
+
 - This VM is meant as a place to practice and learn about data journalism and software development. It's running a lightweight Linux distribtion and uses just enough RAM to not be a headache. While it is my hope you use this VM to learn and create some awesome projects, my greater hope would be that you abandon the VM altogether and configure your laptop directly with either Linux or whatever OS you prefer. Again, more than anything, this is more of a teaching tool than a fully supported out-the-box rig. That could change in the future, but that is the focus of this project as of now.
 
 ## Help

bootstrap.sh

@@ -1,29 +1,34 @@
 #!/bin/sh
-# set the user
-THISUSER=whoami
 
-echo "updating ubuntu"
+# Pretty print messages
+print_header() {
+    echo -e "\n================"
+    echo -e $1
+    echo -e "================\n"
+}
+
+print_header "updating ubuntu"
 sudo apt-get -qq update
 sudo apt-get -qq upgrade
 
 # development
-echo "installing development tools"
-sudo apt-get -qq install build-essential fortune cowsay # hehehe
+print_header "installing development tools"
+sudo apt-get -qq install build-essential fortune cowsay
 
 # Git
-echo "installing git"
+print_header "installing git"
 sudo apt-get -qq install git-core
 
 # zsh
-echo "installing zsh and oh-my-zsh"
+print_header "installing zsh and oh-my-zsh"
 wget --no-check-certificate https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/tools/install.sh -O - | sh
 
 # python scientific stack
-echo "installing python scientific stack"
+print_header "installing python scientific stack"
 sudo apt-get -qq install python-numpy python-scipy python-matplotlib ipython ipython-doc ipython-notebook ipython-qtconsole python-virtualenv python-dev python-pip python-sip pyqt4-dev-tools
 
 # various Python libraries we like
-echo "pip installing favored Python libraries"
+print_header "pip installing favored Python libraries"
 sudo pip install --quiet beautifulsoup4
 sudo pip install --quiet requests
 sudo pip install --quiet django
@@ -32,7 +37,7 @@ sudo pip install --quiet pandas
 sudo pip install --quiet csvkit
 
 # make sure virtalenvwrapper is loaded and works
-echo "  setting up virtualenvwrapper"
+print_header "  setting up virtualenvwrapper"
 mkdir .envs
 echo "export WORKON_HOME=$HOME/.envs" >> ~/.bashrc
 echo "source /usr/local/bin/virtualenvwrapper.sh" >> ~/.bashrc
@@ -41,36 +46,32 @@ echo "export WORKON_HOME=$HOME/.envs" >> ~/.zshrc
 echo "source /usr/local/bin/virtualenvwrapper.sh" >> ~/.zshrc
 
 # postgres
-echo "installing latest PostgreSQL and PostGIS"
+print_header "installing latest PostgreSQL and PostGIS"
 sudo apt-get install -qq postgresql
 sudo apt-get install -qq postgis
 
-# create superuser for for self
-echo "  setting up PostgreSQL superuser"
-sudo su - postgres
-"CREATE USER nicar SUPERUSER;" | psql -d postgres
-exit
+echo "  IMPORTANT: Remember to create a Postgres superuser for your user!"
 
 # MySQL
-echo "Installing MySQL"
+print_header "Installing MySQL"
 sudo apt-get -qq install mysql-server mysql-client libmysqlclient-dev
 
 #qgis
-echo "installing QGIS"
+print_header "installing QGIS"
 sudo apt-get install qgis
 
 # node.js
-echo "installing Node.js"
+print_header "installing Node.js"
 sudo apt-get -qq install nodejs
 sudo ln -s /usr/bin/nodejs /usr/bin/node
 
 
 #install Java
-echo "installing Java"
+print_header "installing Java"
 sudo apt-get -qq install default-jre
 
 # Install Ruby
-echo "install Ruby, rbenv and ruby-build"
+print_header "installing Ruby, rbenv and ruby-build"
 
 #install dependencies
 sudo apt-get install -qq autoconf bison libssl-dev libyaml-dev libreadline6-dev zlib1g-dev libncurses5-dev libffi-dev libgdbm3 libgdbm-dev
@@ -95,10 +96,10 @@ rbenv global 2.2.0
 
 
 #install jruby
-echo "installing jruby for Tabula"
+print_header "installing jruby for Tabula"
 rbenv install jruby-1.7.18
 #install Tabula extractor for awesome command line pdf extraction
-echo "Setting up Tabula"
+echo " Setting up Tabula"
 mkdir tabula
 cd tabula
 rbenv local jruby-1.7.18

install-vm.sh

@@ -12,6 +12,9 @@ set -o errexit
 set -o pipefail
 
 # ## Utilities
+# Store vm filename and display name
+remote_vm_filename="nicar-pre-k-2015.ova"
+remote_vm_display_name="NICAR" # set when the VM is created in VirtualBox
 
 # Store the original `cwd`.
 orig_cwd=`pwd`
@@ -111,18 +114,18 @@ check_ext_pack() {
 
 download_vm() {
     log "Checking for NICAR VM image"
-    if [ ! -f "$vms_home/nicar-pre-k-2014.ova" ]; then
+    if [ ! -f "$vms_home/${remote_vm_filename}" ]; then
         ## fetch files
         log "Fetching Virtual Machine"
-        wget https://s3-us-west-1.amazonaws.com/vms/nicar-pre-k-2014.ova
+        wget https://s3-us-west-1.amazonaws.com/vms/${remote_vm_filename}
 
-        log "saved to $vms_home/nicar-pre-k-2014"
+        log "saved to $vms_home/${remote_vm_filename}"
 
         ## make sure the files match
-        # checksum="434db66814214674877454edabe04551  nicar-pre-k-2014.ova"
+        # checksum="434db66814214674877454edabe04551 ${remote_vm_filename}"
         # echo -e "checking if the file matches the original checksum of " + $checksum + "\n================="
         # get new checksum
-        # newchecksum=$(md5sum nicar-pre-k-2014.ova)
+        # newchecksum=$(md5sum ${remote_vm_filename})
 
         # if [[ $newchecksum -eq $checksum ]]; then
         #     echo -e "Checksum matches. Continuing import!\n================="
@@ -142,21 +145,21 @@ import_vm() {
         log "Importing VM"
         # set the user path, otherwise, import normally
         if ! $user_path; then
-            VBoxManage import "$vms_home/nicar-pre-k-2014.ova" --vsys 0 --unit 11 --disk "$user_path/$vm_name/nicar-pre-k-2014.vmdk"
+            VBoxManage import "$vms_home/${remote_vm_filename}" --vsys 0 --unit 11 --disk "$user_path/$vm_name/${remote_vm_filename}"
         else
-            VBoxManage import "$vms_home/nicar-pre-k-2014.ova"
+            VBoxManage import "$vms_home/${remote_vm_filename}"
         fi
 
         # reinitialize the mac address
-        uuid=$(VBoxManage list vms | grep Nicar | grep -Eo '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}')
+        uuid=$(VBoxManage list vms | grep ${remote_vm_display_name} | grep -Eo '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}')
         log "modifying the MAC address"
         VBoxManage modifyvm $uuid --macaddress1 auto
     fi
 }
 
 start_vm() {
     ## start the VM
-    uuid=$(VBoxManage list vms | grep Nicar | grep -Eo '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}')
+    uuid=$(VBoxManage list vms | grep ${remote_vm_display_name} | grep -Eo '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}')
     log "Starting Virtual Machine"
     VBoxManage startvm $uuid
 }
@@ -169,4 +172,4 @@ check_version
 check_ext_pack
 download_vm
 import_vm
-start_vm
+start_vm