diff --git a/.gitignore b/.gitignore index e669e4e..f9c229e 100644 --- a/.gitignore +++ b/.gitignore @@ -5,6 +5,7 @@ **/.DS_Store .vagrant/ **/*.tar.gz +**/*.tgz config/apt_repo/keys/ansible.pub config/certificate_authority/keys/id_rsa-cert.pub hosts diff --git a/.travis.yml b/.travis.yml index 2d63f95..562a27d 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,10 +1,12 @@ sudo: required dist: trusty before_install: + - ./build/build.sh - ./tests/test_setup.sh + - ./tests/install.sh language: python install: - - python smartcity-middleware.py install -f middleware.conf - - python smartcity-middleware.py start + - ideam install + - ideam start script: - python ./tests/pytest.py diff --git a/README.rst b/README.rst index 92c8b24..9fadcc0 100644 --- a/README.rst +++ b/README.rst @@ -20,13 +20,15 @@ Requirements ============ - ``Docker``: `Installation steps for Docker in Ubuntu/Debian `_ - ``Ansible``: `Installation steps for Ansible `_ -- ``Python-pip``: Python package dependency (pathlib2). Use the command ``python -m pip install pathlib2`` - ``ssh-keygen``: Generate a RSA Key Pair for the user who will install the script. - + + Release ======= -smartcity-middleware v1.0.0_ +ideam v1.0.0_ + +Use the ideam deb file to install in Linux machines after satisfying the requirements. ``dpkg -i ideam.deb`` . .. _v1.0.0: https://github.com/rbccps-iisc/ideam/releases/latest @@ -34,59 +36,43 @@ smartcity-middleware v1.0.0_ Configuration ============= -middleware.conf_ v1.0.0 - -.. _middleware.conf: https://github.com/rbccps-iisc/ideam/blob/master/middleware.conf -middleware.conf:: - - DESCRIPTION - - middleware.conf is the configuration file for the smartcity-middleware application. - - KONG, RABBITMQ, TOMCAT, CATALOGUE, LDAP docker containers requires a persistent data, config storage locations. +ideam.conf_ v1.0.0 - DATA_STORAGE: Specify an empty directory in the system that docker could use - for keeping data files. Here, providing separate directories for kong, rabbitmq, - tomcat, catalogue is recommended. Give all user rwx permission to these directories. +- Config file is located at ``/etc/ideam/ideam.conf``. - CONFIG_STORAGE: Specify an empty directory in the system that docker could use - for keeping config files. +- The data files or docker persistent storage is at ``/var/ideam/data`` directory. - LOG_LOCATION: Specify an empty directory in the system that docker could use - for keeping log files. +- The persistent storage can be changed by modifying the ``ideam.conf`` before installation. - SSH and other service ports can be mapped to host machine ports. - - SYSTEM_CONFIG specify system specific configurations for the installation steps. - - SSH_PUBLIC_KEY: Specify a ssh public key which will be used in ssh authentication of the user to - docker containers. +.. _ideam.conf: https://github.com/rbccps-iisc/ideam/blob/master/ideam.conf Steps to Install ================ -After configuring the ``middleware.conf`` file, do the following steps. +After downloading and installing the deb file, do the following steps. +---------------------------------------+-----------------------------------------------------------------------------+ -| Installation | ``python smartcity-middleware.py install --config-file middleware.conf`` | +| Installation | ``ideam install`` or if you cloned ``./ideam install`` | +---------------------------------------+-----------------------------------------------------------------------------+ -| Start Middleware | ``python smartcity-middleware.py start`` | +| Start Ideam | ``ideam start`` or if you cloned ``./ideam start`` | +---------------------------------------+-----------------------------------------------------------------------------+ -| Serving Middleware at | ``https://localhost:10443`` | +| Serving Ideam at | ``https://localhost:10443`` | +---------------------------------------+-----------------------------------------------------------------------------+ +- The application will be serving with a self-signed certificate. + If you want to use your certificate, have your .crt and .key file as ``/usr/share/ideam/config/kong/default_443.crt`` and + ``/usr/share/ideam/config/kong/default_443.key`` respectively and do a fresh installation. + +- If installation fails at some instance, all the time-dated logs are available at ``/tmp/ideam-%Y-%m-%d-%H-%M.log``. + Comment ======= -- Please satisfy the requirements mentioned in ``middleware.conf`` file. -- If the setup fails at any stage for reasons like internet connection issues, you can continue the failed installation using the following command. - ``python smartcity-middleware.py install --config-file middleware.conf -l kong,tomcat,hypercat,ldapd,elasticsearch,rabbitmq,apt_repo,pushpin`` -- The application will be serving with a self-signed certificate. If you want to use your certificate, have your .crt and .key file as ``config/kong/default_443.crt`` and ``config/kong/default_443.key`` respectively and do a fresh installation. +----------------------------------------------------------------+----------------------------------------------------------+ -| RBCCPS MIDDLEWARE API URLs | MIDDLEWARE API URLs | +| RBCCPS IoT MIDDLEWARE IDEAM API URL | INSTALLED IOT MIDDLEWARE IDEAM API URLs | +================================================================+==========================================================+ | https://smartcity.rbccps.org/api/1.0.0/register | https://localhost:10443/api/1.0.0/register | +----------------------------------------------------------------+----------------------------------------------------------+ @@ -101,171 +87,18 @@ Comment | https://smartcity.rbccps.org/api/1.0.0/db | https://localhost:10443/api/1.0.0/db | +----------------------------------------------------------------+----------------------------------------------------------+ -For example, registration of device to local middleware is as follows. - -command:: - - curl -X POST \ - https://smartcity.rbccps.org/api/1.0.0/register \ - -H 'apikey: guest' \ - -H 'content-type: application/json' \ - -d '{ - "entitySchema": { - "refCatalogueSchema": "generic_iotdevice_schema.json", - "resourceType": "streetlight", - "tags": ["onstreet", "Energy", "still under development!"], - "refCatalogueSchemaRelease": "0.1.0", - "latitude": { - "value": 13.0143335, - "ontologyRef": "http://www.w3.org/2003/01/geo/wgs84_pos#" - }, - "longitude": { - "value": 77.5678424, - "ontologyRef": "http://www.w3.org/2003/01/geo/wgs84_pos#" - }, - "owner": { - "name": "IISC", - "website": "http://www.iisc.ac.in" - }, - "provider": { - "name": "Robert Bosch Centre for Cyber Physical Systems, IISc", - "website": "http://rbccps.org" - }, - "geoLocation": { - "address": "80 ft Road, Bangalore, 560012" - }, - "data_schema": { - "type": "object", - "properties": { - "dataSamplingInstant": { - "type": "number", - "description": "Sampling Time in EPOCH format", - "units": "seconds", - "permissions": "read", - "accessModifier": "public" - }, - "caseTemperature": { - "type": "number", - "description": "Temperature of the device casing", - "units": "degreeCelsius", - "permissions": "read", - "accessModifier": "public" - }, - "powerConsumption": { - "type": "number", - "description": "Power consumption of the device", - "units": "watts", - "permissions": "read", - "accessModifier": "public" - }, - "luxOutput": { - "type": "number", - "description": "lux output of LED measured at LED", - "units": "lux", - "permissions": "read", - "accessModifier": "public" - }, - "ambientLux": { - "type": "number", - "description": "lux value of ambient", - "units": "lux", - "permissions": "read", - "accessModifier": "public" - }, - "targetPowerState": { - "type": "string", - "enum": ["ON", "OFF"], - "units": "dimensionless", - "description": "If set to ON, turns ON the device. If OFF turns OFF the device. Writeable parameter. Writeable only allowed for authorized apps", - "permissions": "read-write", - "accessModifier": "protected" - }, - "targetBrightnessLevel": { - "type": "number", - "description": "Number between 0 to 100 to indicate the percentage brightness level. Writeable only allowed for authorized apps", - "units": "percent", - "permissions": "read-write", - "accessModifier": "protected" - }, - "targetControlPolicy": { - "enum": ["AUTO_TIMER", "AUTO_LUX", "MANUAL"], - "units": "dimensionless", - "permissions": "read-write", - "description": "Indicates which of the behaviours the device should implement. AUTO_TIMER is timer based, AUTO_LUX uses ambient light and MANUAL is controlled by app. Writeable only allowed for authorized apps", - "accessModifier": "protected" - }, - "targetAutoTimerParams": { - "type": "object", - "permissions": "read-write", - "properties": { - "targetOnTime": { - "type": "number", - "description": "Indicates time of day in seconds from 12 midnight when device turns ON in AUTO_TIMER. Writeable only allowed for authorized apps", - "units": "seconds", - "accessModifier": "protected" - }, - "targetOffTime": { - "type": "number", - "description": "Indicates time of day in seconds from 12 midnight when device turns OFF in AUTO_TIMER. Writeable only allowed for authorized apps", - "units": "seconds", - "accessModifier": "protected" - } - } - }, - "targetAutoLuxParams": { - "type": "object", - "permissions": "read-write", - "properties": { - "targetOnLux": { - "type": "number", - "description": "Indicates ambient lux when device turns ON in AUTO_LUX. Writeable only allowed for authorized apps", - "units": "lux", - "accessModifier": "protected" - }, - "targetOffLux": { - "type": "number", - "description": "Indicates ambient lux when device turns OFF in AUTO_LUX. Writeable only allowed for authorized apps", - "units": "lux", - "accessModifier": "protected" - } - } - } - }, - "additionalProperties": false - }, - "serialization_from_device": { - "format": "protocol-buffers", - "schema_ref": { - "type": "proto 2", - "mainMessageName": "sensor_values", - "link": "https://raw.githubusercontent.com/rbccps-iisc/applications-streetlight/master/proto_stm/txmsg/sensed.proto" - } - }, - "serialization_to_device": { - "format": "protocol-buffers", - "schema_ref": { - "type": "proto 2", - "mainMessageName": "targetConfigurations", - "link": "https://raw.githubusercontent.com/rbccps-iisc/applications-streetlight/master/proto_stm/rxmsg/actuated.proto" - } - }, - "id": "streetLight_1A_212" - } - }' - - NOTE ==== - Installation in Linux machines can fail for the following reasons. - - If you are in a corporate network that blocks Google DNS Servers, the ``docker build`` command fails. - + - If you are in a corporate network that blocks Google DNS Servers, the ``ideam install`` command fails. + To fix it, add your corporate DNS servers in DOCKER_OPTS in /etc/default/docker. (for SysV machines) - DOCKER_OPTS="--dns 208.67.222.222 --dns 208.67.220.220" + DOCKER_OPTS="--dns 208.67.222.222 --dns 208.67.220.220" If this fails to set the DNS properly, try updating /etc/docker/daemon.json with the following (for systemd machines) - { "dns": ["208.67.222.222", "208.67.220.220"] } + { "dns": ["208.67.222.222", "208.67.220.220"] } - - Middleware has been tested on macOS as well. +- IDEAM has been tested on MacOS as well. diff --git a/build/.gitignore b/build/.gitignore new file mode 100644 index 0000000..42e527e --- /dev/null +++ b/build/.gitignore @@ -0,0 +1,6 @@ +* +*/ +!debian/ +!debian/* +!.gitignore +!build.sh \ No newline at end of file diff --git a/build/build.sh b/build/build.sh new file mode 100755 index 0000000..eeb1ede --- /dev/null +++ b/build/build.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +cd ./build/ +mkdir ideam_0.0-1 +mkdir -p ideam_0.0-1/usr/local/bin +mkdir -p ideam_0.0-1/usr/share/ideam +mkdir -p ideam_0.0-1/etc/ideam +mkdir -p ideam_0.0-1/DEBIAN +mkdir -p ideam_0.0-1/var/ideam/data/kong +mkdir -p ideam_0.0-1/var/ideam/data/kong-config +mkdir -p ideam_0.0-1/var/ideam/data/catalogue +mkdir -p ideam_0.0-1/var/ideam/data/rabbitmq +mkdir -p ideam_0.0-1/var/ideam/data/ldap +mkdir -p ideam_0.0-1/var/ideam/data/tomcat +mkdir -p ideam_0.0-1/var/ideam/data/logs/kong +mkdir -p ideam_0.0-1/var/ideam/data/logs/rabbitmq +mkdir -p ideam_0.0-1/var/ideam/data/logs/tomcat +cp ../ideam.py ideam_0.0-1/usr/local/bin/ideam +cp debian/control ideam_0.0-1/DEBIAN/control +cp ../ideam.conf ideam_0.0-1/etc/ideam/ +chmod +x ideam_0.0-1/usr/local/bin/ideam +cd ../ +tar --exclude='./build' --exclude='./.git' --exclude='./.idea' --exclude='*.retry' --exclude='*.tar.gz' --exclude='./ideam.tgz' --exclude='*.DS_Store' --exclude='./.gitignore' -zcvf ideam.tgz ./ +tar -xvzf ideam.tgz -C build/ideam_0.0-1/usr/share/ideam/ +cd build/ +chmod -R 777 ideam_0.0-1/var/ideam/data/logs/kong +dpkg-deb --build ideam_0.0-1/ \ No newline at end of file diff --git a/build/debian/control b/build/debian/control new file mode 100644 index 0000000..7b426b0 --- /dev/null +++ b/build/debian/control @@ -0,0 +1,9 @@ +Package: ideam +Version: 0.0-1 +Section: devel +Priority: optional +Architecture: all +Depends: ansible, python-pkg-resources, python-setuptools, docker-ce +Maintainer: Harish Anand +Description: IoT Data Exchange And Middleware + (IDEAM) for smartcities. diff --git a/delete.sh b/delete.sh deleted file mode 100755 index 89cd460..0000000 --- a/delete.sh +++ /dev/null @@ -1,7 +0,0 @@ -#!/usr/bin/env bash -cd /data/kong && rm -rf * -cd /data/tomcat && rm -rf * -cd /data/rabbitmq && rm -rf * -cd /data/logs/kong && rm -rf * -cd /data/logs/rabbitmq && rm -rf * -cd /data/logs/tomcat && rm -rf * diff --git a/ideam.conf b/ideam.conf new file mode 100644 index 0000000..f5af617 --- /dev/null +++ b/ideam.conf @@ -0,0 +1,69 @@ +# ideam.conf +# +# DESCRIPTION +# +# ideam.conf is the configuration file for the ideam application. +# +# KONG, RABBITMQ, TOMCAT, CATALOGUE, LDAP docker containers requires a persistent data, config storage locations. +# +# DATA_STORAGE: Specify an empty directory in the system that docker could use +# for keeping data files. Here, providing separate directories for kong, rabbitmq, +# tomcat, catalogue is recommended. +# +# CONFIG_STORAGE: Specify an empty directory in the system that docker could use +# for keeping config files. +# +# LOG_LOCATION: Specify an empty directory in the system that docker could use +# for keeping log files. +# +# SSH and other service ports can be mapped to host machine ports. +# +# SYSTEM_CONFIG specify system specific configurations for the installation steps. +# +# SSH_PUBLIC_KEY: Specify a ssh public key which will be used in ssh authentication of the user to +# docker containers. +# + +[PUSHPIN] +SSH=10022 +HTTPS=10443 + +[KONG] +DATA_STORAGE = /var/ideam/data/kong +CONFIG_STORAGE = /var/ideam/data/kong-config +LOG_LOCATION = /var/ideam/data/logs/kong +SSH=11022 + +[RABBITMQ] +DATA_STORAGE = /var/ideam/data/rabbitmq +LOG_LOCATION = /var/ideam/data/logs/rabbitmq +SSH=12022 +HTTP = 12080 +MANAGEMENT = 12081 +AMQP = 12082 +MQTT = 12083 + +[ELASTICSEARCH] +SSH=13022 +KIBANA=13081 + +[TOMCAT] +DATA_STORAGE = /var/ideam/data/tomcat +LOG_LOCATION = /var/ideam/data/logs/tomcat +SSH=14022 +HTTP=14080 + +[LDAP] +SSH=15022 +LDAP=15389 + +[CATALOGUE] +DATA_STORAGE = /var/ideam/data/catalogue +SSH=16022 +HTTP=16080 + +[KONGA] +HTTP=17080 + +[SYSTEM_CONFIG] +SSH_PUBLIC_KEY = ~/.ssh/id_rsa.pub diff --git a/smartcity-middleware.py b/ideam.py similarity index 90% rename from smartcity-middleware.py rename to ideam.py index 4cd0ead..954c0bf 100644 --- a/smartcity-middleware.py +++ b/ideam.py @@ -1,9 +1,15 @@ +#!/usr/bin/env python2 +import sys +import os +VERSION = '0.0-1' +if os.path.exists("/etc/ideam/ideam.conf"): + sys.path.append("/usr/share/ideam") + os.chdir("/usr/share/ideam") import modules.download_packages as download_packages import modules.start as container_start import modules.install as container_setup from datetime import datetime from modules.utils import setup_logging -import sys import argparse @@ -17,7 +23,7 @@ def error(self, message): def install(arguments): """ Installs docker images and containers.""" - if args.limit: + if arguments.limit: container_setup.ansible_installation(arguments.limit) else: setup_logging(log_file=arguments.log_file) @@ -48,7 +54,7 @@ def restart(arguments): if __name__ == '__main__': - default_log_file = "/tmp/" + datetime.now().strftime("smartcity-middleware-%Y-%m-%d-%H-%M.log") + default_log_file = "/tmp/" + datetime.now().strftime("ideam-%Y-%m-%d-%H-%M.log") parser = MyParser() subparsers = parser.add_subparsers(dest='command') @@ -64,8 +70,8 @@ def restart(arguments): install_parser.add_argument("--log-file", help="Path to log file", default=default_log_file) install_parser.add_argument("-f", "--config-file", - help="Path to the conf file. See middleware.conf for an example.", - required=True) + help="Path to the conf file. See /etc/ideam/ideam.conf for an example.", + default="/etc/ideam/ideam.conf") # start command start_parser = subparsers.add_parser('start', help='Start all the docker containers in the middleware') @@ -93,4 +99,4 @@ def restart(arguments): elif args.command == "start": start(args) elif args.command == "restart": - restart(args) + start(args) diff --git a/middleware.conf b/middleware.conf deleted file mode 100644 index 0935334..0000000 --- a/middleware.conf +++ /dev/null @@ -1,66 +0,0 @@ -# middleware.conf v0.1.0.alpha -# -# DESCRIPTION -# middleware.conf is the configuration file for the smartcity-middleware application. -# -# KONG, RABBITMQ, TOMCAT, CATALOGUE, LDAP docker containers requires a persistent data, config storage locations. -# DATA_STORAGE: Specify an empty directory in the system that docker could use -# for keeping data files. Here, providing separate directories for kong, rabbitmq, -# tomcat, catalogue is recommended. Give all user rwx permission to these directories. -# -# CONFIG_STORAGE: Specify an empty directory in the system that docker could use -# for keeping config files. -# -# LOG_LOCATION: Specify an empty directory in the system that docker could use -# for keeping log files. -# -# SYSTEM_CONFIG specify system specific configurations for the installation steps. -# SSH_PUBLIC_KEY: Specify a ssh public key which will be used in ssh authentication of the user to -# docker containers. -# -# -[PUSHPIN] -SSH=10022 -HTTPS=10443 - -[KONG] -DATA_STORAGE = /data/kong -CONFIG_STORAGE = /data/kong-config -LOG_LOCATION = /data/logs/kong -SSH=11022 - -[RABBITMQ] -DATA_STORAGE = /data/rabbitmq -LOG_LOCATION = /data/logs/rabbitmq -SSH=12022 -HTTP = 12080 -MANAGEMENT = 12081 -AMQP = 12082 -MQTT = 12083 - -[ELASTICSEARCH] -SSH=13022 -KIBANA=13081 - -[TOMCAT] -DATA_STORAGE = /data/tomcat -LOG_LOCATION = /data/logs/tomcat -SSH=14022 -HTTP=14080 - -[LDAP] -SSH=15022 -LDAP=15389 - -[CATALOGUE] -DATA_STORAGE = /data/catalogue -SSH=16022 -HTTP=16080 - -[KONGA] -HTTP=17080 - -[SYSTEM_CONFIG] -SSH_PUBLIC_KEY = ~/.ssh/id_rsa.pub - - diff --git a/modules/download_packages.py b/modules/download_packages.py index 5511c5a..c134bb8 100644 --- a/modules/download_packages.py +++ b/modules/download_packages.py @@ -1,7 +1,7 @@ import urllib from utils import output_error, output_info, output_ok import traceback -from pathlib2 import Path +import os def download_file(url, filename, success_msg, failure_msg, log_file): @@ -14,8 +14,7 @@ def download_file(url, filename, success_msg, failure_msg, log_file): failure_msg (string): failure message to be displayed when download fails log_file (string): log file path """ - my_file = Path(filename) - if my_file.is_file(): + if os.path.exists(filename): output_ok(success_msg) else: try: diff --git a/modules/install.py b/modules/install.py index 83ca6db..0faf383 100644 --- a/modules/install.py +++ b/modules/install.py @@ -4,9 +4,6 @@ import ConfigParser import os from time import time -kong_log_location="" -rabbitmq_log_location="" -tomcat_log_location="" def remove_containers(log_file): @@ -143,7 +140,7 @@ def unique_value(): return str(int(time())) -def docker_setup(log_file, config_path="middleware.conf"): +def docker_setup(log_file, config_path="ideam.conf"): """ Creates docker instances for kong, ca, hypercat, rabbitmq, elastic search, apache storm, ldap, ntp and bind server from an ubuntu-ssh image. First, docker creates certificate authority (CA) instance and then have the CA certify Ansible user's public key. A new docker image with this CA's public key in TrustedUserCAKeys is created to @@ -170,10 +167,8 @@ def docker_setup(log_file, config_path="middleware.conf"): kong_config_storage = config.get('KONG', 'CONFIG_STORAGE') output_info("Using {0} as Kong's config persistant storage. ".format(kong_config_storage)) rabbitmq_storage = config.get('RABBITMQ', 'DATA_STORAGE') - rabbitmq_log_location = config.get('RABBITMQ', 'LOG_LOCATION') output_info("Using {0} as RabbitMQ's persistant storage. ".format(rabbitmq_storage)) tomcat_storage = config.get('TOMCAT', 'DATA_STORAGE') - tomcat_log_location = config.get('TOMCAT', 'LOG_LOCATION') output_info("Using {0} as Apache Tomcat's persistant storage. ".format(tomcat_storage)) catalogue_storage = config.get('CATALOGUE', 'DATA_STORAGE') output_info("Using {0} as Catalogue's persistant storage. ".format(catalogue_storage)) @@ -186,15 +181,17 @@ def docker_setup(log_file, config_path="middleware.conf"): key = config.get('SYSTEM_CONFIG', 'SSH_PUBLIC_KEY') output_info("Using {0} as your ssh key for certification. ".format(key)) - home = os.path.expanduser('~') with open(home + "/.ssh/configs", 'w+') as f: f.write("IdentityFile {0}\n".format(key)) key = key.replace("~", home) + + subprocess_popen("find "+key, log_file, "Missing ssh public key file in {}. Create one using command ssh-keygen.". + format(key)) cmd = 'cp -r ' + key + ' ' + os.getcwd() + '/config/certificate_authority/keys/id_rsa.pub' subprocess_popen(cmd, log_file, "Copying to /config/certificate_authority/keys/ failed.") - subprocess_with_print("docker build -t ansible/ubuntu-ssh --build-arg CACHEBUST={0} -f images/Dockerfile.ubuntu .". + subprocess_with_print("docker build -t ansible/ubuntu-ssh --no-cache -f images/Dockerfile.ubuntu .". format(unique_value()), success_msg="Created ansible/ubuntu-ssh docker image. ", failure_msg="Building ubuntu image from images/Dockerfile.ubuntu failed.", @@ -216,7 +213,8 @@ def docker_setup(log_file, config_path="middleware.conf"): subprocess_popen(cmd, log_file, "Copying Certificate Authority's cert file to ansible's .ssh/ failed.") output_ok("Copied Certificate Authority's cert file to Ansible's .ssh. ") - cmd = "docker build -t ansible/ubuntu-certified-aptrepo:1.0 --build-arg CACHEBUST={0} -f images/Dockerfile.ubuntu.certified.aptrepo.readytoserve .".format(unique_value()) + cmd = "docker build -t ansible/ubuntu-certified-aptrepo:1.0 --build-arg CACHEBUST={0} " \ + "-f images/Dockerfile.ubuntu.certified.aptrepo.readytoserve .".format(unique_value()) subprocess_with_print(cmd, success_msg="Created ansible/ubuntu-certified-aptrepo:1.0 docker image. ", failure_msg="Building ubuntu image from " @@ -224,20 +222,23 @@ def docker_setup(log_file, config_path="middleware.conf"): log_file=log_file, exit_on_fail=True) - cmd = "docker build -t ansible/ubuntu-certified-catalogue:1.0 --build-arg CACHEBUST={0} -f images/Dockerfile.ubuntu.certified.catalogue .".format(unique_value()) + cmd = "docker build -t ansible/ubuntu-certified-catalogue:1.0 --build-arg CACHEBUST={0} " \ + "-f images/Dockerfile.ubuntu.certified.catalogue .".format(unique_value()) subprocess_with_print(cmd, success_msg="Created ansible/ubuntu-certified-catalogue:1.0 docker image. ", failure_msg="Building ubuntu image from images/Dockerfile.ubuntu.certified.catalogue failed.", log_file=log_file, exit_on_fail=True) - cmd = "docker build -t ansible/ubuntu-certified-kong:1.0 --build-arg CACHEBUST={0} -f images/Dockerfile.ubuntu.certified.kong .".format(unique_value()) + cmd = "docker build -t ansible/ubuntu-certified-kong:1.0 --build-arg CACHEBUST={0} " \ + "-f images/Dockerfile.ubuntu.certified.kong .".format(unique_value()) subprocess_with_print(cmd, success_msg="Created ansible/ubuntu-certified-kong:1.0 docker image. ", failure_msg="Building ubuntu image from images/Dockerfile.ubuntu.certified.kong failed.", log_file=log_file, exit_on_fail=True) - cmd = "docker build -t ansible/ubuntu-certified-rabbitmq:1.0 --build-arg CACHEBUST={0} -f images/Dockerfile.ubuntu.certified.rabbitmq .".format(unique_value()) + cmd = "docker build -t ansible/ubuntu-certified-rabbitmq:1.0 --build-arg CACHEBUST={0} " \ + "-f images/Dockerfile.ubuntu.certified.rabbitmq .".format(unique_value()) subprocess_with_print(cmd, success_msg="Created ansible/ubuntu-certified-rabbitmq:1.0 docker image. ", failure_msg="Building ubuntu image from images/Dockerfile.ubuntu.certified.rabbitmq failed.", @@ -251,7 +252,8 @@ def docker_setup(log_file, config_path="middleware.conf"): log_file=log_file, exit_on_fail=True) - subprocess_with_print("docker build -t ansible/tomcat --build-arg CACHEBUST={0} -f images/Dockerfile.tomcat .".format(unique_value()), + subprocess_with_print("docker build -t ansible/tomcat --build-arg CACHEBUST={0} " + "-f images/Dockerfile.tomcat .".format(unique_value()), success_msg="Created ansible/tomcat docker image. ", failure_msg="Building ansible/tomcat image from images/Dockerfile.tomcat failed.", log_file=log_file, @@ -278,7 +280,8 @@ def docker_setup(log_file, config_path="middleware.conf"): ip, port, details = create_instance("kong", "ansible/ubuntu-certified-kong:1.0", storage_host=kong_storage, storage_guest="/var/lib/postgresql", - log_file=log_file) + log_file=log_file, + log_storage=kong_log_location) instance_details["kong"] = [ip, port] output_ok("Created Kong docker instance. \n " + details) @@ -325,7 +328,8 @@ def docker_setup(log_file, config_path="middleware.conf"): konga = config.get('KONGA', 'HTTP') - cmd = 'docker run -d -p {0}:1337 --net mynet --link kong:kong --name konga -e "NODE_ENV=production" pantsel/konga'.format(konga) + cmd = 'docker run -d -p {0}:1337 --net mynet --link kong:kong --name konga -e "NODE_ENV=production" pantsel/konga'.\ + format(konga) subprocess_with_print(cmd, success_msg="Created KONGA docker instance. ", @@ -335,7 +339,8 @@ def docker_setup(log_file, config_path="middleware.conf"): create_ansible_host_file(instance_details) -def create_instance(server, image, log_file, storage_host="", storage_guest="", config_path="middleware.conf"): +def create_instance(server, image, log_file, storage_host="", storage_guest="", config_path="ideam.conf", + log_storage=""): """ Create a docker instance from the image provided with persistent storages. Args: @@ -345,6 +350,7 @@ def create_instance(server, image, log_file, storage_host="", storage_guest="", storage_host (string): mount point created in the server for persistent storage storage_guest (string): location inside docker where the persistent storage occurs config_path (string): location of config file + log_storage (string): storage space for logs """ port = "" container_id = "" @@ -354,8 +360,9 @@ def create_instance(server, image, log_file, storage_host="", storage_guest="", if server == "kong": # separate kong log storage needed ssh = config.get('KONG', 'SSH') - cmd = "docker run -d -p {4}:22 --net=mynet --hostname={0} -v {2}:{3} -v /data/logs/kong:/tmp --cap-add=NET_ADMIN --name={0} {1}".\ - format(server, image, storage_host, storage_guest, ssh) + cmd = "docker run -d -p {4}:22 --net=mynet --hostname={0} " \ + "-v {2}:{3} -v {5}:/tmp --cap-add=NET_ADMIN --name={0} {1}".\ + format(server, image, storage_host, storage_guest, ssh, log_storage) try: out, err = subprocess_popen(cmd, @@ -372,10 +379,12 @@ def create_instance(server, image, log_file, storage_host="", storage_guest="", http = config.get('RABBITMQ', 'HTTP') amqp = config.get('RABBITMQ', 'AMQP') mqtt = config.get('RABBITMQ', 'MQTT') + log_storage = config.get('RABBITMQ', 'LOG_LOCATION') management = config.get('RABBITMQ', 'MANAGEMENT') - cmd = "docker run -d -p {4}:22 -p {5}:8000 -p {6}:5672 -p {7}:1883 -p {8}:15672 --net=mynet --hostname={0} -v {2}:{3} -v /data/logs/rabbitmq:/var/log/rabbitmq -v /data/logs/rabbitmq:/var/log/supervisor --cap-add=NET_ADMIN --name={0} {1}".\ - format(server, image, storage_host, storage_guest, ssh, http, amqp, mqtt, management) + cmd = "docker run -d -p {4}:22 -p {5}:8000 -p {6}:5672 -p {7}:1883 -p {8}:15672 --net=mynet --hostname={0}" \ + " -v {2}:{3} -v {9}:/var/log/rabbitmq -v {9}:/var/log/supervisor --cap-add=NET_ADMIN --name={0} {1}".\ + format(server, image, storage_host, storage_guest, ssh, http, amqp, mqtt, management, log_storage) try: out, err = subprocess_popen(cmd, @@ -390,8 +399,10 @@ def create_instance(server, image, log_file, storage_host="", storage_guest="", elif server == "tomcat": # separate tomcat log storage needed ssh = config.get('TOMCAT', 'SSH') http = config.get('TOMCAT', 'HTTP') - cmd = "docker run -d -p {4}:22 -p {5}:8080 --net=mynet --hostname={0} -v {2}:{3} -v /data/logs/tomcat:/var/log/supervisor --cap-add=NET_ADMIN --name={0} {1}".\ - format(server, image, storage_host, storage_guest, ssh, http) + log_storage = config.get('TOMCAT', 'LOG_LOCATION') + cmd = "docker run -d -p {4}:22 -p {5}:8080 --net=mynet --hostname={0} -v {2}:{3} -v {6}:/var/log/supervisor" \ + " --cap-add=NET_ADMIN --name={0} {1}".\ + format(server, image, storage_host, storage_guest, ssh, http, log_storage) try: out, err = subprocess_popen(cmd, @@ -406,7 +417,8 @@ def create_instance(server, image, log_file, storage_host="", storage_guest="", elif server == "hypercat": # separate data storage needed ssh = config.get('CATALOGUE', 'SSH') http = config.get('CATALOGUE', 'HTTP') - cmd = "docker run -d -p {4}:22 -p {5}:8000 --net=mynet --hostname={0} -v {2}:{3} --cap-add=NET_ADMIN --name={0} {1}".\ + cmd = "docker run -d -p {4}:22 -p {5}:8000 --net=mynet --hostname={0} " \ + "-v {2}:{3} --cap-add=NET_ADMIN --name={0} {1}".\ format(server, image, storage_host, storage_guest, ssh, http) try: @@ -422,7 +434,8 @@ def create_instance(server, image, log_file, storage_host="", storage_guest="", elif server == "ldapd": # separate data storage needed ssh = config.get('LDAP', 'SSH') ldap = config.get('LDAP', 'LDAP') - cmd = "docker run -d -p {4}:22 -p {5}:8389 --net=mynet --hostname={0} -v {2}:{3} --cap-add=NET_ADMIN --name={0} {1}".\ + cmd = "docker run -d -p {4}:22 -p {5}:8389 --net=mynet --hostname={0} " \ + "-v {2}:{3} --cap-add=NET_ADMIN --name={0} {1}".\ format(server, image, storage_host, storage_guest, ssh, ldap) try: @@ -454,7 +467,8 @@ def create_instance(server, image, log_file, storage_host="", storage_guest="", elif server == "elasticsearch": ssh = config.get('ELASTICSEARCH', 'SSH') kibana = config.get('ELASTICSEARCH', 'KIBANA') - cmd = "docker run -d -p {2}:22 -p {3}:5601 --net=mynet --hostname={0} --cap-add=NET_ADMIN --name={0} {1}".format(server, image, ssh, kibana) + cmd = "docker run -d -p {2}:22 -p {3}:5601 --net=mynet " \ + "--hostname={0} --cap-add=NET_ADMIN --name={0} {1}".format(server, image, ssh, kibana) try: out, err = subprocess_popen(cmd, log_file, diff --git a/publish.sh b/publish.sh deleted file mode 100755 index 0470f45..0000000 --- a/publish.sh +++ /dev/null @@ -1,7 +0,0 @@ -#!/bin/bash -i=0 -while [ $i -lt 1000 ]; -do -curl -i -k -X POST "http://localhost:32768/api/0.1.0/publish" -H 'apikey: 79a7c7ed9aef4b848abf71e75f358fbb' -d '{"exchange": "amq.topic", "key": "testDemo", "body": "Command Data"}' -let i=i+1 -done diff --git a/requirements.txt b/requirements.txt index f1b9d99..d0acc75 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,2 @@ -pathlib2 graphviz==0.8 -wheel==0.24.0 - +wheel==0.24.0 \ No newline at end of file diff --git a/test.yaml b/test.yaml deleted file mode 100644 index 32e28ac..0000000 --- a/test.yaml +++ /dev/null @@ -1,5 +0,0 @@ ---- -- hosts: kong - remote_user: ansible - tasks: - - include: tests/kong_api_pushpin.yml \ No newline at end of file diff --git a/tests/delete.sh b/tests/delete.sh new file mode 100755 index 0000000..7e00fa5 --- /dev/null +++ b/tests/delete.sh @@ -0,0 +1,7 @@ +#!/usr/bin/env bash +cd /var/ideam/data/kong && rm -rf * +cd /var/ideam/data/tomcat && rm -rf * +cd /var/ideam/data/rabbitmq && rm -rf * +cd /var/ideam/data/logs/kong && rm -rf * +cd /var/ideam/data/logs/rabbitmq && rm -rf * +cd /var/ideam/data/logs/tomcat && rm -rf * diff --git a/tests/install.sh b/tests/install.sh new file mode 100755 index 0000000..08c6e0e --- /dev/null +++ b/tests/install.sh @@ -0,0 +1,4 @@ +#!/usr/bin/env bash +# Tests are done using the generated ideam deb file in build directory +cd build/ +sudo dpkg -i ideam_0.0-1.deb \ No newline at end of file diff --git a/tests/test_setup.sh b/tests/test_setup.sh index e693a2b..818021a 100755 --- a/tests/test_setup.sh +++ b/tests/test_setup.sh @@ -1,22 +1,15 @@ #!/usr/bin/env bash +# The following are the packages to be installed in a new vm machine for test purposes. +# Ansible, docker and python. sudo apt-get -y update && sudo apt-get install -y software-properties-common && sudo apt-add-repository ppa:ansible/ansible -y && sudo apt-get -y update && sudo apt-get install -y ansible sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common -sudo apt install python-pip python -y -python -m pip install pathlib2 +sudo apt install python -y curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" -y sudo apt-get update -y sudo apt-get install docker-ce -y -sudo mkdir -p /data/kong -sudo mkdir -p /data/kong-config -sudo mkdir -p /data/catalogue -sudo mkdir -p /data/rabbitmq -sudo mkdir -p /data/ldap -sudo mkdir -p /data/tomcat -sudo mkdir -p /data/logs/kong -sudo mkdir -p /data/logs/rabbitmq -sudo mkdir -p /data/logs/tomcat sudo usermod -aG docker $USER +# create ssh keys ssh-keygen -f $HOME/.ssh/id_rsa -t rsa -N '' -sudo chmod -R 777 /data/* -sudo sysctl -w vm.max_map_count=662144 +# vm.max_map_count must be set to higher value for elasticsearch. +sudo sysctl -w vm.max_map_count=662144 \ No newline at end of file