Version 2.0

You can find the release notes for every releases of GOBii in this page: Version 2.0#menu-link-content

Version Warning

Versioning

When using this document, make sure that you are deploying the correct GOBii version number. The official build string for GOBii version 2.0 is below (a parameter string you need to run the shell scripts):

GOBii Release Version

release-2.0-5

Operating System, bash & Docker Versions

The following are the versions used when developing and testing within GOBii

Operating Systems:

Ubuntu 16.04.5 LTS
CentOS Linux release 7.2.1511 (Core)

Bash Version:

Ubuntu: GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)
CentOS: GNU bash, version 4.2.46(1)-release (x86_64-redhat-linux-gnu)

Docker Version:

Ubuntu: Docker version 18.06.1-ce, build e68fc7a
CentOS: Docker version 18.03.1-ce, build 9ee9f40

GDM Deployment Versions

This shows the Docker versions used for deployment of this release:

GDM: release-2.0-8
KDC: server_1.6.1-plugin_0.14.2-build_98
ownCloud: base
Portainer: latest
sherpa: latest

For any questions or clarifications, please contact Support Portal or send email to support@GOBiiproject.atlassian.net.

Introduction

In this section the Definition of Terms, Background and a brief overview of GOBii is described

Click to expand

Definition of Terms

Nodes = GOBii Nodes
- The term "nodes" here will always refer to the GOBii nodes, which are Docker containers that can be deployed to different servers or virtual environments. Server nodes, on the other hand, will be explicitly called "server node".

Background

GOBii is made up of multiple modules and categorized according to functions. A system diagram that shows these categories (by Docker container), the data flow, and the modules are available here.

Depending on your server topology, the instructions in this page may require some tweaking. For each sections with significant differences on steps depending on server topology, a "Note Box" like the one below will be written.

GOBii's deployment architecture is flexible and node-based. There are three main nodes: computation, database, and web. These nodes are now pre-baked into Docker images and can be deployed in their own server, VM, or in any combination of servers and virtual environments.

To give you an idea, here's an example topology and node-distribution:

Example Topology

Server 1:
	Server Head: GOBii Test (all nodes)
	Server Node1: GOBii Prod Database Node
	Server Node2: GOBii Prod Web Node
	Server Node3: GOBii Prod Compute Node

Current Limitation

You can put GDM nodes of the same GDM instance into one server, but we advise against mixing nodes of different instances into one server. Aside from competing for resources, there are potential conflict points that nodes from different instances may run into.

Initial Installation Prerequisites

Click to expand

The official repository for the deployment scripts is here. Make sure you clone or download the scripts from there. The branch you should get is release/<version> (ex. release/1.5). You can also get the master branch if you are deploying the latest, but because our clients can have varying versions on different servers, all release branches are kept.
Finalize your topology and write it down, because if you are deploying all three GOBii nodes to just one server, you run a different script as opposed to when you deploy GOBii into one server per node or any other variations (you run three scripts).
The servers should have the Docker engine version 17 and up, installed. Make sure the servers have access to the Dockerhub site.
1. Ubuntu: https://docs.Docker.com/engine/installation/linux/Docker-ce/ubuntu/#upgrade-Docker-ce-1
2. CentOS: https://docs.Docker.com/install/linux/Docker-ce/centos/
A mount point or a shared drive that all the nodes can access — this will be a volume mounted to all the three Docker containers.
The user that will run the scripts needs to be a 'sudoer' and under the GOBii and Dockergroups, so, preferably the user 'gadm'. But the username is arbitrary, it just needs to be consistent. You may find 'gadm sudoer' used in the rest of this document. Note: the name is flexible.

sudo usermod -aG Docker gadm

(Optional) A directory where the data of postgres will reside. The default will be Ubuntu's postgres directory in the DB Docker (ex. /var/lib/pgsql/data) which will be linked to Docker's default volume directory (ex. /usr/local/Docker/volumes/postgreslibubuntu)

GOBII TEST

For a test GOBii instance, you can use the vanilla version of the Dockers:

Backups

This section is for existing instances that already have data in storage. This will show the process used to back up the existing data.

Click here to expand...

Steps

Back up the data files from the existing GOBii instance. You can do so by using the backup script we provide. You can find usage instructions here: Version 2.0#onScripts-IncrementalDataBundleBackups. Ideally, you already have this set up to do incremental backups (as a cron job), so that prior to the upgrade, you just need to do run the script in incremental mode to make sure you capture all changes, then disable access to the system to make sure nobody loads new data while the upgrade is in progress. If that's not the case, you can run the script on full backup mode – just note that this may take several hours depending on the size of your data.
Backup the database (postgres) from the existing GOBii instance. You have two options:
1. Run the script we provide (Version 2.0#onScripts-PostgresqlRotatingIncrementalBackup) - this also is ideally set up as a cron job.
2. Manually back up the database
  1. Go into the database node and run pg_dumpall
    $> Docker exec -ti <GOBii_db_node> bash $> su postgres $> pg_dumpall > /data/all_databases.bak
3. The data from postgres actually persists as long as you don't delete the Docker volumes. However, we still recommend you back up the database to make sure we have redundancy.
OPTIONAL: If you have KDCompute Docker container running, back up its files by copying the directory /data/kdcompute_file_storage to a directory of your choice. This contains previous output and logs of QC jobs.

Deployment

This section will detail the scripts, parameters and process to deploy GOBii

Click here to expand...

Deployment Scripts and Parameters

Click here to expand...

Copy the deployment scripts and files from the cloned repository (prerequisite #1) to the shared drive (prerequisite #4).

We update the param files from time to time (i.e. new features being added), so please don't just copy-paste the sample param files below. They are just shown in this page for reference. Instead, pull from our deployment scripts git repository for a particular release you are deploying (ex. release/1.5).

The templates shown below was last updated for version 2.0

Edit the main parameter file. You can find a template in the repository (GOBiideployment/params/template_main.parameter). It will contain all the topology information and deployment credentials. The template is shown below, with each parameter explained on top of the corresponding line:

Default Credentials

All the passwords and some usernames have been omitted on the parameter file templates in this page for security. Make sure you check Version 2.0 to replace the parameters with the correct values. If you can't access the page with the default credentials, contact 00d483e952a2545b0152a4b36870000e or 00d483e961e2874401621ac073490001.

Version 2.0 Update on Dockerhub access

As of version 2.0 the container repos still exist under the user gadm01 but can not be viewed or uploaded to. This was done for security and maintenance purposes.

Please use gadmreader to pull an images from the gadm01 account.

template_main.parameters

This template has been updated with the most recent parameters as of Version 2.0. For ease of use, the template has been expanded with 'white' space between parameters to allow for a more readable and more easily editable structure.

Additionally, this file has taken the most recent parameters added at the bottom for any version deployment beyond the last so as to allow for easy copy and paste into existing parameter files.

Password Prompting...

As of version 2.0 any password set within the *main.parameters file set to "askme" have been configured to request the user password during script deployment. The request and password is hidden during deployment to keep the visibility and clear text passwords to a minimum.

If a password is set the script will continue without prompting for the user pass.

#-----------------------------------------------------------------------------#
### load parameters -- install.parameters for configuration
#-----------------------------------------------------------------------------#
### This file will be used by the_GOBii_ship_*.sh to deploy and configure the Docker images to target hosts.
#-----------------------------------------------------------------------------#

# This is your shared directory (will be mounted as volume to the Dockers), it 
# needs to be accessible from the 3 Dockers. 
### Example: BUNDLE_PARENT_PATH="/storage/dev_test"
BUNDLE_PARENT_PATH="<bundle parent path>"

# The name of the first crop - this needs to match what is in the provisioned 
# Docker image in our Docker hub.
### Example: Docker_CROP1_NAME="dev"
Docker_CROP1_NAME="<crop name>"

# The name of the second crop - this needs to match what is in the provisioned 
# Docker image in our Docker hub.
# Docker_CROP2_NAME="wheat"

# IP/Hostname of the Docker compute host
Docker_COMPUTE_HOST="<hostname or IP>"

# Name of the compute Docker. This is more of an alias to let you access it 
# conveniently.
Docker_COMPUTE_NAME="GOBii-compute-node"

# Port number that will be mapped to compute node's default SSH port. Make 
# sure it's open and doesn't conflict with anything in the host server.
Docker_COMPUTE_SSH_PORT="2222"

# The number of minutes the instruction files age should be before the 
# cronjobs pick them up. This needs to be prefixed by '+', which means 'pick 
# up the files that are greater than n minutes'.
Docker_CRON_FILE_AGE="+2"

# The number of minutes between each cron jobs execution.
Docker_CRON_INTERVAL="2"

# IP/Hostname of the Docker DB host
Docker_DB_HOST="<hostname or IP>"

# Name of the database Docker. This is more of an alias to let you access it 
# conveniently.
Docker_DB_NAME="GOBii-db-node"

# Port number that will be mapped to the DB Docker's 5432 port for postgres 
# connection. Make sure it's open and doesn't conflict with anything in the 
# host server.
Docker_DB_PORT="5433"

# IP/Hostname of the Docker web host
Docker_WEB_HOST="<hostname or IP>"

# Name of the web Docker. This is more of an alias to let you access it 
# conveniently.
Docker_WEB_NAME="GOBii-web-node"

# Port number that will be mapped to port 8080 in the web Docker (the Tomcat 
# default port). Make sure it doesn't conflict with anything in the host.
Docker_WEB_PORT="8081"

# The group ID of the 'GOBii' group in the host machine. The name can be 
# arbitrary, ex. 'icrisat-GOBii', as long as this GID corresponds correctly to 
# it. The 'GOBii' group in all the 3 Dockers will be linked to it.
### Sample command to get it: getent group GOBii (then take the first number) 
# - this depends on your host's OS
# Example: GOBii_GID="1001"
GOBii_GID="<group id>"

# The user ID of the 'gadm' sudoer in the host machine. The name can be 
# arbitrary, ex. 'icrisat-gadm', as long as this UID corresponds correctly to 
# it. The 'gadm' user in all the 3 Dockers will be linked to it.
### Sample command to get it: getent passwd gadm (then take the first number) 
# - this depends on your host's OS
# Example: GOBii_UID="1000"
GOBii_UID="<gadm user id>"

# As of 1.4, this parameter is passed directly, hence this line is ignored. 
# OBSOLETE: File name of the parameter file that will be used for the GOBii 
# instance's configuration once installed. The GOBii-web.xml file will be 
# generated based on the values in this parameter file. 
CONFIGURATOR_PARAM_FILE="*_install.parameters"

# The gadm password inside the Docker containers, you can find the actual 
# password in a restricted page "Defaut Credentials" in GOBii confluence.
Docker_GOBii_ADMIN_PASSWORD="askme"

# Name of the compute Docker repository under GOBii's account (gadm01) you 
# want to pull from.
Docker_HUB_COMPUTE_NAME="GOBii_compute_vanilla_ubuntu"

# Name of the DB Docker repository under GOBii's account (gadm01) you want to 
# pull from.
Docker_HUB_DB_NAME="GOBii_db_vanilla_ubuntu"

# Name of the web Docker repository under GOBii's account (gadm01) you want to 
# pull from.
Docker_HUB_WEB_NAME="GOBii_web_vanilla_ubuntu"

# Username of the Docker hub account you want to use. Change accordingly. If 
# you get permission issues
Docker_HUB_USERNAME="gadm01"

# If set to "askme" then the scripts will prompt for the pass each time or the # pass can be set in parameters and user will not be prompted
Docker_HUB_PASSWORD="askme"

# Name of the sudoer account in the HOST server that the Dockers' gadm account 
# will correspond to (it can be named differently as long as their UIDs match)
Docker_SUDOER_USERNAME="gadm"

# The name of the GOBii application data bundle. Keep the default unless 
# otherwise changed in the Docker images.
Docker_BUNDLE_NAME="GOBii_bundle"

# Internally used by the Dockers. Keep the default unless otherwise changed in 
# the Docker images.
BUNDLE_TEMP_PATH="/var/GOBii_bundle"

# Postgres Volumes Path -- No need to change this, just create a symlink from 
# the Docker volume directory to where you want postgres datafiles to reside. 
# Because as of now, changing these volumes path from here hasn't been tested.
POSTGRES_ETC="GOBiipostgresetcubuntu"
POSTGRES_LOG="GOBiipostgreslogubuntu"
POSTGRES_LIB="GOBiipostgreslibubuntu"

#-----------------------------------------------------------------------------#
### KDCompute parameters
#-----------------------------------------------------------------------------#

# Activate/decativate encryption
ACTIVATE_ENCRYPTION="false"

# IP/Hostname of the Docker web host
Docker_KDC_HOST="<hostname or IP>"

# Docker KDC Node name as it will appear on host
Docker_KDC_NAME="GOBii-kdc-node"

# KDC Image name as it appears in Docker hub
Docker_HUB_KDC_NAME="GOBii_kdc_ubuntu"

# [Deprecated] KDC file storage on host. Assumption is this dir lies with 
# GOBii_parent and accessible via "/data" symlink. This value should match 
# with the param kdcompute.working-directory in kdc-application.properties in 
# kdc Docker image. (Although Deprecated this is being left for historical 
# purposes if ever needed in the future.)
###>>> KDC_FILE_STORAGE_DIR="kdcompute_file_storage"

#-----------------------------------------------------------------------------#
###############################################################################
### !!! NOTICE: Post 1.2 params starts here                                 ###
###############################################################################
#-----------------------------------------------------------------------------#

#-----------------------------------------------------------------------------#
### Liquibase contexts
#-----------------------------------------------------------------------------#

# This handles the migration path and the seed data -- depending on the flavor 
# of GOBii that you are trying to deploy. A quick summary of what these 
# contexts are:

### 1. general = this context contains schema changes (ie. dropped columns, 
# new tables, dropped tables, etc)

### 2. seed_general = this context contains the basic seed data. It is mainly 
# for controlled vocabularies and ontologies, ie. all seed data that needs to 
# exist in all clients' databases.

### 3. seed_crop1 and seed_crop2 = these are example crop-specific seed 
# contexts. You will need to ensure that the context you are using exists. New 
# clients will have specific contexts created for them.

### These contexts contain contact information (ie. list of GOBii users for 
# that instance, along with their usernames, email, and roles). Anything 
# seed-data related that are specific to certain crops goes to this context.
LIQUIBASE_CROP1_CONTEXTS="general,seed_general,seed_crop1"
LIQUIBASE_CROP2_CONTEXTS="general,seed_general,seed_crop2"


#-----------------------------------------------------------------------------#
###############################################################################
### !!! NOTICE: Post 1.5 params starts here                                 ###
###############################################################################
#-----------------------------------------------------------------------------#

#-----------------------------------------------------------------------------#
### Container Hostnames
# This will specify the hostname in cli when working within the container
#-----------------------------------------------------------------------------#

# This will set the hostname within the container
Docker_COMPUTE_CONTAINER_HOSTNAME="compute-node"

# This will set the hostname within the container
Docker_DB_CONTAINER_HOSTNAME="db-node"

# This will set the hostname within the container
Docker_WEB_CONTAINER_HOSTNAME="web-node"

# This will set the hostname within the container
Docker_KDC_CONTAINER_HOSTNAME="kdc-node"

#-----------------------------------------------------------------------------#
### Docker hub user login name
#-----------------------------------------------------------------------------#
# This user is new as of 2.0 and used to login only, the Docker_HUB_USERNAME 
# is still used when doing the Docker run and Docker pull commands
#-----------------------------------------------------------------------------#

Docker_HUB_LOGIN_USERNAME="gadmreader"

#-----------------------------------------------------------------------------#
### KDCompute Version
# KDC has a new build process and the version must be specified for the 
# specific contianer to be deployed
#-----------------------------------------------------------------------------#

# KDC Version
### Example: KDC_RELEASE_VERSION="server_1.6.1-plugin_0.14.2-build_98"
KDC_RELEASE_VERSION="<kdc release version>"

#-----------------------------------------------------------------------------#
### Timescope Configurations
#-----------------------------------------------------------------------------#

# This is a static configuration and shouldn't need to be updated unless 
# another change is made within timescope DB
#TIMESCOPER_USER="timescoper"

# Timescoper password for DB user
# This can be set to "askme" to be prompted for inputed pass
#TIMESCOPER_PASS="<timescoper pass>"

# GDM deployed version
#TIMESCOPER_VERSION="<#>"

#-----------------------------------------------------------------------------#
### ownCloud Container Deployment Parameters
#-----------------------------------------------------------------------------#

# IP/Hostname of the Docker web host
Docker_OC_HOST="<hostname or IP>"

# Docker ownCloud Node name as it will appear on host
Docker_OC_NAME="GOBii-oc-node"

# This will set the hostname within the container
Docker_OC_CONTAINER_HOSTNAME="oc-node"

# ownCloud Image name as it appears in Docker hub
Docker_HUB_OC_NAME="GOBii_oc_vanilla_ubuntu"

# ownCloud web Port for browser UI
Docker_OC_PORT="8084"

# ownCloud Docker Image version
Docker_OC_VERSION='base'

# ownCloud Admin username
OC_ADMIN_USER="admin"

# ownCloud LDAP gadm username 
### Example: OC_GADM_USER="gadm"
#OC_GADM_USER="<gadm username>"

# The password used in LDAP for the gadm user to authenticate and mount 
# directories
OC_GADM_PASSWORD="askme"

#-----------------------------------------------------------------------------#
### ownCloud LDAP Connection Parameters
#-----------------------------------------------------------------------------#

# this is to specify LDAP connection server
### Example: OC_LDAP_SERVER="ldap://<LDAP server name or IP>"
OC_LDAP_SERVER="<ldap host link>"

# The port used for LDAP Connection
OC_LDAP_PORT="389"

# Admin cn name
# Example: OC_LDAP_CN="cn=admin"
OC_LDAP_CN="<admin cn name>"

# gadm LDAP Password; will be prompted for password if set to "askme"
OC_LDAP_CN_PASSWORD="askme"

# LDAP DN
# Example: OC_LDAP_DN="dc=GOBiiproject,dc=org"
OC_LDAP_DN="<DN used for LDAP>"

# Orgizational Unit [if Applicable]
# Example: OC_LDAP_OU="ou=GOBii"
OC_LDAP_OU="ou=<organizational Unit>"

# gadm LDAP uid 
# Example: OC_LDAP_UID="uid=gadm"
OC_LDAP_UID="uid=<gadm ldap username>"

# used for setting the user information for ldap login.
OC_LDAP_LOGIN_FILTER="\"(&(|(objectclass=inetOrgPerson))(uid=%uid))\""

# sets user type from ldap
OC_LDAP_USER_FILTER="\"(|(objectclass=inetOrgPerson))\""

#-----------------------------------------------------------------------------#
### ownCloud users
#-----------------------------------------------------------------------------#
### Users need to be given permissions to each directory mounted 
#-----------------------------------------------------------------------------#

# This is a user on the host that is used to mount the SFTP external storage
### Example: OC_SFTP_USER_1="gadm"
OC_SFTP_USER_1="<ldap user name>"

#-----------------------------------------------------------------------------#
### ownCloud Directory Mount Parameters [Per Crop]
#-----------------------------------------------------------------------------#

### High level directories

# Directory location of GOBii_bundle
OC_SFTP_DIR1="/data/GOBii_bundle"

# Directory location of logs
OC_SFTP_DIR2="/data/GOBii_bundle/logs"

# Directory location of GOBii_bundle
OC_SFTP_DIR3="/data/GOBii_bundle/crops"

### Directory Location Link Name
# This will set the name of the directory link shown in the owncloud UI

# Directory link name of GOBii_bundle
OC_SFTP_DIR1_NAME="GOBii_bundle"

# Directory link name of logs
OC_SFTP_DIR2_NAME="GOBii_bundle/logs"

# Directory link name of GOBii_bundle
OC_SFTP_DIR3_NAME="GOBii_bundle/crops"


#-----------------------------------------------------------------------------#
### GOBii Portal Configurations
#-----------------------------------------------------------------------------#
### Most configurations below are static but if the portal is to be configured
# post deployment they must be added to the parameters as well as the 
# deployment script lest the sysadmin is to update manually
#-----------------------------------------------------------------------------#

# This is a static path within wthe web-node that points to the .xml used for 
# configuring the portal links, location and icons
PORTAL_CONFIG_PATH="/usr/local/tomcat/webapps/GOBii-portal/config/launchers.xml"

# First crop defaulted; xpath points to the block of .xml configuration
### Example: PORTAL_CROP1_URL_XPATH=".//launcher[name='Dev GDM Extractor']/url"
PORTAL_CROP1_URL_XPATH=".//launcher[name='Dev GDM Extractor']/url"

# URL linked to the icon/bookmark configured
### Example: PORTAL_CROP1_URL="http://<Host FQDN or IP>:8081/GOBii-dev/"
PORTAL_CROP1_URL="http://<Host FQDN or IP>:8081/<crop-name>"

# First crop defaulted; xpath points to the block of .xml configuration
### Example: PORTAL_CROP1_NAME_XPATH=""
PORTAL_CROP1_NAME_XPATH=".//launcher[name='Dev GDM Extractor']/name"

# Name shown on the link/bookmark in the portal to define each crop
### Example: PORTAL_CROP1_NAME="dev"
PORTAL_CROP1_NAME="<crop name>"

# Static xpath configuration which should only be changed if the defaulted 
# template from the portal repo is altered
### Example: PORTAL_TIMESCOPE_URL_XPATH=".//launcher[name='Timescope']/url"
PORTAL_TIMESCOPE_URL_XPATH=".//launcher[name='Timescope']/url"

# URL linked to the icon/bookmark configured
### Example: PORTAL_TIMESCOPE_URL="<http://<Host FQDN or IP>:8081/timescope"
PORTAL_TIMESCOPE_URL="http://<Host FQDN or IP>:8081/timescope"

# Static xpath configuration which should only be changed if the defaulted 
# template from the portal repo is altered
### Example: PORTAL_OWNCLOUD_URL_XPATH=".//launcher[name='File Browser']/url"
PORTAL_OWNCLOUD_URL_XPATH=".//launcher[name='File Browser']/url"

# URL linked to the icon/bookmark configured
### Example: PORTAL_OWNCLOUD_URL="<http://<Host FQDN or IP>:8084"
PORTAL_OWNCLOUD_URL="http://<Host FQDN or IP>:8084"

# Static xpath configuration which should only be changed if the defaulted 
# template from the portal repo is altered
### Example: PORTAL_KDC_XPATH=".//launcher[name='KDCompute']/url"
PORTAL_KDC_XPATH=".//launcher[name='KDCompute']/url"

# URL linked to the icon/bookmark configured
### Example: PORTAL_KDC_URL="<http://<Host FQDN or IP>:8083/kdcompute"
PORTAL_KDC_URL="http://<Host FQDN or IP>:8083/kdcompute"

# Static xpath configuration which should only be changed if the defaulted 
# template from the portal repo is altered
### Example: PORTAL_LOADER_URL_XPATH=".//launcher[name='GDM Data Loader']/url"
PORTAL_LOADER_URL_XPATH=".//launcher[name='GDM Data Loader']/url"

# URL linked to the icon/bookmark configured
###>>> This URL is dynamic and will need to be updated for each release as the #URL location will change
###>>> This path links directly to the download ONLY FOR WINDOWS
### Example: PORTAL_LOADER_URL="http://GOBiin1.bti.cornell.edu:6083/projects/LUG/repos/loaderui/raw/distribution/windows/distribution.zip?at=refs%2Fheads%2Frelease%2F2.0"
PORTAL_LOADER_URL="http://GOBiin1.bti.cornell.edu:6083/projects/LUG/repos/loaderui/raw/distribution/windows/distribution.zip?at=refs%2Fheads%2Frelease%2F2.0"

# URL linked to the icon/bookmark configured
### Example: PORTAL_PORTAINER_URL="<http://<Host FQDN or IP>:9000"
PORTAL_PORTAINER_URL="http://<Host FQDN or IP>:9000"

#-----------------------------------------------------------------------------#
### Portainer 
#-----------------------------------------------------------------------------#

# IP/Hostname of the Docker portainer host
Docker_PORTAINER_HOST="<Host FQDN or IP>"

# The container name on the host 
Docker_PORTAINER_NAME="GOBii-portainer-node"

# This will set the hostname within the container
Docker_PORTAINER_CONTAINER_HOSTNAME="portainer-node"

# version of portainer deployed as in portainer/portainer:latest
### Example: Docker_PORTAINER_VERSION="latest"
Docker_PORTAINER_VERSION="<portainer container tag>"

#-----------------------------------------------------------------------------#
### Portainer Sherpa Agent Configurations
#-----------------------------------------------------------------------------#

# IP/Hostname of the Docker portainer host
Docker_SHERPA_HOST="<Host FQDN or IP>"

# container name on the host
Docker_SHERPA_AGENT_NAME="portainer-sherpa-node"

# hostname of the container within the container
Docker_SHERPA_CONTAINER_HOSTNAME="portainer-sherpa-node"

# All allowed networks to access the sherpa agent port
# Make sure to use the format from example and \" to exit each quote
### Example: Docker_SHERPA_NETWORK_RULES="\"10.0.0.0/8\", \"192.168.0.0/16\", \"172.0.0.0/8\""
Docker_SHERPA_NETWORK_RULES="<networks allowed to access sherpa port>"

# The port used to access the container from portainer. 
### Example: Docker_SHERPA_PORT="4550"
Docker_SHERPA_PORT="4550"

#-----------------------------------------------------------------------------#
###############################################################################
### !!! NOTICE: This section is not used unless deploying local OpenLDAP!!! ###
###############################################################################
#-----------------------------------------------------------------------------#
### Local LDAP Parameters (These are only used for configure local OpenLDAP)
#-----------------------------------------------------------------------------#

# LDAP Host which to point and authenticate against
#Docker_LDAP_SERVER="<hostname or IP>"

# Docker LDAP Node name as it will appear on host
###>>> Docker_LDAP_NAME="GOBii_ldap_node"

# This will set the hostname within the container
###>>> Docker_LDAP_CONTAINER_HOSTNAME="ldap-node"

# LDAP Image name as it appears in Docker hub
###>>> Docker_HUB_LDAP_NAME="GOBii_ldap_base_ubuntu"

# The port used for LDAP Connection
#Docker_LDAP_PORT="389"

# LDAP CN Configuration
# Example: Docker_LDAP_CN="cn=admin"
#Docker_LDAP_CN="<admin cn name>"

# LDAP DN Configuration
# Example: Docker_LDAP_DN="dc=GOBiiproject,dc=org"
#Docker_LDAP_DN="dn="

# LDAP Organizational Unit Configuration
# Example: Docker_LDAP_OU="ou=GOBii"
#Docker_LDAP_OU="ou="

# User ID used to configure LDAP and other services [Default: uid=gadm] 
# this is only used if a user is added to a configuration such as ownCloud
#Docker_LDAP_UID="uid=gadm"

# uses image tag to determine which image to pull and run
### Example: Docker_LDAP_VERSION="base"
#Docker_LDAP_VERSION="<tag name>"

You can name this file however you want. The full file path is passed to the deployment script.

For anything not clear or if you're not sure of what to put on a parameter's value, please ask 00d483e952a2545b0152a4b36870000e.

If a seed context for your crop is not available and you would like to have one (ex. seed_crop3), please contact 00d483e961e2874401621ac073490001 or 00d483e952a2545b0152a4b36870000e.

Edit the 'install' parameter file. You can find a template in the repository (GOBiideployment/params/template_install.parameter). It will contain all the GOBii instance's configuration (i.e. runtime configuration via the GOBii-web.xml content). The template is shown below, with each parameter explained on top of the corresponding line:

template_install.parameters

For ease of use the template has been expanded with 'white' space between parameters to allow for a more readable and more easily editable structure.

#-----------------------------------------------------------------------------#
### install.parameters - Used to create GOBiiconfig_wrapper.sh
#-----------------------------------------------------------------------------#
### This parameter file will be used by GOBiiconfig_wrapper.sh to generate a 
# proper GOBii-web.xml -- the main configuration file of a GOBii instance.
### Note that you need to specify this file into the *_main.parameters so that 
# the deployment script will pick it up.
### Also note that all the paths here are on the point of view of the running 
# Dockers, mainly the web Docker, hence the /data prefix on most of them.
#-----------------------------------------------------------------------------#


### The /data directory is the default working directory of all the GOBii 
# Dockers, this is also where the application bundle is located. It is a 
# volume mapped to the shared directory visible to all 3 Dockers.
### The GOBii application data bundle's path in the context of the Docker 
# containers. Keep the default unless otherwise changed in the Docker images.
BUNDLE_PATH="/data/GOBii_bundle"

### The generated GOBii-web.xml path. Keep the default unless otherwise 
# changed in the Docker images.
CONFIG_XML="/data/GOBii_bundle/config/GOBii-web.xml"

### The authentication type. If you want to be able to login using the test 
# user upon installation, set this to "TEST". If you want to immediately 
# connect to LDAP upon installation, set to "LDAP".
### We suggest to set it to "TEST" at first, then verify that everything works 
# by logging into the extractor UI. Then manually set it to "LDAP" in the 
# GOBii-web.xml file, then restart Tomcat and you're all set.
AUTH_TYPE="TEST"

### The LDAP distinguised name
LDAP_DN="uid={0}"

### The LDAP URL
LDAP_URL="ldaps://test.cornell.edu/ou=people,dc=testl,dc=testnet"

### The LDAP Bind User
LDAP_BIND_USER="uid=GOBii-user"

### The LDAP Bind User's password
LDAP_BIND_PASSWORD="dummypass"

### The LDAP background user -- the webservices use this to query for valid users, etc
LDAP_BACKGROUND_USER="GOBii-user"

### The LDAP backround user's password
LDAP_BACKGROUND_PASSWORD="dummypass"

### The mail host. This can be your local mail host or a Gmail SMTP server.
MAIL_HOST="smtp.gmail.com"

### The mail server's port.
MAIL_PORT=587

### The mail user account.
MAIL_USERNAME="GOBii.test@gmail.com"

### The mail user account's password.
MAIL_PASSWORD="dummypass"

### The type of mail server, Gmail defaults to SMTP.
MAIL_TYPE="SMTP"

### The mail server's hash
MAIL_HASH="na"

### The name of the first crop. This should match what is in the Docker hub 
# images.
CROP1="maize"

### The name of the first crop. This should match what is in the Docker hub 
# images.
CROP2="wheat"

### The web host's domain name or IP address
WEB_HOST="172.1.2.3"

### The web host's web server port. This should match what is in the 
# *_main.parameter file.
WEB_PORT="8081"

### The first crop's context path (the web application). This should match 
# what is in the Docker hub images.
CROP1_CONTEXT_PATH="/GOBii-maize"

### The second crop's context path (the web application). This should match 
# what is in the Docker hub images.
CROP2_CONTEXT_PATH="/GOBii-wheat"

### The database host's domain name or IP address
DB_HOST="172.1.2.4"

### The database host's postgres port. This should match what is in the 
# *_main.parameter file.
DB_PORT="5433"

### The main GOBii database user
DB_USERNAME="dummyuser"

### The main GOBii database user's password. If you want to change this 
# password, you can do so once the whole automated deployment finish 
# successfully. But make sure after changing it in postgres, change the 
# corresponding tag in the GOBii-web.xml file.
DB_PASS="dummypass"

### The database name of the first crop. This should match what is in the 
# Docker hub images.
DB_NAME_CROP1="GOBii_maize"

### The database name of the second crop. This should match what is in the 
# Docker hub images.
DB_NAME_CROP2="GOBii_wheat"

#-----------------------------------------------------------------------------#
### KDCompute related params used by GOBiiconfig jar
#-----------------------------------------------------------------------------#

### KDcompute host
KDC_HOST="changeme"

### KDcompute port
KDC_PORT="8083"

### kdcompute application context on server
KDC_CONTEXT_PATH="kdcompute"

### KDcompute job start param
KDC_JOB_START="qcStart"

### KDcompute job status param
KDC_JOB_STATUS="qcStatus"

### KDcompute job downloand
KDC_JOB_DOWNLOAD="qcDownload"

### Seconds to wait between status checks
KDC_JOB_CHECK_STATUS="60"

### Minutes until job is hung
KDC_JOB_FAIL_THRESHOLD="2880"

### KDcompute purge the jobs
KDC_PURGE="qcPurge"

### KDcompute is active (false|true) for GOBii 
KDC_ACTIVE="false"

As of version 1.3, the *install.parameter file is also passed as a parameter to the main call to the GOBii_ship scripts. Hence, it is not necessary to be set in the CONFIGURATOR_PARAM_FILE of the *main.parameter file.

For anything not clear or if you're not sure of what to put on a parameter's value, please ask 00d483e952a2545b0152a4b36870000e.

Running the Deployment Script

Click here to expand...

No Sudo

This script should not be run using sudo or as the root user. Some commands will automatically prompt you if they need elevated permissions.

If you are deploying GOBii into just one machine, you run the_GOBii_ship.sh to pull, deploy, and configure all 3 Docker containers to one target server. To do so, you run a command similar to:

#Usage: bash.sh the_GOBii_ship.sh <path-of-main-param-file> <path-of-install-param-file> <Dockerhubpassw | askme> <GOBii_release_version>
#Set Dockerhubpassw parameter to 'askme' for the script to prompt for password instead.

bash the_GOBii_ship.sh params/template_main_<CG Center File Name>.parameters params/template_install_<CG Center File Name>.parameters askme release-2.0-5

Troubleshooting

If you make a mistake and want to start over or if there are other Dockers in the server you want to get rid of, do a cleanup by running Docker stop, rm, and rmi.

The GOBii Ship...

As of version 2.0 it is only required to run one script.

the_gobii_ship.sh

#!/usr/bin/env bash

set -e # Abort script at first error, when a command exits with non-zero status
set -u # Attempt to use undefined variable outputs error message, and forces 
# an exit
#set -x # xtrace: Similar to -v, but expands commands [to unset and hide 
# passwords us "set +x"]

# set -v # sets verbosity to high echoing commands before executing

#-----------------------------------------------------------------------------#
### !!! WARNING !!!
#-----------------------------------------------------------------------------#
### Do not run this script as SUDO! There are explicit sudo commands which
### will prompt you for password. But not everything should run as sudo.

### usage: bash <bash_script_name>.sh <path of main.parameters> <path of 
# install.parameters> <Dockerhubpassw | "askme"> <GOBii_release_version>

### This a stand-alone equivalent of my THE_GOBii_SHIP Bamboo plan for the 
### ownCloud NODE

#-----------------------------------------------------------------------------#
### Requirements: 

###>>> 1. The user that will run this script needs to be a sudoer and under the GOBii and Docker groups. So preferably the user 'gadm'.

###>>> 2. The working directory needs to be where the GOBiiconfig_wrapper.sh 
# is as well, typically <GOBii_bundle>/conf/
###--->>> NOTE: The order of execution is important.
###--->>> NOTE: If weird things start happening on your containers, try 
# removing the images as well by running 'Docker rmi' on each of the 3 nodes.
#-----------------------------------------------------------------------------#
### If you want a delete-all-images command, run this: 
###>>> [sudo Docker stop $(sudo Docker ps -qa) || true && sudo Docker rm 
# $(sudo Docker ps -aq) || true && sudo Docker rmi $(sudo Docker images -aq) 
# || true]
#-----------------------------------------------------------------------------#

#-----------------------------------------------------------------------------#
#@author: (palace) kdp44@cornell.edu
###>>> Co-Author: RLPetrie (rlp243@cornell.edu)
#-----------------------------------------------------------------------------#


#-----------------------------------------------------------------------------#
### Error on running with sudo
#-----------------------------------------------------------------------------#

if [[ $EUID -eq 0 ]]; then
    echo "This script should not be run using sudo or as the root user. Some commands will automatically prompt you if they need elevated permissions."
    exit 1
fi

if [ $# -lt 3 ];
  then
    echo "No arguments supplied."
    echo "Usage: bash <bash_script_name>.sh <path of main.parameters> <path of install.parameters> <Dockerhubpassw | \"askme\"> <GOBii_release_version>"
    echo "Set Dockerhubpassw parameter to 'askme' for the script to prompt for password instead."
    exit 1
fi

#-----------------------------------------------------------------------------#
### load parameters -- main.parameters for deployment
#-----------------------------------------------------------------------------#

source $1

#-----------------------------------------------------------------------------#
### load parameters -- install.parameters for configuration
#-----------------------------------------------------------------------------#

source $2

#-----------------------------------------------------------------------------#
### Set release version and Dockerhub password
###>>> if pass set to 'askme' call for user to enter password at cli
#-----------------------------------------------------------------------------#

Docker_HUB_PASSWORD=$3


#-----------------------------------------------------------------------------#
### Menu
#-----------------------------------------------------------------------------#

clear

echo "
#------------------------------#
### GOBii Deployment Options ###
#------------------------------#
"
PS3='
Please enter your choice: '

options=(
"Full GDM Deployment [DB, Web, Compute, KDC, ownCloud, Portainer]" 
"GDM + KDC [DB, Web, Compute, KDC]" 
"GDM [DB, Web, Compute]" 
"db-node" 
"web-node" 
"compute-node" 
"KDCompute" 
"ownCloud" 
"Portainer" 
"Sherpa [Portainer Agent]" 
"Quit"
)

select opt in "${options[@]}"


do
        case $opt in

                "Full GDM Deployment [DB, Web, Compute, KDC, ownCloud, Portainer]")
                        echo;
                        if [[ $REPLY == 1 ]]; then

    						echo;

    						echo "You have selected to deploy the full GDM Suite..."
    						echo "[DB, Web, Compute, KDC, ownCloud, Portainer]"
						    echo;

    						read -p "Are you sure you want to deploy the full GDM Suite [y/n]: " SUITE_REPLY
    						echo;

    						if [[ $SUITE_REPLY == 'y' ]]; then

        						GDM_RELEASE_VERSION=$4
						        echo;

						        echo "Main Parameters: $1"
						        echo "Install Parameters: $2"
						        echo "GDM release version: $4"
						        echo;

						        read -p "Please press any key to continue... " -n1 -s
						        echo;


								if [ $Docker_HUB_PASSWORD = "askme" ]; then
    								read -sp "Please enter your Docker Hub password: " Docker_HUB_PASSWORD
								fi

								echo;


						        echo "Shipping DB Container..."
						        echo;

						        bash container_scripts/the_GOBii_ship_db.sh $1 $2 $Docker_HUB_PASSWORD $4
						        echo;

						        echo "The db-node deployment has completed."
						        echo;

						        echo "Moving on to the web-node..."
						        sleep 3
						        echo;

						        bash container_scripts/the_GOBii_ship_web.sh $1 $2 $Docker_HUB_PASSWORD $4
						        echo;

						        echo "The web-node deployment has completed."
						        echo;

						        echo "Moving on to the compute-node..."
						        sleep 3
						        echo;


						        bash container_scripts/the_GOBii_ship_compute.sh $1 $2 $Docker_HUB_PASSWORD $4
						        echo;

						        echo "The compute-node deployment has completed."
						        echo "Moving on to the kdc-node..."
						        sleep 3
						        echo;

						        if [ $KDC_RELEASE_VERSION = "askme" ]; then
    								read -p "Please enter KDCompute release version: " KDC_RELEASE_VERSION
								fi
						        echo;

						        echo "Main Parameters: $1"
						        echo "Install Parameters: $2"
						        echo "KDCompute release version: $KDC_RELEASE_VERSION"
						        echo;

						        bash container_scripts/the_GOBii_ship_kdc.sh $1 $2 $Docker_HUB_PASSWORD $KDC_RELEASE_VERSION
						        echo;

						        echo "The kdc-node deployment has completed."
						        echo;

						        echo "Moving on to the oc-node..."
						        sleep 3
						        echo;

						        bash container_scripts/the_GOBii_ship_oc.sh $1 $2 $Docker_HUB_PASSWORD base
						        echo;

						        echo "The oc-node deployment has completed."
						        echo;

						        echo "Moving on to the Portainer deploy..."
						        sleep 3
						        echo;

						        bash container_scripts/the_GOBii_ship_portainer.sh $1 $2 $Docker_HUB_PASSWORD
						        echo;

						        echo "The portainer-node deployment has completed."
						        echo;


						    else
						        echo "Exiting the_GOBii_ship.sh"
						        echo;
						        break
						    fi
						echo;
						break
						fi
                        ;;

                "GDM + KDC [DB, Web, Compute, KDC]")
                        echo;
                        if [[ $REPLY == 2 ]]; then

    						echo;
    						echo "You have selected to deploy GDM with KDCompute..."
    						echo "[DB, Web, Compute & KDC]"
						    echo;

    						read -p "Are you sure you want to deploy the GDM with KDCompute [y/n]: " GDMK_REPLY
    						echo;

    						if [[ $GDMK_REPLY == 'y' ]]; then

        						GDM_RELEASE_VERSION=$4

						        read -p "Please enter KDC Release Version: " KDC_RELEASE_VERSION
								echo;

						        echo "Main Parameters: $1"
						        echo "Install Parameters: $2"
						        echo "KDCompute release version: $KDC_RELEASE_VERSION"
						        echo "GDM release version: $4"
						        echo;

								if [ $Docker_HUB_PASSWORD = "askme" ]; then
    								read -sp "Please enter your Docker Hub password: " Docker_HUB_PASSWORD
								fi
								echo;

								echo "Shipping DB Container..."
						        echo;

						        bash container_scripts/the_GOBii_ship_db.sh $1 $2 $Docker_HUB_PASSWORD $4
						        echo;

						        echo "The db-node deployment has completed."
						        echo;

						        echo "Moving on to the web-node..."
						        sleep 3
						        echo;

						        bash container_scripts/the_GOBii_ship_web.sh $1 $2 $Docker_HUB_PASSWORD $4
						        echo;

						        echo "The web-node deployment has completed."
						        echo;

						        echo "Moving on to the compute-node..."
						        sleep 3
						        echo;


						        bash container_scripts/the_GOBii_ship_compute.sh $1 $2 $Docker_HUB_PASSWORD $4
						        echo;

						        echo "The compute-node deployment has completed."
						        echo;

						        echo "Moving on to the kdc-node..."
						        sleep 3
						        echo;

						        bash container_scripts/the_GOBii_ship_kdc.sh $1 $2 $Docker_HUB_PASSWORD $KDC_RELEASE_VERSION
						        echo;

						        echo "The web-node deployment has completed."
						        echo;

						    else
						        echo "Exiting the_GOBii_ship.sh"
						        echo;
						        break
						    fi
						echo;
						break
						fi
                        ;;

                "GDM [DB, Web, Compute]")
                        echo;
                        if [[ $REPLY == 3 ]]; then
    						echo;

    						echo "You have selected to deploy base GDM..."
    						echo "[DB, Web & Compute]"
						    echo;

    						read -p "Are you sure you want to deploy the base GDM Containers [y/n]: " GDM_REPLY
    						echo;

    						if [[ $GDM_REPLY == 'y' ]]; then

        						GDM_RELEASE_VERSION=$4
						        echo;

						        echo "Main Parameters: $1"
						        echo "Install Parameters: $2"
						        echo "GDM release version: $4"
						        echo;

						        read -p "Please press any key to continue... " -n1 -s
						        echo;


								if [ $Docker_HUB_PASSWORD = "askme" ]; then
    								read -sp "Please enter your Docker Hub password: " Docker_HUB_PASSWORD
								fi

								echo;


						        echo "Shipping DB Container..."
						        echo;

						        bash container_scripts/the_GOBii_ship_db.sh $1 $2 $Docker_HUB_PASSWORD $4
						        echo;

						        echo "The db-node deployment has completed."
						        echo;

						        echo "Moving on to the web-node..."
						        sleep 3
						        echo;

						        bash container_scripts/the_GOBii_ship_web.sh $1 $2 $Docker_HUB_PASSWORD $4
						        echo;

						        echo "The web-node deployment has completed."
						        echo;

						        echo "Moving on to the compute-node..."
						        sleep 3
						        echo;


						        bash container_scripts/the_GOBii_ship_compute.sh $1 $2 $Docker_HUB_PASSWORD $4
						        echo;

						        echo "The compute-node deployment has completed."
						        echo;

						    else
						        echo "Exiting the_GOBii_ship.sh"
						        echo;

						        break
						    fi
						echo;

						break
						fi
                        ;;

                "db-node")
                        echo;
                        if [[ $REPLY == 4 ]]; then
    						echo;

    						echo "You have selected to deploy db-node..."
						    echo;

    						read -p "Are you sure you want to deploy the DB Container [y/n]: " DB_REPLY
    						echo;

    						if [[ $DB_REPLY == 'y' ]]; then

        						GDM_RELEASE_VERSION=$4
						        echo;

						        echo "Main Parameters: $1"
						        echo "Install Parameters: $2"
						        echo "GDM release version: $4"
						        echo;

						        read -p "Please press any key to continue... " -n1 -s
						        echo;


								if [ $Docker_HUB_PASSWORD = "askme" ]; then
    								read -sp "Please enter your Docker Hub password: " Docker_HUB_PASSWORD
								fi

								echo;

						        echo "Shipping DB Container..."
						        echo;

						        bash container_scripts/the_GOBii_ship_db.sh $1 $2 $Docker_HUB_PASSWORD $4
						        echo;

						    else
						        echo "Exiting the_GOBii_ship.sh"
						        echo;

						        break
						    fi
						echo;

						break
						fi
                        ;;

                "web-node")
                        echo;
                        if [[ $REPLY == 5 ]]; then
    						echo;

    						echo "You have selected to deploy web-node..."
						    echo;

    						read -p "Are you sure you want to deploy the Web Container [y/n]: " WEB_REPLY
    						echo;

    						if [[ $WEB_REPLY == 'y' ]]; then

        						GDM_RELEASE_VERSION=$4
						        echo;

						        echo "Main Parameters: $1"
						        echo "Install Parameters: $2"
						        echo "GDM release version: $4"
						        echo;

						        read -p "Please press any key to continue... " -n1 -s
						        echo;


								if [ $Docker_HUB_PASSWORD = "askme" ]; then
    								read -sp "Please enter your Docker Hub password: " Docker_HUB_PASSWORD
								fi

								echo;

						        echo "Shipping Web Container..."
						        echo;

						        bash container_scripts/the_GOBii_ship_web.sh $1 $2 $Docker_HUB_PASSWORD $4
						        echo;

						    else
						        echo "Exiting the_GOBii_ship.sh"
						        echo;

						        break
						    fi
						echo;

						break
						fi
                        ;;

                "compute-node")
                        echo;
                        if [[ $REPLY == 6 ]]; then
    						echo;

    						echo "You have selected to deploy compute-node..."
						    echo;

    						read -p "Are you sure you want to deploy the Compute Container [y/n]: " COMPUTE_REPLY
    						echo;

    						if [[ $COMPUTE_REPLY == 'y' ]]; then

        						GDM_RELEASE_VERSION=$4
						        echo;

						        echo "Main Parameters: $1"
						        echo "Install Parameters: $2"
						        echo "GDM release version: $4"
						        echo;

						        read -p "Please press any key to continue... " -n1 -s
						        echo;

								if [ $Docker_HUB_PASSWORD = "askme" ]; then
    								read -sp "Please enter your Docker Hub password: " Docker_HUB_PASSWORD
								fi

								echo;					

						        echo "Shipping Compute Container..."
						        echo;

						        bash container_scripts/the_GOBii_ship_compute.sh $1 $2 $Docker_HUB_PASSWORD $4
						        echo;

						    else
						        echo "Exiting the_GOBii_ship.sh"
						        echo;
						        break
						    fi
						echo;
						break
						fi
                        ;;

                "KDCompute")
                        echo;
                        if [[ $REPLY == 7 ]]; then
    						echo;

    						echo "You have selected to deploy kdc-node..."
						    echo;

    						read -p "Are you sure you want to deploy KDCompute Container [y/n]: " KDC_REPLY
    						echo;

    						if [[ $KDC_REPLY == 'y' ]]; then

    							if [ $KDC_RELEASE_VERSION = "askme" ]; then
    								read -p "Please enter KDCompute release version: " KDC_RELEASE_VERSION
						        fi
						        echo;

						        echo "Main Parameters: $1"
						        echo "Install Parameters: $2"
						        echo "KDCompute release version: $KDC_RELEASE_VERSION"
						        echo;

						        read -p "Please press any key to continue... " -n1 -s
						        echo;

								if [ $Docker_HUB_PASSWORD = "askme" ]; then
    								read -sp "Please enter your Docker Hub password: " Docker_HUB_PASSWORD
								fi

								echo;

						        echo "Shipping KDCompute..."
						        echo;

						        bash container_scripts/the_GOBii_ship_kdc.sh $1 $2 $Docker_HUB_PASSWORD $KDC_RELEASE_VERSION
						        echo;

						    else
						        echo "Exiting the_GOBii_ship.sh"
						        echo;

						        break
						    fi
						echo;

						break
						fi
                        ;;

                "ownCloud")
                        echo;
						if [[ $REPLY == 8 ]]; then
    						echo;

    						echo "You have selected to deploy oc-node..."
						    echo;

    						read -p "Are you sure you want to deploy the ownCloud Container [y/n]: " OC_REPLY
    						echo;

    						if [[ $OC_REPLY == 'y' ]]; then

						        echo "Main Parameters: $1"
						        echo "Install Parameters: $2"
						        echo "ownCloud release version: base"
						        echo;

						        read -p "Please press any key to continue... " -n1 -s
						        echo;


								if [ $Docker_HUB_PASSWORD = "askme" ]; then
    								read -sp "Please enter your Docker Hub password: " Docker_HUB_PASSWORD
								fi

								echo;


						        echo "Shipping ownCloud Container..."
						        echo;

						        bash container_scripts/the_GOBii_ship_oc.sh $1 $2 $Docker_HUB_PASSWORD base
						        echo;

						    else
						        echo "Exiting the_GOBii_ship.sh"
						        echo;

						        break
						    fi
						echo;

						break
						fi                
                        ;;

                "Portainer")
                        echo;
                        if [[ $REPLY == 9 ]]; then
    						echo;

    						echo "You have selected to deploy portainer-node..."
						    echo;

    						read -p "Are you sure you want to deploy the Portainer Container [y/n]: " P_REPLY
    						echo;

    						if [[ $P_REPLY == 'y' ]]; then

        						echo "Main Parameters: $1"
						        echo;

						        read -p "Please press any key to continue... " -n1 -s
						        echo;

						        if [ $Docker_HUB_PASSWORD = "askme" ]; then
    								read -sp "Please enter your Docker Hub password: " Docker_HUB_PASSWORD
								fi

								echo "Shipping Portainer..."
						        echo;

						        bash container_scripts/the_GOBii_ship_portainer.sh $1 $2 $Docker_HUB_PASSWORD
						        echo;

						    else
						        echo "Exiting the_GOBii_ship.sh"
						        echo;

						        break
						    fi
						echo;
						break
						fi
                        ;;

                "Sherpa [Portainer Agent]")
                        echo;

                        if [[ $REPLY == 10 ]]; then
    						echo;

    						echo "You have selected to deploy the portainer-sherpa-node..."
						    echo;

    						read -p "Are you sure you want to deploy the Portainer Sherpa Container [y/n]: " S_REPLY
    						echo;

    						if [[ $S_REPLY == 'y' ]]; then

        						echo "Main Parameters: $1"
						        echo;

						        read -p "Please press any key to continue... " -n1 -s
						        echo;

						        if [ $Docker_HUB_PASSWORD = "askme" ]; then
    								read -sp "Please enter your Docker Hub password: " Docker_HUB_PASSWORD
								fi

								echo "Shipping Portainer Sherpa Node..."
						        echo;

						        bash container_scripts/the_GOBii_ship_sherpa_agent.sh $1 $2 $Docker_HUB_PASSWORD
						        echo;

						    else
						        echo "Exiting the_GOBii_ship.sh"
						        echo;

						        break
						    fi
						echo;

						break
						fi
                        ;;

                "Quit")
                        break
                        ;;

                *) echo "invalid option $REPLY";;

        esac

done


echo "Script exited, Good Bye"
echo;

Usability

This deployment script will be updated if new containers need to be deployed along side of GDM but now has the ability to deploy full deployment and call each script in the proper order for deployment or each container individually.

This was updated to make sure the scripts did not need to be maintained in two places when configuration or updates where done to pre-existing containers.

VIM for SysAds

On deployment, vim is now installed on web, db, compute and kdc nodes. This will eventually be built into the Dockerhub containers in the repo but for the time being they will get vim the traditional way.

LDAP Install Cert

After running the deployment scripts and doing the verification step #1 below, turn on LDAP authentication, if it wasn't already (details on verification step #2). When turning on LDAP, make sure that the LDAP certificate is loaded to the JVM. You can do so by:

#########
#run the install cert
#Usage: bash install_cert.sh </data/mycert.der> <ldap_host> </usr/local/lib/keytool> </usr/local/jdk/jre/lib/cacerts> <changeit>
  
Docker exec GOBii-web-node bash -c '
cd /data/GOBii_bundle/config;
bash install_cert.sh /data/cacert_mgs1.der cbsu_mgs1 /usr/lib/jvm/java-8-oracle/bin/keytool /usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts changeit;
'
#########

You should see a confirmation message saying "certificates added to keystore". Finally, restart Tomcat, making sure it runs via user gadm:

Docker exec GOBii-web-node bash -c '
sudo su gadm;
sh /usr/local/tomcat/bin/shutdown.sh;
sh /usr/local/tomcat/bin/startup.sh;
'

As you can see, /data/cacart_mgs1.der is the certificate file. The command above looks for it in the web Docker home volume, which is /data. So wherever that volume points to in the host server, make sure you put the DER file there first before running the command. Lastly, the paths to keytool and cacerts will most likely stay the same as we're only distributing Dockers based on Ubuntu, but in case in the future we offer other linux flavor, or JVM changes, then those paths may change.

Make sure that the ports you assigned the Dockers with (typically 8081, 8083, 8084, 5433, and 2222) are open. Otherwise the containers won't be able to communicate with each other and will fail with internal server error 500 (although in the future we may have more specific error message than this). The more specific error message will be shown in Tomcat's log (catalina.out). Opening a port will differ from OS to OS.

Example: CentOS 6 and 7

$ sudo iptables -I INPUT -p tcp -m tcp --dport <port_number> -j ACCEPT
$ sudo service iptables save

Make sure that the /data symlink in the web node was created for the LoaderUI to work properly. If not, do the following:

/data symlink

sudo ln -sfn $BUNDLE_PARENT_PATH /data

If run in the same terminal session you ran GOBii_ship*.sh scripts, then run this as is, the $BUNDLE_PARENT_PATH variable should be there. If not, replace it with that parameter's value in the *_main.parameters file that you have.

If it is not possible for the /data symlink to be created because in the target host, /data is already a directory or a drive mount point, create a symlink manually from /data/GOBii_bundle to point to $BUNDLE_PARENT_PATH/GOBii_bundle, effectively still making /data/GOBii_bundle point to the correct location.

Restoring backup data

This section details the data restoration process used after a backup and deployment has completed.

Click here to expand...

Steps

Simply run the restore script with the correct parameters: Version 2.0#onScripts-RestoreDataBundlefromBackup

Go to the link above as the syntax in running these scripts have changed slightly from version 1.4.

Verify that the data was restored by opening any crop's ExtractorUI. You should see previously loaded datasets.

OPTIONAL: If you have a KDCompute Docker container running before, restore its files from backup simply by copying the TestOutput_UserDirs subdirectory of kdcompute_file_storage backup to /data/kdcompute_file_storage/TestOutput_UserDirs

[Deprecated] Configure Timescope

This section has been deprecated as the process has been built into the deployment scripts. This section will remain on this version deployment documentation for future deployment references.

Click here to expand...

As of version 1.5, we are adding a new web application called "Timescope". This will allow users to browse and delete data permanently from the database.

With this, there are additional steps that needs to be done – but only needs to be done once (i.e. if you upgrade to any version >1.5 in the future you won't need to do the following anymore).

[Deprecated] Creating Timescope User

This process should not be needed as the 'timescoper' user is already built into the deployed DB. This section is being kept for future references.

Click here to expand...

As of version 1.5, we are adding a new web application called "Timescope". This will allow users to browse and delete data permanently from the database.

With this, there are additional steps that needs to be done – but only needs to be done once (i.e. if you upgrade to any version >1.5 in the future you won't need to do the following anymore).

Create the database user for timescope:
1. SSH into the database node, then go into the database Docker container via:
```
Docker exec -ti <GOBii_db_node> bash
```
2. Switch to the postgres user
  1. su - postgres
3. Create the timescoper db user
  1. create user timescoper with superuser password 'yourpasswordofchoice' valid until 'infinity';

Timescope Configuration

The shell scripts have created a new database user for Timescope, called "timescoper". See Version 2.0. Take note of this. You can either use the default or change the password of this database user.
Configure the Timescope web app to use the new timescoper database user
1. SSH into the web node, then go into the web Docker container and switch t:
  1. Docker exec -ti <GOBii_web_node> bash
2. Modify the timescope config file ( /usr/local/tomcat/webapps/timescope/WEB-INF/classes/config.properties), replacing the credentials with the ones you made in step 1c.
  1. # Timescope credentials #db db.username=timescoper db.pw=<default or yourpassword if you've changed it> #do not change the version parameter version=1.5
3. Reload the webapp via Tomcat Manager
  1. Open your browser and go to <web_node_url>:<web_node_port>, then click on Tomcat Manager
  2. You should see timescope from the link, click "reload":

To verify that Timescope is properly deployed: open your browser and navigate to <web_node_url>:<web_node_port>/timescope. Upon initial install, there will only be one superuser account in your Timescope database. The credentials are in this page: Version 2.0. When you first login, please change this password using the Timescope UI for security. If you cannot access it, contact either (00d483e952a2545b0152a4b36870000e ) or 00d483e961e2874401621ac073490001.

A few things to take note regarding Timescope:

You will need to create accounts (using the User tab) for everyone who needs to access Timescope
- You need to assign temporary passwords for each user and ask them to change it upon login. There is no mandatory password change feature (yet).
Each crop database's user management for Timescope are separate, i.e. you can have one user added to maize but not to wheat, but also if another user needs to be on both, you'll have to add the user manually to both crops
You only need to provision accounts once. Future deployments will always preserve postgres data – as long as the Docker volumes don't get deleted.

Timescope Verification

To verify that Timescope is properly deployed: open your browser and navigate to <web_node_url>:<web_node_port>/timescope. Upon initial install, there will only be one superuser account in your Timescope database. The credentials are in this page: Version 2.0. When you first log in, please change this password using the Timescope UI for security. If you cannot access it, contact either me (00d483e952a2545b0152a4b36870000e ) or 00d483e961e2874401621ac073490001.

A few things to take note regarding Timescope:

You will need to create accounts (using the User tab) for everyone who needs to access Timescope
1. You need to assign temporary passwords for each user and ask them to change it upon login. There is no mandatory password change feature (yet).
Each crop database's user management for Timescope are separate, i.e. you can have one user added to maize but not to wheat, but also if another user needs to be on both, you'll have to add the user manually to both crops
You only need to provision accounts once. Future deployments will always preserve postgres data – as long as the Docker volumes don't get deleted.

GOBii Portal

This section will show the portal that links all products and features with GDM.

Screeshot

Multiple crops and Additional links in GOBii portal

Click here to expand...

The *_main.parameters will need new lines indicating the name of the new crops. These parameters can be anywhere within the *_main.parameters file.

PORTAL_CROP2_NAME="GOBii-<crop2 name>"
PORTAL_CROP3_NAME="GOBii-<crop3 name>"
PORTAL_CROP4_NAME="GOBii-<crop4 name>"
PORTAL_CROP5_NAME="GOBii-<crop5 name>"
PORTAL_CROP6_NAME="GOBii-<crop6 name>"

Location: xml_config_parser.py

/data/GOBii_bundle/config/utils/xml_config_parser.py

Location: launchers.xml

/usr/local/tomcat/webapps/GOBii-portal/config/launchers.xml

Deploying more than one crop or adding links into the portal utilizes /data/GOBii_bundle/config/utils/xml_config_parser.py which changes and updates the /usr/local/tomcat/webapps/GOBii-portal/config/launchers.xml

During deployment the script configures the original crop for crop 1 noted in the parameters file but additional crops and links will need to be added by adding the following to the script or manually running the following commands.

Example below is defaulted in the GOBii web script template for adding portainer to the deployment.

Docker exec -u gadm $Docker_WEB_NAME bash -c "
cd /data/GOBii_bundle/config/utils; python xml_config_parser.py -f /usr/local/tomcat/webapps/GOBii-portal/config/launchers.xml -a \"<launcher> <name>Portainer</name> <url>http://$Docker_PORTAINER_HOST:9000</url> <logo>portainer.png</logo> <description>Portainer</description> <color>color-green</color> <category>Container Management</category> <type>Web Apps</type> <documentationList /> </launcher>\" -x . ;
";

Manual Update...

If needed, the configurations need to be changed and scripts are erroring you can add the above configurations manually. The webpage will update dynamically.

ownCloud

This section will show the setup and configuration required post deployment. This section assumes the container was deployed but the LDAP, Storage and Shares have not been configured.

Click here to expand...

After ownCloud deployment log in with ownCloud default user and pass. This user and pass will have to be updated by the deploying system administrator as the user and password are stored and salted in the DB.

Default ownCloud user/pass

admin/admin

Once logged in, select user name "Admin" > "Settings" > On left panel under Admin select "User Authentication". The configuration on "Server " tab will show the configurations made in the *_main.parameters file. If the configurations were correct at deployment at the bottom will show "".

Screenshot

If the below screenshot shows "" instead, update the configurations within this tab until it shows OK for your authentication configuration.

LDAP Certificates

If using a certificate, the configuration will show "OK" once the properly setup but will fail to return any users or groups. Though within the "Login Attributes" tab, a username can be verified even without the certificate, but this is the extent until the certificate is added to the container.

On deployment, the /data directory is mounted to the ownCloud container. Place the certificate anywhere within /data then copy to the /var/www/owncloud directory. The system should pick this up on the next attempt to authenticate.

ownCloud Active Directory Configuration

ownCloud works well with LDAP but needs additonal settings configured for systems using Active Directory. Within the "Expert" tab at the right settings for Internal Username and UUID may need to be updated.

Screenshot

ownCloud Storage

The scripts should configure the storage as shown in the "Storage" link in the left panel allowing the Admin user to share these external directories. If the script deployment did not which has been the case in errors of missed or wrong parameters configure the external storage using the following:

Select "Enable external storage"
Enter the folder name for logs and crops
Select sftp
Select username and password
Under "Configuration"
- <hostname or IP>
- directory location on host [ie. /data/GOBii_bundle/logs]
- gadm username
- gadm password
Within "Available for" if there is no user or group added, these mounts are available for all users.
- GOBii suggested configuration is to add only local owncloud "admin" user to lock down the access of these to only the admin user.

Verify under the gear icon the "Enable Preview" and "Enable Sharing" are checked

Screenshot

Sharing External Storage with Users

Find the directories in admin home > select ellipsis > select "Details"

Screenshot

Select "Sharing" > Under User and Groups enter the "GOBii" group and select

Screenshot

Select the down arrow > uncheck
- can share
- can edit
- create change delete

This will allow the GOBii group to see and use the files and directories shared but will be unable to edit or change them.

Screenshot

Enabling File Scan

Add the following line into the root cronjobs within the ownCloud container. The files available when shared will not be updated after unless the following line is added to preform an ownCloud file scan to pick up the new files.

[cronjob] owncloud file scan

*/5 * * * *  occ files:scan --all >> /var/log/cron.log 2>&1

Portainer

Portainer is a container that sits on a system and monitors all Docker/container information. This system can monitor multiple end points by deploying the sherpa container opening any specific port. This allows for the portainer container to access and monitor all containers on a remote system.

Screenshot

Portainer Initial Login

On the initial login, portainer will request the admin to setup the password.

Portainer Configuration Files

Portainer holds configurations under the /data directory. If the system is removed and redeployed the same configurations will remain as the portainer files within /data are not removed.

Screenshot

Select "Local" > Select "Connect"

this will allow for local monitoring and allow for adding remote endpoints to be monitored post deployment.

Screenshot

Adding Sherpa Agent Node

Select "Endpoints" in left panel > Select "Add endpoint"

Screenshot

Add the configurations for the sherpa node under "Environment Details":

Name
Enpoint URL
Public IP

Screenshot

Version Error [Ubuntu]

During testing of portainer, the latest has problems adding endpoints and would fail to add with a very undescriptive error. This error only occurred when attempting to connect ubuntu 16.04 server VMs together with latest portainer and latest sherpa on both of these versions of portainer/sherpa and Ubuntu 16.04 OS on a VM. This error was not seen between these:

CentOS to CentOS
Ubuntu to CentOS
CentOS to Ubuntu

Deploying Sherpa Agent Container

Click here to expand...

Sherpa opens the contain port for external access but is limited to the specified networks setup in the parameters:

# All allowed networks to access the sherpa agent port
# Make sure to use the format from example and \" to exit each quote
### Example: Docker_SHERPA_NETWORK_RULES="\"10.0.0.0/8\", \"192.168.0.0/16\", \"172.0.0.0/8\""
Docker_SHERPA_NETWORK_RULES="<networks allowed to access sherpa port>"

The portainer container will be unable to monitor the remote host unless the communication to specified port is stated.

Deploying Sherpa via GOBii scripts

Verify all parameters are updated for the sherpa agent

To deploy the the_GOBii_ship.sh and select the sherpa agent.

Deploying Sherpa manually

Using the configurations specific to the environment, run the following command to deploy the sherpa remote agent container: [Settings below are defaulted for local access]
- Network rules syntax: 10.0.0.0/24
- Port: This can be any port. Portainer defaults to 2375 and GOBii normally uses 4550

Docker run -d \
--name sherpa_portainer-agent \
-h portainer-agent-node \
-e CONFIG='[ 
    { 
        "Path" : "/", 
        "Access": "allow", 
        "Addresses": [<network rules see above>]
    }
]' \
-v /var/run/Docker.sock:/tmp/Docker.sock \
-p <port>:<port> \
djenriquez/sherpa --allow

Docker start sherpa_portainer-agent

Post Deployment Verification [Smoke Testing]

This section is large enough, therefore it warrants its own document. Please follow the link below to get the version deployment Smoke Testing documentation.

Version 2.0