/
Version 2.1

Version 2.1

Owned by Yaw Nti-Addae
Last updated: Apr 13, 2020 by Monica Franciscus (Unlicensed)

You can find the release notes for every releases of GOBii in this page: System Requirements and Release Notes#menu-link-content

 

Versioning

When using this document, make sure that you are deploying the correct GOBii version number. The official build string for GOBii version 2.0 is below (a parameter string you need to run the shell scripts):

release-2.1

Operating System, bash & Docker Versions

The following are the versions used when developing and testing within GOBii

Operating Systems:

  • Ubuntu 16.04.5 LTS

  • CentOS Linux release 7.2.1511 (Core)

Bash Version:

  • Ubuntu: GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)

  • CentOS: GNU bash, version 4.2.46(1)-release (x86_64-redhat-linux-gnu)

Docker Version:

  • Ubuntu: Docker version 18.06.1-ce, build e68fc7a

  • CentOS: Docker version 18.03.1-ce, build 9ee9f40

git Version

  • Ubuntu: git version 2.7.4

  • CentOS: git version 1.8.3.1

GDM Deployment Versions

This shows the Docker versions used for deployment of this release:

  • GDM: release-2.1

  • KDC: server_1.6.1-plugin_0.14.3-build_110

  • ownCloud: base

  • Portainer: latest

  • sherpa: latest

For any questions or clarifications, please contact Kevin Palis or Roy Petrie. 



Introduction

In this section the Definition of Terms, Background and a brief overview of GOBii is described

Definition of Terms

  • Nodes = GOBii Nodes

    • The term "nodes" here will always refer to the GOBii nodes, which are ‘docker containers’ that can be deployed to different servers or virtual environments. Server nodes, on the other hand, will be explicitly called "server node".

Background

GOBii is made up of multiple modules and categorized according to functions.  A system diagram that shows these categories (by Docker container), the data flow, and the modules is available here.

Depending on your server topology, the instructions on this page may require some tweaking. For each sections with significant differences on steps, depending on server topology, a "Note Box" like the one below will be written.


GOBii's deployment architecture is flexible and node-based. There are three main nodes: computation, database, and web. These nodes are now pre-baked into Docker images and can be deployed in their own server, VM, or in any combination of servers and virtual environments.

To give you an idea, here's an example topology and node-distribution:

Server 1: Server Head: GOBII Test (all nodes) Server Node1: GOBII Prod Database Node Server Node2: GOBII Prod Web Node Server Node3: GOBII Prod Compute Node


You can put GOBii nodes of the same GOBii instance into one server, but we advise against mixing nodes of different GOBii instances into one server. Aside from competing for resources, there are potential conflict points that nodes from different instances may run into.

 



Initial Installation Prerequisites

 

1.The official repository for the deployment scripts is here. Make sure you clone or download the scripts from there. The branch you should get is release/<version> (ex. release/2.1). You can also get the master branch if you are deploying the latest, but because our clients can have varying versions on different servers, all release branches are kept.

2.Finalize your topology and write it down. Because if you are deploying all 3 GOBii nodes to just one server, you run a different script as opposed to when you deploy GOBii into one server per node or any other variations (you run 3 scripts).

3.The servers should have the docker engine version 17 and up installed. Make sure the servers have access to the dockerhub site.

-Ubuntu: https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/#upgrade-docker-ce-1

-CentOS: https://docs.docker.com/install/linux/docker-ce/centos/

-A mount point or a shared drive that all the nodes can access — this will be a volume mounted to all the 3 docker containers.

4.The user that will run the scripts needs to be a sudoer and under the ‘gobii’ and ‘docker groups’. So preferably the user ‘gadm'. But the username is arbitrary, it just needs to be consistent. You may find 'gadm sudoer’ used in the rest of this document, just note that the name is flexible.



sudo usermod -aG docker gadm



  1. (Optional) A directory where the data of postgres will reside. The default will be Ubuntu's postgres directory in the DB docker (ex. /var/lib/pgsql/data) which will be linked to Docker's default volume directory (ex. /usr/local/docker/volumes/postgreslibubuntu)

For a test GOBii instance, you can use the vanilla version of the dockers:









Non-Destructive Deployment [NDD]

This section is for showing the non-destructive deployment architecture.



This architecture needs to be implemented prior to any deployment post version 2.0



NDD Diagram

This is a diagram that represents the current architecture of the directories and the symlink





In this architecture the /data/gobii_bundle directory will be destroyed during a new deployment and replaced with the latest version. Due to the persistent data being symlinks these directories are saved in the /storage/persistent_data directory.











Backups

This section is for existing instances that already have data in storage. This will show the process used to back up the existing data.


Release 2.1+

During release 2.1 the system has been developed for non-destructive deployments using symlinks to access data. Though it is still very important to perform data backups the restore process described later in this document has been deprecated.

 

  1. Back up the data files from the existing GOBii instance. You can do so by using the backup script we provide. You can find usage instructions here: GOBII Add-on Scripts#onScripts-IncrementalDataBundleBackups. Ideally, you already have this setup to do incremental backups (as a cron job), so that prior to the upgrade you just need to run the script in incremental mode to make sure you capture all changes, then disable access to the system to make sure nobody loads new data while the upgrade is in progress. If that's not the case, you can run the script on full backup mode – just note that this may take several hours depending on the size of your data.

  2. Back up the database (postgres) from the existing GOBii instance. You have two options:

    1. Run the script we provide (GOBII Add-on Scripts#onScripts-PostgresqlRotatingIncrementalBackup) - this also is ideally set up as a cron job.

    2. Manually back up the database

      1. Go into the database node and run pg_dumpall

        $> docker exec -ti <gobii_db_node> bash $> su postgres $> pg_dumpall > /data/all_databases.bak



    3. The data from postgres actually persists as long as you don't delete the Docker volumes. However, we still recommend you back up the database to make sure we have redundancy.



  3. OPTIONAL: If you have KDCompute Docker container running, back up its files by copying the directory /data/kdcompute_file_storage to a directory of your choice. This contains previous output and logs of QC jobs.







Deployment

This section will detail the scripts, parameters and process to deploy GOBii

Deployment Scripts and Parameters

Copy the deployment scripts and files from the cloned repository (prerequisite #1) to the shared drive (prerequisite #4).

We update the param files from time to time (i.e., new features being added), so please don't just copy-paste the sample param files below. They are just shown in this page for reference. Instead, pull from our deployment scripts git repository for a particular release you are deploying (ex. release/1.5).

The templates shown below were last updated for version 2.0



Edit the main parameter file. You can find a template in the repository (gobiideployment/params/template_main.parameter). It will contain all the topology information and deployment credentials. The template is shown below, with each parameter explained on top of the corresponding line: 



All the passwords and some usernames have been omitted on the parameter file templates in this page for security. Make sure you check Default Credentials [CONFIDENTIAL] to replace the parameters with the correct values. If you can't access the page with the default credentials, contact Kevin Palis or Roy Petrie.



Version 2.0+; dockerhub access

As of version 2.0 the container repos still exist under the user gadm01 but you cannot uploaded to it. This was done for security and maintenance purposes.

Please use gadmreader to pull an images from the gadm01 account.



template_main.parameters

This template has been updated with the most recent parameters as of Version 2.0. For ease of use, the template has been expanded with 'white' space between parameters to allow for a more readable and more easily editable structure.

Additionally, this file has taken the most recent parameters added at the bottom for any version deployment beyond the last so as to allow for easy copy and paste into existing parameter files.

As of version 2.0 any password set within the *main.parameters file set to "askme" have been configured to request the user password during script deployment. The request and password is hidden during deployment to keep the visibility and clear text passwords to a minimum.

If a password is set, the script will continue without prompting for the user pass.



#-----------------------------------------------------------------------------# ### load parameters -- install.parameters for configuration #-----------------------------------------------------------------------------# ### This file will be used by the_gobii_ship_*.sh to deploy and configure the docker images to target hosts. #-----------------------------------------------------------------------------# # This is your shared directory (will be mounted as volume to the dockers), it # needs to be accessible from the 3 dockers. ### Example: BUNDLE_PARENT_PATH="/storage/dev_test" BUNDLE_PARENT_PATH="<bundle parent path>" # This is the path of the data kept separated from gobii_bundle and symlinked # to ### Example: NDD_PATH="/storage/persistent_data" NDD_PATH="</path/to/persistent_data>" # The name of the first crop - this needs to match what is in the provisioned # docker image in our docker hub. ### Example: DOCKER_CROP1_NAME="dev" DOCKER_CROP1_NAME="<crop name>" # The name of the second crop - this needs to match what is in the provisioned # docker image in our docker hub. # DOCKER_CROP2_NAME="wheat" # IP/Hostname of the docker compute host DOCKER_COMPUTE_HOST="<hostname or IP>" # Name of the compute docker. This is more of an alias to let you access it # conveniently. DOCKER_COMPUTE_NAME="gobii-compute-node" # Port number that will be mapped to compute node's default SSH port. Make # sure it's open and doesn't conflict with anything in the host server. DOCKER_COMPUTE_SSH_PORT="2222" # The number of minutes the instruction files age should be before the # cronjobs pick them up. This needs to be prefixed by '+', which means 'pick # up the files that are greater than n minutes'. DOCKER_CRON_FILE_AGE="+2" # The number of minutes between each cron jobs execution. DOCKER_CRON_INTERVAL="2" # IP/Hostname of the docker DB host DOCKER_DB_HOST="<hostname or IP>" # Name of the database docker. This is more of an alias to let you access it # conveniently. DOCKER_DB_NAME="gobii-db-node" # Port number that will be mapped to the DB docker's 5432 port for postgres # connection. Make sure it's open and doesn't conflict with anything in the # host server. DOCKER_DB_PORT="5433" # IP/Hostname of the docker web host DOCKER_WEB_HOST="<hostname or IP>" # Name of the web docker. This is more of an alias to let you access it # conveniently. DOCKER_WEB_NAME="gobii-web-node" # Port number that will be mapped to port 8080 in the web docker (the Tomcat # default port). Make sure it doesn't conflict with anything in the host. DOCKER_WEB_PORT="8081" # The group ID of the 'gobii' group in the host machine. The name can be # arbitrary, ex. 'icrisat-gobii', as long as this GID corresponds correctly to # it. The 'gobii' group in all the 3 dockers will be linked to it. ### Sample command to get it: getent group gobii (then take the first number) # - this depends on your host's OS # Example: GOBII_GID="1001" GOBII_GID="<group id>" # The user ID of the 'gadm' sudoer in the host machine. The name can be # arbitrary, ex. 'icrisat-gadm', as long as this UID corresponds correctly to # it. The 'gadm' user in all the 3 dockers will be linked to it. ### Sample command to get it: getent passwd gadm (then take the first number) # - this depends on your host's OS # Example: GOBII_UID="1000" GOBII_UID="<gadm user id>" # As of 1.4, this parameter is passed directly, hence this line is ignored. # OBSOLETE: File name of the parameter file that will be used for the GOBII # instance's configuration once installed. The gobii-web.xml file will be # generated based on the values in this parameter file. CONFIGURATOR_PARAM_FILE="*_install.parameters" # The gadm password inside the docker containers, you can find the actual # password in a restricted page "Defaut Credentials" in GOBii confluence. DOCKER_GOBII_ADMIN_PASSWORD="askme" # Name of the compute docker repository under GOBII's account (gadm01) you # want to pull from. DOCKER_HUB_COMPUTE_NAME="gobii_compute_vanilla_ubuntu" # Name of the DB docker repository under GOBII's account (gadm01) you want to # pull from. DOCKER_HUB_DB_NAME="gobii_db_vanilla_ubuntu" # Name of the web docker repository under GOBII's account (gadm01) you want to # pull from. DOCKER_HUB_WEB_NAME="gobii_web_vanilla_ubuntu" # Username of the docker hub account you want to use. Change accordingly. If # you get permission issues DOCKER_HUB_USERNAME="gadm01" # If set to "askme" then the scripts will prompt for the pass each time or the # pass can be set in parameters and user will not be prompted DOCKER_HUB_PASSWORD="askme" # Name of the sudoer account in the HOST server that the dockers' gadm account # will correspond to (it can be named differently as long as their UIDs match) DOCKER_SUDOER_USERNAME="gadm" # The name of the GOBII application data bundle. Keep the default unless # otherwise changed in the docker images. DOCKER_BUNDLE_NAME="gobii_bundle" # Internally used by the dockers. Keep the default unless otherwise changed in # the docker images. BUNDLE_TEMP_PATH="/var/gobii_bundle" # Postgres Volumes Path -- No need to change this, just create a symlink from # the docker volume directory to where you want postgres datafiles to reside. # Because as of now, changing these volumes path from here hasn't been tested. POSTGRES_ETC="gobiipostgresetcubuntu" POSTGRES_LOG="gobiipostgreslogubuntu" POSTGRES_LIB="gobiipostgreslibubuntu" #-----------------------------------------------------------------------------# ### KDCompute parameters #-----------------------------------------------------------------------------# # Activate/decativate encryption ACTIVATE_ENCRYPTION="false" # IP/Hostname of the docker web host DOCKER_KDC_HOST="<hostname or IP>" # Docker KDC Node name as it will appear on host DOCKER_KDC_NAME="gobii-kdc-node" # KDC Image name as it appears in docker hub DOCKER_HUB_KDC_NAME="gobii_kdc_ubuntu" # [Deprecated] KDC file storage on host. Assumption is this dir lies with # gobii_parent and accessible via "/data" symlink. This value should match # with the param kdcompute.working-directory in kdc-application.properties in # kdc docker image. (Although Deprecated this is being left for historical # purposes if ever needed in the future.) ###>>> KDC_FILE_STORAGE_DIR="kdcompute_file_storage" #-----------------------------------------------------------------------------# ############################################################################### ### !!! NOTICE: Post 1.2 params starts here ### ############################################################################### #-----------------------------------------------------------------------------# #-----------------------------------------------------------------------------# ### Liquibase contexts #-----------------------------------------------------------------------------# # This handles the migration path and the seed data -- depending on the flavor # of GOBii that you are trying to deploy. A quick summary of what these # contexts are: ### 1. general = this context contains schema changes (ie. dropped columns, # new tables, dropped tables, etc) ### 2. seed_general = this context contains the basic seed data. It is mainly # for controlled vocabularies and ontologies, ie. all seed data that needs to # exist in all clients' databases. ### 3. seed_crop1 and seed_crop2 = these are example crop-specific seed # contexts. You will need to ensure that the context you are using exists. New # clients will have specific contexts created for them. ### These contexts contain contact information (ie. list of GOBii users for # that instance, along with their usernames, email, and roles). Anything # seed-data related that are specific to certain crops goes to this context. LIQUIBASE_CROP1_CONTEXTS="general,seed_general,seed_crop1" LIQUIBASE_CROP2_CONTEXTS="general,seed_general,seed_crop2" #-----------------------------------------------------------------------------# ############################################################################### ### !!! NOTICE: Post 1.5 params starts here ### ############################################################################### #-----------------------------------------------------------------------------# #-----------------------------------------------------------------------------# ### Container Hostnames # This will specify the hostname in cli when working within the container #-----------------------------------------------------------------------------# # This will set the hostname within the container DOCKER_COMPUTE_CONTAINER_HOSTNAME="compute-node" # This will set the hostname within the container DOCKER_DB_CONTAINER_HOSTNAME="db-node" # This will set the hostname within the container DOCKER_WEB_CONTAINER_HOSTNAME="web-node" # This will set the hostname within the container DOCKER_KDC_CONTAINER_HOSTNAME="kdc-node" #-----------------------------------------------------------------------------# ### Docker hub user login name #-----------------------------------------------------------------------------# # This user is new as of 2.0 and used to login only, the DOCKER_HUB_USERNAME # is still used when doing the docker run and docker pull commands #-----------------------------------------------------------------------------# DOCKER_HUB_LOGIN_USERNAME="gadmreader" #-----------------------------------------------------------------------------# ### KDCompute Version # KDC has a new build process and the version must be specified for the # specific contianer to be deployed #-----------------------------------------------------------------------------# # KDC Version ### Example: KDC_RELEASE_VERSION="server_1.6.1-plugin_0.14.2-build_98" KDC_RELEASE_VERSION="<kdc release version>" #-----------------------------------------------------------------------------# ### Timescope Configurations #-----------------------------------------------------------------------------# # This is a static configuration and shouldn't need to be updated unless # another change is made within timescope DB #TIMESCOPER_USER="timescoper" # Timescoper password for DB user # This can be set to "askme" to be prompted for inputed pass #TIMESCOPER_PASS="<timescoper pass>" # GDM deployed version #TIMESCOPER_VERSION="<#>" #-----------------------------------------------------------------------------# ### ownCloud Container Deployment Parameters #-----------------------------------------------------------------------------# # IP/Hostname of the docker web host DOCKER_OC_HOST="<hostname or IP>" # Docker ownCloud Node name as it will appear on host DOCKER_OC_NAME="gobii-oc-node" # This will set the hostname within the container DOCKER_OC_CONTAINER_HOSTNAME="oc-node" # ownCloud Image name as it appears in docker hub DOCKER_HUB_OC_NAME="gobii_oc_vanilla_ubuntu" # ownCloud web Port for browser UI DOCKER_OC_PORT="8084" # ownCloud Docker Image version DOCKER_OC_VERSION='base' # ownCloud Admin username OC_ADMIN_USER="admin" # ownCloud LDAP gadm username ### Example: OC_GADM_USER="gadm" #OC_GADM_USER="<gadm username>" # The password used in LDAP for the gadm user to authenticate and mount # directories OC_GADM_PASSWORD="askme" #-----------------------------------------------------------------------------# ### ownCloud LDAP Connection Parameters #-----------------------------------------------------------------------------# # this is to specify LDAP connection server ### Example: OC_LDAP_SERVER="ldap://<LDAP server name or IP>" OC_LDAP_SERVER="<ldap host link>" # The port used for LDAP Connection OC_LDAP_PORT="389" # Admin cn name # Example: OC_LDAP_CN="cn=admin" OC_LDAP_CN="<admin cn name>" # gadm LDAP Password; will be prompted for password if set to "askme" OC_LDAP_CN_PASSWORD="askme" # LDAP DN # Example: OC_LDAP_DN="dc=gobiiproject,dc=org" OC_LDAP_DN="<DN used for LDAP>" # Orgizational Unit [if Applicable] # Example: OC_LDAP_OU="ou=gobii" OC_LDAP_OU="ou=<organizational Unit>" # gadm LDAP uid # Example: OC_LDAP_UID="uid=gadm" OC_LDAP_UID="uid=<gadm ldap username>" # used for setting the user information for ldap login. OC_LDAP_LOGIN_FILTER="\"(&(|(objectclass=inetOrgPerson))(uid=%uid))\"" # sets user type from ldap OC_LDAP_USER_FILTER="\"(|(objectclass=inetOrgPerson))\"" #-----------------------------------------------------------------------------# ### ownCloud users #-----------------------------------------------------------------------------# ### Users need to be given permissions to each directory mounted #-----------------------------------------------------------------------------# # This is a user on the host that is used to mount the SFTP external storage ### Example: OC_SFTP_USER_1="gadm" OC_SFTP_USER_1="<ldap user name>" #-----------------------------------------------------------------------------# ### ownCloud Directory Mount Parameters [Per Crop] #-----------------------------------------------------------------------------# ### High level directories # Directory location of gobii_bundle OC_SFTP_DIR1="/data/gobii_bundle" # Directory location of logs OC_SFTP_DIR2="/data/gobii_bundle/logs" # Directory location of gobii_bundle OC_SFTP_DIR3="/data/gobii_bundle/crops" ### Directory Location Link Name # This will set the name of the directory link shown in the owncloud UI # Directory link name of gobii_bundle OC_SFTP_DIR1_NAME="gobii_bundle" # Directory link name of logs OC_SFTP_DIR2_NAME="gobii_bundle/logs" # Directory link name of gobii_bundle OC_SFTP_DIR3_NAME="gobii_bundle/crops" #-----------------------------------------------------------------------------# ### GOBii Portal Configurations #-----------------------------------------------------------------------------# ### Most configurations below are static but if the portal is to be configured # post deployment they must be added to the parameters as well as the # deployment script lest the sysadmin is to update manually #-----------------------------------------------------------------------------# # This is a static path within wthe web-node that points to the .xml used for # configuring the portal links, location and icons PORTAL_CONFIG_PATH="/usr/local/tomcat/webapps/gobii-portal/config/launchers.xml" # First crop defaulted; xpath points to the block of .xml configuration ### Example: PORTAL_CROP1_URL_XPATH=".//launcher[name='Dev GDM Extractor']/url" PORTAL_CROP1_URL_XPATH=".//launcher[name='Dev GDM Extractor']/url" # URL linked to the icon/bookmark configured ### Example: PORTAL_CROP1_URL="http://<Host FQDN or IP>:8081/gobii-dev/" PORTAL_CROP1_URL="http://<Host FQDN or IP>:8081/<crop-name>" # First crop defaulted; xpath points to the block of .xml configuration ### Example: PORTAL_CROP1_NAME_XPATH="" PORTAL_CROP1_NAME_XPATH=".//launcher[name='Dev GDM Extractor']/name" # Name shown on the link/bookmark in the portal to define each crop ### Example: PORTAL_CROP1_NAME="dev" PORTAL_CROP1_NAME="<crop name>" # Static xpath configuration which should only be changed if the defaulted # template from the portal repo is altered ### Example: PORTAL_TIMESCOPE_URL_XPATH=".//launcher[name='Timescope']/url" PORTAL_TIMESCOPE_URL_XPATH=".//launcher[name='Timescope']/url" # URL linked to the icon/bookmark configured ### Example: PORTAL_TIMESCOPE_URL="<http://<Host FQDN or IP>:8081/timescope" PORTAL_TIMESCOPE_URL="http://<Host FQDN or IP>:8081/timescope" # Static xpath configuration which should only be changed if the defaulted # template from the portal repo is altered ### Example: PORTAL_OWNCLOUD_URL_XPATH=".//launcher[name='File Browser']/url" PORTAL_OWNCLOUD_URL_XPATH=".//launcher[name='File Browser']/url" # URL linked to the icon/bookmark configured ### Example: PORTAL_OWNCLOUD_URL="<http://<Host FQDN or IP>:8084" PORTAL_OWNCLOUD_URL="http://<Host FQDN or IP>:8084" # Static xpath configuration which should only be changed if the defaulted # template from the portal repo is altered ### Example: PORTAL_KDC_XPATH=".//launcher[name='KDCompute']/url" PORTAL_KDC_XPATH=".//launcher[name='KDCompute']/url" # URL linked to the icon/bookmark configured ### Example: PORTAL_KDC_URL="<http://<Host FQDN or IP>:8083/kdcompute" PORTAL_KDC_URL="http://<Host FQDN or IP>:8083/kdcompute" # Static xpath configuration which should only be changed if the defaulted # template from the portal repo is altered ### Example: PORTAL_LOADER_URL_XPATH=".//launcher[name='GDM Data Loader']/url" PORTAL_LOADER_URL_XPATH=".//launcher[name='GDM Data Loader']/url" # URL linked to the icon/bookmark configured ###>>> This URL is dynamic and will need to be updated for each release as the #URL location will change ###>>> This path links directly to the download ONLY FOR WINDOWS ### Example: PORTAL_LOADER_URL="http://gobiin1.bti.cornell.edu:6083/projects/LUG/repos/loaderui/raw/distribution/windows/distribution.zip?at=refs%2Fheads%2Frelease%2F2.0" PORTAL_LOADER_URL="http://gobiin1.bti.cornell.edu:6083/projects/LUG/repos/loaderui/raw/distribution/windows/distribution.zip?at=refs%2Fheads%2Frelease%2F2.0" # URL linked to the icon/bookmark configured ### Example: PORTAL_PORTAINER_URL="<http://<Host FQDN or IP>:9000" PORTAL_PORTAINER_URL="http://<Host FQDN or IP>:9000" #-----------------------------------------------------------------------------# ### Portainer #-----------------------------------------------------------------------------# # IP/Hostname of the docker portainer host DOCKER_PORTAINER_HOST="<Host FQDN or IP>" # The container name on the host DOCKER_PORTAINER_NAME="gobii-portainer-node" # This will set the hostname within the container DOCKER_PORTAINER_CONTAINER_HOSTNAME="portainer-node" # version of portainer deployed as in portainer/portainer:latest ### Example: DOCKER_PORTAINER_VERSION="latest" DOCKER_PORTAINER_VERSION="<portainer container tag>" #-----------------------------------------------------------------------------# ### Portainer Sherpa Agent Configurations #-----------------------------------------------------------------------------# # IP/Hostname of the docker portainer host DOCKER_SHERPA_HOST="<Host FQDN or IP>" # container name on the host DOCKER_SHERPA_AGENT_NAME="portainer-sherpa-node" # hostname of the container within the container DOCKER_SHERPA_CONTAINER_HOSTNAME="portainer-sherpa-node" # All allowed networks to access the sherpa agent port # Make sure to use the format from example and \" to exit each quote ### Example: DOCKER_SHERPA_NETWORK_RULES="\"10.0.0.0/8\", \"192.168.0.0/16\", \"172.0.0.0/8\"" DOCKER_SHERPA_NETWORK_RULES="<networks allowed to access sherpa port>" # The port used to access the container from portainer. ### Example: DOCKER_SHERPA_PORT="4550" DOCKER_SHERPA_PORT="4550" #-----------------------------------------------------------------------------# ############################################################################### ### !!! NOTICE: This section is not used unless deploying local OpenLDAP!!! ### ############################################################################### #-----------------------------------------------------------------------------# ### Local LDAP Parameters (These are only used for configure local OpenLDAP) #-----------------------------------------------------------------------------# # LDAP Host which to point and authenticate against #DOCKER_LDAP_SERVER="<hostname or IP>" # Docker LDAP Node name as it will appear on host ###>>> DOCKER_LDAP_NAME="gobii_ldap_node" # This will set the hostname within the container ###>>> DOCKER_LDAP_CONTAINER_HOSTNAME="ldap-node" # LDAP Image name as it appears in docker hub ###>>> DOCKER_HUB_LDAP_NAME="gobii_ldap_base_ubuntu" # The port used for LDAP Connection #DOCKER_LDAP_PORT="389" # LDAP CN Configuration # Example: DOCKER_LDAP_CN="cn=admin" #DOCKER_LDAP_CN="<admin cn name>" # LDAP DN Configuration # Example: DOCKER_LDAP_DN="dc=gobiiproject,dc=org" #DOCKER_LDAP_DN="dn=" # LDAP Organizational Unit Configuration # Example: DOCKER_LDAP_OU="ou=gobii" #DOCKER_LDAP_OU="ou=" # User ID used to configure LDAP and other services [Default: uid=gadm] # this is only used if a user is added to a configuration such as ownCloud #DOCKER_LDAP_UID="uid=gadm" # uses image tag to determine which image to pull and run ### Example: DOCKER_LDAP_VERSION="base" #DOCKER_LDAP_VERSION="<tag name>"





You can name this file however you want. The full file path is passed to the deployment script.

For anything not clear or if you're not sure of what to put on a parameter's value, please ask Kevin Palis.

If a seed context for your crop is not available and you would like to have one (ex. seed_crop3), please contact Roy Petrie or Kevin Palis.



Edit the ‘install’ parameter file. You can find a template in the repository (gobiideployment/params/template_install.parameter). It will contain all the GOBii instance's configuration (i.e. runtime configuration via the gobii-web.xml content). The template is shown below, with each parameter explained on top of the corresponding line:



template_install.parameters

For ease of use the template has been expanded with 'white' space between parameters to allow for a more readable and more easily editable structure.

#-----------------------------------------------------------------------------# ### install.parameters - Used to create gobiiconfig_wrapper.sh #-----------------------------------------------------------------------------# ### This parameter file will be used by gobiiconfig_wrapper.sh to generate a # proper gobii-web.xml -- the main configuration file of a GOBII instance. ### Note that you need to specify this file into the *_main.parameters so that # the deployment script will pick it up. ### Also note that all the paths here are on the point of view of the running # dockers, mainly the web docker, hence the /data prefix on most of them. #-----------------------------------------------------------------------------# ### The /data directory is the default working directory of all the GOBII # dockers, this is also where the application bundle is located. It is a # volume mapped to the shared directory visible to all 3 dockers. ### The GOBII application data bundle's path in the context of the docker # containers. Keep the default unless otherwise changed in the docker images. BUNDLE_PATH="/data/gobii_bundle" ### The generated gobii-web.xml path. Keep the default unless otherwise # changed in the docker images. CONFIG_XML="/data/gobii_bundle/config/gobii-web.xml" ### The authentication type. If you want to be able to login using the test # user upon installation, set this to "TEST". If you want to immediately # connect to LDAP upon installation, set to "LDAP". ### We suggest to set it to "TEST" at first, then verify that everything works # by logging into the extractor UI. Then manually set it to "LDAP" in the # gobii-web.xml file, then restart Tomcat and you're all set. AUTH_TYPE="TEST" ### The LDAP distinguised name LDAP_DN="uid={0}" ### The LDAP URL LDAP_URL="ldaps://test.cornell.edu/ou=people,dc=testl,dc=testnet" ### The LDAP Bind User LDAP_BIND_USER="uid=gobii-user" ### The LDAP Bind User's password LDAP_BIND_PASSWORD="dummypass" ### The LDAP background user -- the webservices use this to query for valid users, etc LDAP_BACKGROUND_USER="gobii-user" ### The LDAP backround user's password LDAP_BACKGROUND_PASSWORD="dummypass" ### The mail host. This can be your local mail host or a Gmail SMTP server. MAIL_HOST="smtp.gmail.com" ### The mail server's port. MAIL_PORT=587 ### The mail user account. MAIL_USERNAME="gobii.test@gmail.com" ### The mail user account's password. MAIL_PASSWORD="dummypass" ### The type of mail server, Gmail defaults to SMTP. MAIL_TYPE="SMTP" ### The mail server's hash MAIL_HASH="na" ### The name of the first crop. This should match what is in the docker hub # images. CROP1="maize" ### The name of the first crop. This should match what is in the docker hub # images. CROP2="wheat" ### The web host's domain name or IP address WEB_HOST="172.1.2.3" ### The web host's web server port. This should match what is in the # *_main.parameter file. WEB_PORT="8081" ### The first crop's context path (the web application). This should match # what is in the docker hub images. CROP1_CONTEXT_PATH="/gobii-maize" ### The second crop's context path (the web application). This should match # what is in the docker hub images. CROP2_CONTEXT_PATH="/gobii-wheat" ### The database host's domain name or IP address DB_HOST="172.1.2.4" ### The database host's postgres port. This should match what is in the # *_main.parameter file. DB_PORT="5433" ### The main GOBII database user DB_USERNAME="dummyuser" ### The main GOBII database user's password. If you want to change this # password, you can do so once the whole automated deployment finish # successfully. But make sure after changing it in postgres, change the # corresponding tag in the gobii-web.xml file. DB_PASS="dummypass" ### The database name of the first crop. This should match what is in the # docker hub images. DB_NAME_CROP1="gobii_maize" ### The database name of the second crop. This should match what is in the # docker hub images. DB_NAME_CROP2="gobii_wheat" #-----------------------------------------------------------------------------# ### KDCompute related params used by gobiiconfig jar #-----------------------------------------------------------------------------# ### KDcompute host KDC_HOST="changeme" ### KDcompute port KDC_PORT="8083" ### kdcompute application context on server KDC_CONTEXT_PATH="kdcompute" ### KDcompute job start param KDC_JOB_START="qcStart" ### KDcompute job status param KDC_JOB_STATUS="qcStatus" ### KDcompute job downloand KDC_JOB_DOWNLOAD="qcDownload" ### Seconds to wait between status checks KDC_JOB_CHECK_STATUS="60" ### Minutes until job is hung KDC_JOB_FAIL_THRESHOLD="2880" ### KDcompute purge the jobs KDC_PURGE="qcPurge" ### KDcompute is active (false|true) for GOBII KDC_ACTIVE="false"



As of version 1.3, the *install.parameter file is also passed as a parameter to the main call to the gobii_ship scripts. Hence, it is not necessary to be set in the CONFIGURATOR_PARAM_FILE of the *main.parameter file.

For anything not clear or if you're not sure of what to put on a parameter's value, please ask Kevin Palis.





Running the Deployment Script



This script should not be run using sudo or as the root user. Some commands will automatically prompt you if they need elevated permissions.



If you are deploying GOBii into just one machine, you run ‘the_gobii_ship.sh’ to pull, deploy, and configure all 3 docker containers to one target server. To do so, you run a command similar to:

#Usage: bash.sh the_gobii_ship.sh <path-of-main-param-file> <path-of-install-param-file> <dockerhubpassw | askme> <gobii_release_version> #Set dockerhubpassw parameter to 'askme' for the script to prompt for password instead. bash the_gobii_ship.sh params/template_main_<CG Center File Name>.parameters params/template_install_<CG Center File Name>.parameters askme release-2.1-1



If you made a mistake and want to start over or if there are other dockers in the server you want to get rid of, do a cleanup by running docker stop, rm, and rmi.



The GOBii Ship...

As of version 2.0 it is only required to run one script.



#!/usr/bin/env bash set -e # Abort script at first error, when a command exits with non-zero status set -u # Attempt to use undefined variable outputs error message, and forces # an exit #set -x # xtrace: Similar to -v, but expands commands [to unset and hide # passwords us "set +x"] # set -v # sets verbosity to high echoing commands before executing #-----------------------------------------------------------------------------# ### !!! WARNING !!! #-----------------------------------------------------------------------------# ### Do not run this script as SUDO! There are explicit sudo commands which ### will prompt you for password. But not everything should run as sudo. ### usage: bash <bash_script_name>.sh <path of main.parameters> <path of # install.parameters> <dockerhubpassw | "askme"> <gobii_release_version> ### This a stand-alone equivalent of my THE_GOBII_SHIP Bamboo plan for the ### ownCloud NODE #-----------------------------------------------------------------------------# ### Requirements: ###>>> 1. The user that will run this script needs to be a sudoer and under the gobii and docker groups. So preferably the user 'gadm'. ###>>> 2. The working directory needs to be where the gobiiconfig_wrapper.sh # is as well, typically <gobii_bundle>/conf/ ###--->>> NOTE: The order of execution is important. ###--->>> NOTE: If weird things start happening on your containers, try # removing the images as well by running 'docker rmi' on each of the 3 nodes. #-----------------------------------------------------------------------------# ### If you want a delete-all-images command, run this: ###>>> [sudo docker stop $(sudo docker ps -qa) || true && sudo docker rm # $(sudo docker ps -aq) || true && sudo docker rmi $(sudo docker images -aq) # || true] #-----------------------------------------------------------------------------# #-----------------------------------------------------------------------------# #@author: (palace) kdp44@cornell.edu ###>>> Co-Author: RLPetrie (rlp243@cornell.edu) #-----------------------------------------------------------------------------# #-----------------------------------------------------------------------------# ### Error on running with sudo #-----------------------------------------------------------------------------# if [[ $EUID -eq 0 ]]; then echo "This script should not be run using sudo or as the root user. Some commands will automatically prompt you if they need elevated permissions." exit 1 fi if [ $# -lt 3 ]; then echo "No arguments supplied." echo "Usage: bash <bash_script_name>.sh <path of main.parameters> <path of install.parameters> <dockerhubpassw | \"askme\"> <gobii_release_version>" echo "Set dockerhubpassw parameter to 'askme' for the script to prompt for password instead." exit 1 fi #-----------------------------------------------------------------------------# ### load parameters -- main.parameters for deployment #-----------------------------------------------------------------------------# source $1 #-----------------------------------------------------------------------------# ### load parameters -- install.parameters for configuration #-----------------------------------------------------------------------------# source $2 #-----------------------------------------------------------------------------# ### Set release version and Dockerhub password ###>>> if pass set to 'askme' call for user to enter password at cli #-----------------------------------------------------------------------------# DOCKER_HUB_PASSWORD=$3 #-----------------------------------------------------------------------------# ### Menu #-----------------------------------------------------------------------------# clear echo " #------------------------------# ### GOBii Deployment Options ### #------------------------------# " PS3=' Please enter your choice: ' options=( "Full GDM Deployment [DB, Web, Compute, KDC, ownCloud, Portainer]" "GDM + KDC [DB, Web, Compute, KDC]" "GDM [DB, Web, Compute]" "db-node" "web-node" "compute-node" "KDCompute" "ownCloud" "Portainer" "Sherpa [Portainer Agent]" "Quit" ) select opt in "${options[@]}" do case $opt in "Full GDM Deployment [DB, Web, Compute, KDC, ownCloud, Portainer]") echo; if [[ $REPLY == 1 ]]; then echo; echo "You have selected to deploy the full GDM Suite..." echo "[DB, Web, Compute, KDC, ownCloud, Portainer]" echo; read -p "Are you sure you want to deploy the full GDM Suite [y/n]: " SUITE_REPLY echo; if [[ $SUITE_REPLY == 'y' ]]; then GDM_RELEASE_VERSION=$4 echo; echo "Main Parameters: $1" echo "Install Parameters: $2" echo "GDM release version: $4" echo; read -p "Please press any key to continue... " -n1 -s echo; if [ $DOCKER_HUB_PASSWORD = "askme" ]; then read -sp "Please enter your Docker Hub password: " DOCKER_HUB_PASSWORD fi echo; echo "Shipping DB Container..." echo; bash container_scripts/the_gobii_ship_db.sh $1 $2 $DOCKER_HUB_PASSWORD $4 echo; echo "The db-node deployment has completed." echo; echo "Moving on to the web-node..." sleep 3 echo; bash container_scripts/the_gobii_ship_web.sh $1 $2 $DOCKER_HUB_PASSWORD $4 echo; echo "The web-node deployment has completed." echo; echo "Moving on to the compute-node..." sleep 3 echo; bash container_scripts/the_gobii_ship_compute.sh $1 $2 $DOCKER_HUB_PASSWORD $4 echo; echo "The compute-node deployment has completed." echo "Moving on to the kdc-node..." sleep 3 echo; if [ $KDC_RELEASE_VERSION = "askme" ]; then read -p "Please enter KDCompute release version: " KDC_RELEASE_VERSION fi echo; echo "Main Parameters: $1" echo "Install Parameters: $2" echo "KDCompute release version: $KDC_RELEASE_VERSION" echo; bash container_scripts/the_gobii_ship_kdc.sh $1 $2 $DOCKER_HUB_PASSWORD $KDC_RELEASE_VERSION echo; echo "The kdc-node deployment has completed." echo; echo "Moving on to the oc-node..." sleep 3 echo; bash container_scripts/the_gobii_ship_oc.sh $1 $2 $DOCKER_HUB_PASSWORD base echo; echo "The oc-node deployment has completed." echo; echo "Moving on to the Portainer deploy..." sleep 3 echo; bash container_scripts/the_gobii_ship_portainer.sh $1 $2 $DOCKER_HUB_PASSWORD echo; echo "The portainer-node deployment has completed." echo; else echo "Exiting the_gobii_ship.sh" echo; break fi echo; break fi ;; "GDM + KDC [DB, Web, Compute, KDC]") echo; if [[ $REPLY == 2 ]]; then echo; echo "You have selected to deploy GDM with KDCompute..." echo "[DB, Web, Compute & KDC]" echo; read -p "Are you sure you want to deploy the GDM with KDCompute [y/n]: " GDMK_REPLY echo; if [[ $GDMK_REPLY == 'y' ]]; then GDM_RELEASE_VERSION=$4 read -p "Please enter KDC Release Version: " KDC_RELEASE_VERSION echo; echo "Main Parameters: $1" echo "Install Parameters: $2" echo "KDCompute release version: $KDC_RELEASE_VERSION" echo "GDM release version: $4" echo; if [ $DOCKER_HUB_PASSWORD = "askme" ]; then read -sp "Please enter your Docker Hub password: " DOCKER_HUB_PASSWORD fi echo; echo "Shipping DB Container..." echo; bash container_scripts/the_gobii_ship_db.sh $1 $2 $DOCKER_HUB_PASSWORD $4 echo; echo "The db-node deployment has completed." echo; echo "Moving on to the web-node..." sleep 3 echo; bash container_scripts/the_gobii_ship_web.sh $1 $2 $DOCKER_HUB_PASSWORD $4 echo; echo "The web-node deployment has completed." echo; echo "Moving on to the compute-node..." sleep 3 echo; bash container_scripts/the_gobii_ship_compute.sh $1 $2 $DOCKER_HUB_PASSWORD $4 echo; echo "The compute-node deployment has completed." echo; echo "Moving on to the kdc-node..." sleep 3 echo; bash container_scripts/the_gobii_ship_kdc.sh $1 $2 $DOCKER_HUB_PASSWORD $KDC_RELEASE_VERSION echo; echo "The web-node deployment has completed." echo; else echo "Exiting the_gobii_ship.sh" echo; break fi echo; break fi ;; "GDM [DB, Web, Compute]") echo; if [[ $REPLY == 3 ]]; then echo; echo "You have selected to deploy base GDM..." echo "[DB, Web & Compute]" echo; read -p "Are you sure you want to deploy the base GDM Containers [y/n]: " GDM_REPLY echo; if [[ $GDM_REPLY == 'y' ]]; then GDM_RELEASE_VERSION=$4 echo; echo "Main Parameters: $1" echo "Install Parameters: $2" echo "GDM release version: $4" echo; read -p "Please press any key to continue... " -n1 -s echo; if [ $DOCKER_HUB_PASSWORD = "askme" ]; then read -sp "Please enter your Docker Hub password: " DOCKER_HUB_PASSWORD fi echo; echo "Shipping DB Container..." echo; bash container_scripts/the_gobii_ship_db.sh $1 $2 $DOCKER_HUB_PASSWORD $4 echo; echo "The db-node deployment has completed." echo; echo "Moving on to the web-node..." sleep 3 echo; bash container_scripts/the_gobii_ship_web.sh $1 $2 $DOCKER_HUB_PASSWORD $4 echo; echo "The web-node deployment has completed." echo; echo "Moving on to the compute-node..." sleep 3 echo; bash container_scripts/the_gobii_ship_compute.sh $1 $2 $DOCKER_HUB_PASSWORD $4 echo; echo "The compute-node deployment has completed." echo; else echo "Exiting the_gobii_ship.sh" echo; break fi echo; break fi ;; "db-node") echo; if [[ $REPLY == 4 ]]; then echo; echo "You have selected to deploy db-node..." echo; read -p "Are you sure you want to deploy the DB Container [y/n]: " DB_REPLY echo; if [[ $DB_REPLY == 'y' ]]; then GDM_RELEASE_VERSION=$4 echo; echo "Main Parameters: $1" echo "Install Parameters: $2" echo "GDM release version: $4" echo; read -p "Please press any key to continue... " -n1 -s echo; if [ $DOCKER_HUB_PASSWORD = "askme" ]; then read -sp "Please enter your Docker Hub password: " DOCKER_HUB_PASSWORD fi echo; echo "Shipping DB Container..." echo; bash container_scripts/the_gobii_ship_db.sh $1 $2 $DOCKER_HUB_PASSWORD $4 echo; else echo "Exiting the_gobii_ship.sh" echo; break fi echo; break fi ;; "web-node") echo; if [[ $REPLY == 5 ]]; then echo; echo "You have selected to deploy web-node..." echo; read -p "Are you sure you want to deploy the Web Container [y/n]: " WEB_REPLY echo; if [[ $WEB_REPLY == 'y' ]]; then GDM_RELEASE_VERSION=$4 echo; echo "Main Parameters: $1" echo "Install Parameters: $2" echo "GDM release version: $4" echo; read -p "Please press any key to continue... " -n1 -s echo; if [ $DOCKER_HUB_PASSWORD = "askme" ]; then read -sp "Please enter your Docker Hub password: " DOCKER_HUB_PASSWORD fi echo; echo "Shipping Web Container..." echo; bash container_scripts/the_gobii_ship_web.sh $1 $2 $DOCKER_HUB_PASSWORD $4 echo; else echo "Exiting the_gobii_ship.sh" echo; break fi echo; break fi ;; "compute-node") echo; if [[ $REPLY == 6 ]]; then echo; echo "You have selected to deploy compute-node..." echo; read -p "Are you sure you want to deploy the Compute Container [y/n]: " COMPUTE_REPLY echo; if [[ $COMPUTE_REPLY == 'y' ]]; then GDM_RELEASE_VERSION=$4 echo; echo "Main Parameters: $1" echo "Install Parameters: $2" echo "GDM release version: $4" echo; read -p "Please press any key to continue... " -n1 -s echo; if [ $DOCKER_HUB_PASSWORD = "askme" ]; then read -sp "Please enter your Docker Hub password: " DOCKER_HUB_PASSWORD fi echo; echo "Shipping Compute Container..." echo; bash container_scripts/the_gobii_ship_compute.sh $1 $2 $DOCKER_HUB_PASSWORD $4 echo; else echo "Exiting the_gobii_ship.sh" echo; break fi echo; break fi ;; "KDCompute") echo; if [[ $REPLY == 7 ]]; then echo; echo "You have selected to deploy kdc-node..." echo; read -p "Are you sure you want to deploy KDCompute Container [y/n]: " KDC_REPLY echo; if [[ $KDC_REPLY == 'y' ]]; then if [ $KDC_RELEASE_VERSION = "askme" ]; then read -p "Please enter KDCompute release version: " KDC_RELEASE_VERSION fi echo; echo "Main Parameters: $1" echo "Install Parameters: $2" echo "KDCompute release version: $KDC_RELEASE_VERSION" echo; read -p "Please press any key to continue... " -n1 -s echo; if [ $DOCKER_HUB_PASSWORD = "askme" ]; then read -sp "Please enter your Docker Hub password: " DOCKER_HUB_PASSWORD fi echo; echo "Shipping KDCompute..." echo; bash container_scripts/the_gobii_ship_kdc.sh $1 $2 $DOCKER_HUB_PASSWORD $KDC_RELEASE_VERSION echo; else echo "Exiting the_gobii_ship.sh" echo; break fi echo; break fi ;; "ownCloud") echo; if [[ $REPLY == 8 ]]; then echo; echo "You have selected to deploy oc-node..." echo; read -p "Are you sure you want to deploy the ownCloud Container [y/n]: " OC_REPLY echo; if [[ $OC_REPLY == 'y' ]]; then echo "Main Parameters: $1" echo "Install Parameters: $2" echo "ownCloud release version: base" echo; read -p "Please press any key to continue... " -n1 -s echo; if [ $DOCKER_HUB_PASSWORD = "askme" ]; then read -sp "Please enter your Docker Hub password: " DOCKER_HUB_PASSWORD fi echo; echo "Shipping ownCloud Container..." echo; bash container_scripts/the_gobii_ship_oc.sh $1 $2 $DOCKER_HUB_PASSWORD base echo; else echo "Exiting the_gobii_ship.sh" echo; break fi echo; break fi ;; "Portainer") echo; if [[ $REPLY == 9 ]]; then echo; echo "You have selected to deploy portainer-node..." echo; read -p "Are you sure you want to deploy the Portainer Container [y/n]: " P_REPLY echo; if [[ $P_REPLY == 'y' ]]; then echo "Main Parameters: $1" echo; read -p "Please press any key to continue... " -n1 -s echo; if [ $DOCKER_HUB_PASSWORD = "askme" ]; then read -sp "Please enter your Docker Hub password: " DOCKER_HUB_PASSWORD fi echo "Shipping Portainer..." echo; bash container_scripts/the_gobii_ship_portainer.sh $1 $2 $DOCKER_HUB_PASSWORD echo; else echo "Exiting the_gobii_ship.sh" echo; break fi echo; break fi ;; "Sherpa [Portainer Agent]") echo; if [[ $REPLY == 10 ]]; then echo; echo "You have selected to deploy the portainer-sherpa-node..." echo; read -p "Are you sure you want to deploy the Portainer Sherpa Container [y/n]: " S_REPLY echo; if [[ $S_REPLY == 'y' ]]; then echo "Main Parameters: $1" echo; read -p "Please press any key to continue... " -n1 -s echo; if [ $DOCKER_HUB_PASSWORD = "askme" ]; then read -sp "Please enter your Docker Hub password: " DOCKER_HUB_PASSWORD fi echo "Shipping Portainer Sherpa Node..." echo; bash container_scripts/the_gobii_ship_sherpa_agent.sh $1 $2 $DOCKER_HUB_PASSWORD echo; else echo "Exiting the_gobii_ship.sh" echo; break fi echo; break fi ;; "Quit") break ;; *) echo "invalid option $REPLY";; esac done echo "Script exited, Good Bye" echo;



This deployment script will be updated if new containers need to be deployed along side of GDM but now has the ability to deploy full deployment and call each script in the proper order for deployment or each container individually.

This was updated to make sure the scripts did not need to be maintained in two places when configuration or updates where done to pre-existing containers.

On deployment vim is now installed on web, db, compute and kdc nodes. This will eventually be built into the dockerhub containers in the repo but for the time being they will get vim the traditional way.





LDAP Install Cert

After running the deployment scripts and doing the verification step #1 below, turn on LDAP authentication, if it wasn't already. (details on verification step #2). When turning on LDAP, make sure that the LDAP certificate is loaded to the JVM. You can do so by:

######### #run the install cert #Usage: bash install_cert.sh </data/mycert.der> <ldap_host> </usr/local/lib/keytool> </usr/local/jdk/jre/lib/cacerts> <changeit> docker exec gobii-web-node bash -c ' cd /data/gobii_bundle/config; bash install_cert.sh /data/cacert_mgs1.der cbsu_mgs1 /usr/lib/jvm/java-8-oracle/bin/keytool /usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts changeit; ' #########



You should see a confirmation message saying "certificates added to keystore". Finally, restart Tomcat, making sure it runs via user gadm:

docker exec gobii-web-node bash -c ' sudo su gadm; sh /usr/local/tomcat/bin/shutdown.sh; sh /usr/local/tomcat/bin/startup.sh; '



As you can see, /data/cacart_mgs1.der is the certificate file. The command above looks for it in the web docker home volume, which is /data. So, wherever that volume points to in the host server, make sure you put the DER file there first before running the command. Lastly, the paths to keytool and cacerts will most likely stay the same as we're only distributing dockers based on Ubuntu, but in case in the future we offer other linux flavor, or JVM changes, then those paths may change.



Make sure that the ports you assigned the dockers with (typically 8081, 8083, 8084, 5433, and 2222) are open. Otherwise the containers won't be able to communicate with each other and will fail with internal server error 500 (although in the future we may have more specific error message). The more specific error message will be shown in Tomcat's log (catalina.out). Opening a port will differ from OS to OS. 



Example: CentOS 6 and 7

$ sudo iptables -I INPUT -p tcp -m tcp --dport <port_number> -j ACCEPT $ sudo service iptables save



Make sure that the /data symlink in the web node was created for the LoaderUI to work properly. If not, do the following:

sudo ln -sfn $BUNDLE_PARENT_PATH /data

If run in the same terminal session you ran gobii_ship*.sh scripts, then run this as is, the $BUNDLE_PARENT_PATH variable should be there. If not, replace it with that parameter's value in the *_main.parameters file that you have.



If it is not possible for the /data symlink to be created because in the target host, /data is already a directory or a drive mount point, create a symlink manually from /data/gobii_bundle to point to $BUNDLE_PARENT_PATH/gobii_bundle -effectively still making /data/gobii_bundle point to the correct location.











This is a script called at the start of the deployment to verify if the system is going to be WIPED of data both the files associated with the DB as well as the database volumes are removed but it also makes sure you verify multiple times!



#!/usr/bin/env bash #-----------------------------------------------------------------------------# ### GDM KEEP/WIPE #-----------------------------------------------------------------------------# # This script is intended to verify all the data is kept or destroyed on a # deployment. It is important that a system is not partially wiped by removing # files but keeping DB data. #-----------------------------------------------------------------------------# ### @author: (rpetrie) rlp243@cornell.edu #-----------------------------------------------------------------------------# #-----------------------------------------------------------------------------# ### Environmental Settings set -e # Abort script at first error, when a command exits with non-zero status set -u # Attempt to use undefined variable outputs error message, and forces # an exit #set -x # xtrace: Similar to #-v, but expands commands [to unset and hide # passwords us "set +x"] #-----------------------------------------------------------------------------# #-----------------------------------------------------------------------------# ### Error on running with sudo if [[ $EUID -eq 0 ]]; then echo "This script should not be run using sudo or as the root user. Some commands will automatically prompt you if they need elevated permissions." exit 1 fi if [ $# -lt 1 ]; then echo "No arguments supplied." echo "Usage: bash <bash_script_name>.sh <path of main.parameters> <path of install.parameters> <dockerhubpassw | \"askme\"> <gobii_release_version>" echo "Set dockerhubpassw parameter to 'askme' for the script to prompt for password instead." exit 1 fi #-----------------------------------------------------------------------------# ### load parameters -- main.parameters for deployment source $1 #-----------------------------------------------------------------------------# clear read -p "Would you like to perform a wipe of the GOBii data? [y/n]: " confirm1 echo; # First Confirmation if [ $confirm1 == 'y' ]; then clear echo " ############################################################################### #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!# #! !# #! # # # ###### # # ### # # ##### !# #! # # # # # # # ## # # ## # # # !# #! # # # # # # # # # # # # # # # !# #! # # # # # ###### # # # # # # # # #### !# #! # # # ####### # # # # # # # # # # # !# #! # # # # # # # # ## # # ## # # !# #! ## ## # # # # # # ### # # ##### !# #! !# #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!# ############################################################################### #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!# #! !# #! # ####### ####### ####### # # ####### ### ####### # # !# #! # # # # # ## # # # # # ## # !# #! # # # # # # # # # # # # # # # !# #! # # # # ##### # # # # # # # # # # !# #! ####### # # # # # # # # # # # # # !# #! # # # # # # ## # # # # # ## !# #! # # # # ####### # # # ### ####### # # !# #! !# #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!# ############################################################################### " echo; echo "In performing the following the GOBii Database volumes and all data currently stored within the gobii_bundle will be wiped of data." read -p "Please verify you are aware and would like to continue to wipe the GOBii Data! [y/n]: " confirm2 echo; read -p "Please press any key to continue... " -n1 -s echo; else echo "You did not confirm you wanted a wipe of the GOBii Data ..." echo; echo "Continuing without a data wipe." echo; read -p "Please press any key to continue... " -n1 -s echo; exit fi # Second Confirmation if [ $confirm2 == 'y' ]; then clear echo ' #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!# ############################################################################### #! !# #! Failing to read the following message could result in loss of data!!! !# #! !# ############################################################################### #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!# ' echo; echo "Continuing will result in the following directories being deleted:" echo "• $NDD_PATH/persistent_data/gobii_bundle/crops" echo "• $NDD_PATH/persistent_data/gobii_bundle/logs/" echo "• $NDD_PATH/persistent_data/gobii_bundle/config/gobii-web.xml" echo; echo "Continuing will result in the following docker volumes being deleted:" echo "• gobiipostgresetcubuntu" echo "• gobiipostgreslibubuntu" echo "• gobiipostgreslogubuntu" echo; read -p "Please confirm that you are aware these files are going to be deleted [y/n]: " confirm3 echo; read -p "Please press any key to continue... " -n1 -s echo; else echo "You did not confirm you wanted a wipe of the GOBii Data ..." echo; echo "Continuing without a data wipe." echo; read -p "Please press any key to continue... " -n1 -s echo; exit fi if [ $confirm3 == 'y' ]; then clear echo "You have confirmed that you are aware the following will be destroyed and permanetly removed from this system: " echo; echo "Directories:" echo "• $NDD_PATH/persistent_data/gobii_bundle/crops" echo "• $NDD_PATH/persistent_data/gobii_bundle/logs/" echo "• $NDD_PATH/persistent_data/gobii_bundle/config/gobii-web.xml" echo; echo "Docker volumes:" echo "• gobiipostgresetcubuntu" echo "• gobiipostgreslibubuntu" echo "• gobiipostgreslogubuntu" echo; echo "This is the last chance before data will be wiped!" echo; read -p "Please verify you have backups of all your data outside of GDM and you are willingly wiping the data from GOBii Data Manager [y/n]: " confirm4 echo; if [ $confirm4 == 'y' ]; then echo "Continuing with data wipe..." echo; echo "Should you need to exit the Data Wipe please use Ctrl + C to cancel the script and exit now." echo; read -p "Please press any key to continue... " -n1 -s echo; clear echo "Performing Data Wipe of:" echo "• $NDD_PATH/persistent_data/gobii_bundle/crops" echo "• $NDD_PATH/persistent_data/gobii_bundle/logs/" echo "• $NDD_PATH/persistent_data/gobii_bundle/config/gobii-web.xml" echo; rm -rfv $NDD_PATH/persistent_data/gobii_bundle/crops echo; echo "Deleted: $NDD_PATH/persistent_data/gobii_bundle/crops" echo; rm -rfv $NDD_PATH/persistent_data/gobii_bundle/logs echo; echo "Deleted: $NDD_PATH/persistent_data/gobii_bundle/logs" echo; rm -rfv $NDD_PATH/persistent_data/gobii_bundle/config/gobii-web.xml echo; echo "Deleted: $NDD_PATH/persistent_data/gobii_bundle/config/gobii-web.xml" echo; echo "Moving on to Docker Volumes..." echo; read -p "Please press any key to continue... " -n1 -s echo; echo "Performing Data Wipe of:" echo "• gobiipostgresetcubuntu" echo "• gobiipostgreslibubuntu" echo "• gobiipostgreslogubuntu" echo; echo "Stopping gobii-db-node..." docker stop gobii-db-node || true; echo; echo "Removing gobii-db-node container..." docker rm gobii-db-node || true; echo; echo "Removing gobii-db-node volumes..." docker volume rm gobiipostgresetcubuntu || true; docker volume rm gobiipostgreslibubuntu || true; docker volume rm gobiipostgreslogubuntu || true; echo; echo "GDM Data Wipe has been completed." echo; echo "Please make sure to use the ndd_pathing.sh script to rebuild the Non-Destructive deployment symlinks and directories post deployment." echo; read -p "Please press any key to continue... " -n1 -s echo; else echo "You did not confirm you wanted a wipe of the GOBii Data ..." echo; echo "Continuing without a data wipe." echo; read -p "Please press any key to continue... " -n1 -s echo; exit fi else echo "You did not confirm you wanted a wipe of the GOBii Data ..." echo; echo "Continuing without a data wipe." echo; read -p "Please press any key to continue... " -n1 -s echo; exit fi echo "Script has completed... Good Bye." exit







Additional Scripts





This script has been built into the_gobii_ship.sh and it is recommended to have this running, but since its creation the .jar used within the script has been moved from its original location.



This jar can be used and processed manually until the jar has been replaced into the gobiideployment repo.



The kdc_passwd.sh was built to help update the kdc admin password.



#!/usr/bin/env bash #-----------------------------------------------------------------------------# # Update KDC UI admin password #-----------------------------------------------------------------------------# DB_PASSWORD=$1 new_password=$2 #-----------------------------------------------------------------------------# # Updating password # kdc variables DB_USER=root DB_NAME=kdcdb BCRYPT_PROGRAM=/usr/local/tomcat/kdcompute_file_storage/TestOutput_UserDirs/workspace/bcrypt/build/libs/bcrypt.jar hashed_password=`java -jar "$BCRYPT_PROGRAM" $new_password` mysql --user="$DB_USER" --password="$DB_PASSWORD" --database="$DB_NAME" --execute="update User set HashedPassword = '$hashed_password' where UserName = 'admin';" #-----------------------------------------------------------------------------# # KDC changing default admin password for UI [Manual process] # log into kdc-node # run the following command to set default gobii pass as admin password ### Note the -p and the password can not have a space between them when # running the mysql command below #-----------------------------------------------------------------------------# # manually get hash from : https://www.dailycred.com/article/bcrypt-calculator #-----------------------------------------------------------------------------# #mysql -u root -p<Default gobii pass> #use kdcdb; #update User set \ #HashedPassword = '<hashed generated password>'\ #where UserName = 'admin'; #exit; #-----------------------------------------------------------------------------# # Please be aware this is only placed within the script as a backup to calling # the orginal process so as to verify any system administrator has the power # to update the kdc UI password. #-----------------------------------------------------------------------------#









The Non-Destructive Deployment architecture must be in place in order to run this script.

  • Additionally this script requires the *main.parameters to be passed into it with the new parameter added into the parameters file

# This is the path of the data kept separated from gobii_bundle and symlinked # to ### Example: NDD_PATH="/storage/persistent_data" NDD_PATH="<Directory path for persistent_data>"



This script catches the directory locations and and files existences. Depending on the existence and current link status the script will make gobii-web.xml backups for use later, if needed, and will be sure to not remove and files or directories in persistent_data to prevent data loss.



#!/usr/bin/env bash #-----------------------------------------------------------------------------# ### ndd_pathing.sh ### This script will manually setup the Non-Destructive deployment. This # removes current data locations under /data from the deployment and then # creates symlinks to the persistent data location for crops, logs, gobii_adl # and the gobii-web.xml. ### This script also backups up newly deployed gobii-web.xml to $NDD_PATH/ # gobii_bundle/config/archived_xml/gobii-web.xml_$TAG and will request whether # to utilize new xml or continue to use old xml. #-----------------------------------------------------------------------------# #@author: RLPetrie (rlp243@cornell.edu) #-----------------------------------------------------------------------------# #-----------------------------------------------------------------------------# # setting bash environment set -e # Abort script at first error, when a command exits with non-zero status set -u # Attempt to use undefined variable outputs error message, and forces # an exit set -x # xtrace: Similar to -v, but expands commands [to unset and hide # passwords us "set +x"] # set -v # sets verbosity to high echoing commands before executing #-----------------------------------------------------------------------------# ### !!! WARNING !!! #-----------------------------------------------------------------------------# ### Do not run this script as SUDO! There are explicit sudo commands which # will prompt you for password. But not everything should run as sudo. ### usage: bash <bash_script_name>.sh <path of main.parameters> <path of # install.parameters> <dockerhubpassw | "askme"> <gobii_release_version> #-----------------------------------------------------------------------------# ### Error on running with sudo #-----------------------------------------------------------------------------# if [[ $EUID -eq 0 ]]; then echo "This script should not be run using sudo or as the root user. Some commands will automatically prompt you if they need elevated permissions." exit 1 fi #-----------------------------------------------------------------------------# ### load parameters -- main.parameters for deployment #-----------------------------------------------------------------------------# set +x # sourcing *_main.parameters source $1 #-----------------------------------------------------------------------------# # Setting Date/time tag variable TAG=$(date +"%Y-%m-%d_%H-%M-%S") #-----------------------------------------------------------------------------# # Set variables for all persistent data locations and symlink (NDD) locations # Crops crops_persistent="$NDD_PATH/gobii_bundle/crops" crops_sym="$BUNDLE_PARENT_PATH/gobii_bundle/crops" # logs logs_persistent="$NDD_PATH/gobii_bundle/logs" logs_sym="$BUNDLE_PARENT_PATH/gobii_bundle/logs" # gobii_adl adl_persistent="$NDD_PATH/gobii_bundle/core/gobii_adl" adl_sym="$BUNDLE_PARENT_PATH/gobii_bundle/core/gobii_adl" # gobii-web.xml gdm_xml_persistent="$NDD_PATH/gobii_bundle/config/gobii-web.xml" gdm_xml_sym="$BUNDLE_PARENT_PATH/gobii_bundle/config/gobii-web.xml" gdm_xml_tag=("$gdm_xml_sym"_"$TAG") gdm_xml_old_tag=("$gdm_xml_persistent"_"$TAG") xml_backup_path=("$NDD_PATH/gobii_bundle/config/archived_xml") xml_backup_tag=("$xml_backup_path"_"$TAG") #-----------------------------------------------------------------------------# # Create arrays for all persistent and sym locations persistent_locations=( "$crops_persistent" "$logs_persistent" "$adl_persistent" "$gdm_xml_persistent" ) sym_locations=( "$crops_sym" "$logs_sym" "$adl_sym" "$gdm_xml_sym" ) #-----------------------------------------------------------------------------# # clearing clear #-----------------------------------------------------------------------------# # Check file condition for each location before continuing echo; for i in "${sym_locations[@]}"; do if [ -L ${i} ] ; then if [ -e ${i} ] ; then echo "Is a Symlink: $i" echo; else echo "Broken Symlink: $i" echo; fi elif [ -e ${i} ] ; then echo "Not a symlink: $i" echo; else echo "Does not Exist: $i" echo; fi done echo; #-----------------------------------------------------------------------------# # Remove symlocations if a directory and replace with symlinks to persistent # data echo " Before continuing please verify all data is backed up to a location outside of the GDM directory structure. If you are unsure please use \"Ctrl + C\" to exit the script and verify then rerun this script. " echo; read -p "Please verify you would like to continue [y/n]: " verify1 echo; #-----------------------------------------------------------------------------# while [[ "$verify1" == "y" ]]; do # Verify crops/ is a directory and not a symlink if [ -e ${crops_sym} ] ; then if [ -L ${crops_sym} ]; then echo "$crops_sym is already a symlink." else read -p "$crops_sym is a directory. Would you like to remove and replace with the non-destructive deployment symlink to persistent data? [y/n]: " crops_verify echo; # Verify continue then remove crops/ directory and create symlink if [[ "$crops_verify" == "y" ]]; then # Recursive, force and verbose deletion of crops/ echo "Removing..." echo "rm -rfv $crops_sym || true;" echo; rm -rfv $crops_sym || true; echo; # Symbolic, force, no-dereference [treat as a normal file] # creation of crops symlink echo "Creating symlink to persistent data..." echo "ln -sfn $crops_persistent $crops_sym || true;" echo; ln -sfn $crops_persistent $crops_sym || true; echo; else echo "Not creating crops symlink." echo; fi fi else echo "$crops_sym does not exist." echo; fi #-------------------------------------------------------------------------# # Verify logs/ is a directory and not a symlink if [ -e ${logs_sym} ] ; then if [ -L ${logs_sym} ]; then echo "$logs_sym is already a symlink." else read -p "$logs_sym is a directory. Would you like to remove and replace with the non-destructive deployment symlink to persistent data? [y/n]: " logs_verify echo; # Verify continue then remove logs/ directory and create symlink if [[ "$logs_verify" == "y" ]]; then # Recursive, force and verbose deletion of logs/ echo "Removing..." echo "rm -rfv $logs_sym || true;" echo; rm -rfv $logs_sym || true; echo; # Symbolic, force, no-dereference [treat as a normal file] # creation of logs symlink echo "Creating symlink to persistent data..." echo "ln -sfn $logs_persistent $logs_sym || true;" echo; ln -sfn $logs_persistent $logs_sym || true; echo; else echo "Not creating logs symlink." echo; fi fi else echo "$logs_sym does not exist." echo; fi #-------------------------------------------------------------------------# # Checking gobii_adl and creating symlink if [ -L ${adl_sym} ] ; then if [ -e ${adl_sym} ] ; then echo "$adl_sym" echo "Is already a Symlink... " echo; else echo "$adl_sym" echo "Is a Broken Symlink. Please make sure to get the latest gobii_adl/ for testing and add it to the persistent data location." echo; fi elif [ -e ${adl_sym} ] ; then echo "$adl_sym" echo "Is not a symlink and exists." echo "Currently this should be the case on a deploy but for GDM 2.0 and later and until this can be updated this must be added manually to deployment locations." echo; echo "It is recommended to place gobii_adl/ within persistent_data/gobii_bundle/core/ directory patch to be symlinked to." echo; else # Symbolic, force, no-dereference [treat as a normal file] # creation of logs symlink echo "$adl_sym" echo "Does not Exist. Creating Symbolic link to persistent data path." echo "ln -sfn $adl_persistent $adl_sym || true;" echo; ln -sfn $adl_persistent $adl_sym || true; echo; fi #-------------------------------------------------------------------------# if [ -L ${gdm_xml_sym} ] ; then if [ -e ${gdm_xml_sym} ] ; then echo "$gdm_xml_sym" echo "Is already a symlink." echo; else echo "$gdm_xml_sym" echo "Is a broken symlink. Please verify gobii-web.xml exists in persistent data location." echo; fi elif [ -e ${gdm_xml_sym} ] ; then echo "$gdm_xml_sym" echo "Exists..." echo; echo "Backing up new xml to: $xml_backup_tag" echo; echo "mkdir -p $xml_backup_path || true;" echo "mv $gdm_xml_sym $xml_backup_tag || true;" echo; mkdir -p $xml_backup_path || true; mv $gdm_xml_sym $xml_backup_tag || true; echo; #---------------------------------------------------------------------## Determine if new xml will be used PS3='Replace gobii-web.xml with newly Deployed XML?: ' #---------------------------------------------------------------------# # the below options are set to a numbered menu that sets the $REPLY # variable options=( "Yes [Backup old gobii-web.xml and replace with new]" "No [Backup new xml and use old gobii-web.xml]" ) #---------------------------------------------------------------------# select yn in "${options[@]}" do case $yn in "Yes [Backup old gobii-web.xml and replace with new]") if [[ $REPLY == 1 ]]; then echo "Renaming current xml to: $gdm_xml_old_tag" echo; echo "mv $gdm_xml_persistent $gdm_xml_old_tag || true;" echo; mv $gdm_xml_persistent $gdm_xml_old_tag || true; echo "Coping new xml from: $xml_backup_tag" echo "To: $gdm_xml_persistent" echo; echo " cp $xml_backup_tag $gdm_xml_persistent || true; " echo; cp $xml_backup_tag $gdm_xml_persistent || true; echo; echo "ln -sfn $gdm_xml_persistent $gdm_xml_sym || true;" echo; # Symbolic, force, no-dereference [treat as a normal # file] creation of logs symlink ln -sfn $gdm_xml_persistent $gdm_xml_sym || true; echo; echo; break fi ;; "No [Backup new xml and use old gobii-web.xml]") if [[ $REPLY == 2 ]]; then ln -sfn $gdm_xml_persistent $gdm_xml_sym || true; echo; break fi ;; *) echo "invalid option $REPLY";; esac done echo "ndd_pathing.sh script completion!" echo; else echo "$gdm_xml_sym" echo "Does not Exist." fi break done exit















Livelinks

Livelinks are links sent within the notification emails for loads and extracts that point to the files location within the owncloud file browser





When running the following line to enable livelinks, it will need to be run from the /data/gobii_bundle/config directory



java -jar gobiiconfig.jar -wfqpn gobii-web.xml -ownc -ocERR logs -soR crops -soH <hostname or IP> -soN 8084 -soU gadm -soP <gadm user pass>







[Deprecated] Restoring backup data



Due to the implementation of Non-Destructive deployment the restore process is no longer needed.



This section details the data restoration process used after a backup and deployment has completed.



Simply run the restore script with the correct parameters: GOBII Add-on Scripts#onScripts-RestoreDataBundlefromBackup



Go to the link above as the syntax in running these scripts have changed slightly from version 1.4.



Verify that the data was restored by opening any crop's ExtractorUI. You should see previously loaded datasets.

  • OPTIONAL: If you have a KDCompute Docker container running before, restore its files from backup simply by copying the TestOutput_UserDirs subdirectory of kdcompute_file_storage backup to /data/kdcompute_file_storage/TestOutput_UserDirs









[Deprecated] Configure Timescope

This section has been deprecated as the process has been built into the deployment scripts.  This section will remain on this version deployment documentation for future deployment references.

As of version 1.5, we are adding a new web application called "Timescope". This will allow users to browse and delete data permanently from the database.

With this, there are additional steps that needs to be done – but only needs to be done once (i.e. if you upgrade to any version >1.5 in the future you won't need to do the following anymore).



[Deprecated] Creating Timescope User

This process should not be needed as the 'timescoper' user is already built into the deployed DB. This section is being kept for future references.

As of version 1.5, we are adding a new web application called "Timescope". This will allow users to browse and delete data permanently from the database.

With this, there are additional steps that needs to be done – but only needs to be done once (i.e. if you upgrade to any version >1.5 in the future you won't need to do the following anymore).

  1. Create the database user for timescope:

    1. SSH into the database node, then go into the database docker container via:

      docker exec -ti <gobii_db_node> bash



    2. Switch to the postgres user



      1. su - postgres



    3. Create the timescoper db user



      1. create user timescoper with superuser password 'yourpasswordofchoice' valid until 'infinity';





 

Layered System Architecture

Created by 

Yaw Nti-Addae

Jun 26, 2019

Analytics

 

This architecture stack is for batch operations. Metadata size and genotype can easily get too large for conventional data loading to handle. The main differences of this stack from the "general" architecture are the data access layer and the business layer. The digester serve as the business layer. It will convert whatever input files (raw files like hmp, csv, etc + instruction files from the presentation layer) to a format that the data access layer will understand for loading (IFL). It is also responsible for giving the instructions on what( information-add) to extract and pass (out -)them to the metadata extractor (MDE). The data access layer here is broken into two parts based on functionality. IFL is for batch loading data to the different data stores while MDE is for extracting data in batches and writing them to files. You can also think of IFLs and MDEs as including the functions provided to load and extract the genotype matrix from HDF5/MonetDB. The whole communication line of the digesters and the data access layer is facilitated by cron jobs (as indicated in the gear icons below).

  1.  

    1.  

      1. docker exec -ti <gobii_web_node> bash



    2. Modify the timescope config file ( /usr/local/tomcat/webapps/timescope/WEB-INF/classes/config.properties), replacing the credentials with the ones you made in step 1c.



      1. # Timescope credentials #db db.username=timescoper db.pw=<default or yourpassword if you've changed it> #do not change the version parameter version=1.5



    3. Reload the webapp via Tomcat Manager

      1. Open your browser and go to <web_node_url>:<web_node_port>, then click on Tomcat Manager

      2. You should see timescope from the link, click "reload":

To verify that Timescope is properly deployed: Open your browser and navigate to <web_node_url>:<web_node_port>/timescope. Upon initial install, there will only be one superuser account in your Timescope database. The credentials are in this page: Default Credentials. When you first log in, please change this password using the Timescope UI for security. If you cannot access it, contact either Kevin Palis or Roy Petrie.

A few things to take note regarding Timescope:

  • You will need to create accounts (using the User tab) for everyone who needs to access Timescope

    • You need to assign temporary passwords for each user and ask them to change it upon log in. There is no mandatory password change feature (yet).

  • Each crop database's user management for Timescope are separate. i.e. you can have one user added to maize but not to wheat, but also if another user needs to be on both, you'll have to add the user manually to both crops

  • You only need to provision accounts once. Future deployments will always preserve postgres data – as long as the Docker volumes don't get deleted.



Timescope Verification

To verify that Timescope is properly deployed: Open your browser and navigate to <web_node_url>:<web_node_port>/timescope. Upon initial install, there will only be one superuser account in your Timescope database. The credentials are on this page: Default Credentials. When you first log in, please change this password using the Timescope UI for security. If you cannot access it, contact either Kevin Palis or Roy Petrie.

A few things to note regarding Timescope:

  1. You will need to create accounts (using the User tab) for everyone who needs to access Timescope

    1. You need to assign temporary passwords for each user and ask them to change it upon login. There is no mandatory password change feature (yet).

  2. Each crop database's user management for Timescope are separate, i.e., you can have one user added to maize but not to wheat, but also if another user needs to be on both, you'll have to add the user manually to both crops.

  3. You only need to provision accounts once. Future deployments will always preserve postgres data – as long as the Docker volumes don't get deleted.









GOBii Portal

This section shows the portal that links all products and features with GDM. 



Post initial deployment is recommended to back up current launchers.xml file used within web-node and replace after deployment.

/usr/local/tomcat/webapps/gobii-portal/config/launchers.xml









Multiple crops and Additional links in GOBii portal

The *_main.parameters will need new lines indicating the name of the new crops. These parameters can be anywhere within the *_main.parameters file.

PORTAL_CROP2_NAME="gobii-<crop2 name>" PORTAL_CROP3_NAME="gobii-<crop3 name>" PORTAL_CROP4_NAME="gobii-<crop4 name>" PORTAL_CROP5_NAME="gobii-<crop5 name>" PORTAL_CROP6_NAME="gobii-<crop6 name>"



Location: xml_config_parser.py

/data/gobii_bundle/config/utils/xml_config_parser.py



Location: launchers.xml

/usr/local/tomcat/webapps/gobii-portal/config/launchers.xml



Deploying more than one crop or adding links into the portal utilizes /data/gobii_bundle/config/utils/xml_config_parser.py which changes and updates the /usr/local/tomcat/webapps/gobii-portal/config/launchers.xml

During deployment, the script configures the original crop for crop 1 noted in the parameters file but additional crops and links will need to be added by adding the following to the script or manually running the following commands.

Example below is defaulted in the GOBii web script template for adding portainer to the deployment.

docker exec -u gadm $DOCKER_WEB_NAME bash -c " cd /data/gobii_bundle/config/utils; python xml_config_parser.py -f /usr/local/tomcat/webapps/gobii-portal/config/launchers.xml -a \"<launcher> <name>Portainer</name> <url>http://$DOCKER_PORTAINER_HOST:9000</url> <logo>portainer.png</logo> <description>Portainer</description> <color>color-green</color> <category>Container Management</category> <type>Web Apps</type> <documentationList /> </launcher>\" -x . ; ";



If the configurations need to be changed and scripts are erroring you can add the above configurations manually. The webpage will update dynamically.









ownCloud

This section will show the setup and configuration required post deployment. This section assumes the container was deployed but the LDAP, Storage and Shares have not been configured.

After ownCloud deployment login with ownCloud default user and pass. This user and pass will have to be updated by the deploying system administrator as the user and password are stored and salted in the DB.

admin/admin



Once logged in, select user name "Admin" > "Settings" > On left panel, under Admin, select "User Authentication". The configuration on "Server " tab will show the configurations made in the *_main.parameters file. If the configurations were correct at deployment at the bottom will show "".

If the below screenshot shows "" instead, update the configurations within this tab until it shows OK for your authentication configuration.



LDAP Certificates

If using a certificate the configuration will show "OK" once it’s properly setup but will fail to return any users or groups. Though within the "Login Attributes" tab a username can be verified even without the certificate but this is the extent until the certificate is added to the container.

On deployment, the /data directory is mounted to the ownCloud container. Place the certificate anywhere within /data then copy to the /var/www/owncloud directory. The system should pick this up on the next attempt to authenticate.



ownCloud Active Directory Configuration

ownCloud works well with LDAP but needs additonal settings configured for systems using Active Directory. Within the "Expert" tab at the right settings for Internal Username and UUID may need to be updated.



  • Select "Enable external storage"

  • Enter the folder name for logs and crops

  • Select sftp

  • Select username and password

  • Under "Configuration"

    • <hostname or IP>

    • directory location on host [i.e. /data/gobii_bundle/logs]

    • gadm username

    • gadm password

  • Within "Available for" if there is no user or group added these mounts are available for all users.

    • GOBii suggested configuration is to add only local owncloud "admin" user to lock down the access of these to only the admin user

Verify under the gear icon the "Enable Preview" and "Enable Sharing" are checked



Sharing External Storage with Users

  • Find the directories in admin home > select ellipsis > select "Details"

  • Select "Sharing" > Under User and Groups enter the "GOBii" group and select

  • Select the down arrow > uncheck

    • can share

    • can edit

    • create change delete

This will allow the GOBii group to see and use the files and directories shared but will be unable to edit or change them.


Enabling File Scan

Add the following line into the root cronjobs within the ownCloud container. The files available when shared, will not be updated afterwards unless the following line is added to preform an ownCloud file scan to pick up the new files.

*/5 * * * * occ files:scan --all >> /var/log/cron.log 2>&1



:docker exec gobii-oc-node bash -c 'occ user:sync "OCA\User_LDAP\User_Proxy" -m disable -r'



Disable File Locking

Make sure to follow this process in both config.php and overwrite.config.php

Add or update the following files with the subsequent configurations. This will make sure files are not locked when accessing and performing scans which the cronjobs can overlap can cause issues within the system running multiple overlapping filescans over the full system. This is highly recommend for the systems that use large files.



config/config.php

'filelocking.enabled' => false,



config/overwrite.config.php

$config['filelocking.enabled'] = getenv('OWNCLOUD_FILELOCKING_ENABLED') == 'false';



Enable local file mounts

Enable this in the configurations to allow for local system mounting of files. This will allow for the files and volumes mounted into the container to be accessed and mounted for file sharing.

A common error encountered: the owncloud instance will be unable to raise permissions when attempting to share a local mounted file or directory. This can be fixed by increasing the permissions of the file or directory.



It is recommended to use the following command against the shared directories:

sudo chmod -R 777 /data



These permissions can be dropped to acceptible levels once the locations have been shared within owncloud.



Within the configuration, update and add the following line:



config/config.php

'files_external_allow_create_new_local' => 'true',









Portainer

Portainer is a container that sits on a system and monitors all docker/container information. This system can monitor multiple end points by deploying the sherpa container opening any specific port. This allows for the portainer container to access and monitor all containers on a remote system.



Portainer Initial Login

On the initial login, portainer will request the admin to set up the password.

Portainer holds configurations under the /data directory. If the system is removed and redeployed the same configurations will remain as the portainer files within /data are not removed.





Select "Local" > Select "Connect"

this will allow for local monitoring and allow for adding remote endpoints to be monitored post deployment.



Adding Sherpa Agent Node

Select "Endpoints" in left panel > Select "Add endpoint"



Add the configurations for the sherpa node under "Environment Details":

  • Name

  • Enpoint URL

  • Public IP



During testing of portainer the latest has problems adding endpoints and would fail to add with a very undescriptive error. This error only occurred when attempting to connect ubuntu 16.04 server VMs together with latest portainer and latest sherpa on both of these versions of portainer/sherpa and Ubuntu 16.04 OS on a VM. This error was not seen between

  • CentOS to CentOS

  • Ubuntu to CentOS

  • CentOS to Ubuntu





Deploying Sherpa Agent Container



Sherpa opens the contain port for external access but is limited to the specified networks setup in the parameters:

# All allowed networks to access the sherpa agent port # Make sure to use the format from example and \" to exit each quote ### Example: DOCKER_SHERPA_NETWORK_RULES="\"10.0.0.0/8\", \"192.168.0.0/16\", \"172.0.0.0/8\"" DOCKER_SHERPA_NETWORK_RULES="<networks allowed to access sherpa port>"

The portainer container will be unable to monitor the remote host unless the communication to specified port is specified.



Deploying Sherpa via GOBii scripts

  • Verify all parameters are updated for the sherpa agent

  • To deploy the the_gobii_ship.sh and select the sherpa agent

Deploying Sherpa manually

  • Using the configurations specific to the environment run the following command to deploy the sherpa remote agent container: [Settings below are defaulted for local access]

    • Network rules syntax: 10.0.0.0/24

    • Port: This can be any port. Portainer defaults to 2375 and GOBii normally uses 4550

docker run -d \ --name sherpa_portainer-agent \ -h portainer-agent-node \ -e CONFIG='[ { "Path" : "/", "Access": "allow", "Addresses": [<network rules see above>] } ]' \ -v /var/run/docker.sock:/tmp/docker.sock \ -p <port>:<port> \ djenriquez/sherpa --allow docker start sherpa_portainer-agent







Post Deployment Verification [Smoke Testing]

This section is large enough it warrants its own document.  Please follow the link below to get the version deployment Smoke Testing documentation.

DevOps Smoke Testing Process