GOBii Applications and Modules

Data Warehouse (PostgreSQL)

Kevin Palis

The main data warehouse as modeled here: https://gobiiproject.atlassian.net/wiki/spaces/GDW/pages/249200646

This also includes the Data Access Layer. For more info, see the main documentation space here: https://gobiiproject.atlassian.net/wiki/spaces/GDW

2.2

Done and awaiting deployment [CIMMYT, IRRI, ICRISAT]

database container

This module is actively being developed and improved. It ships with the entire GDM stack and so versioning is the same.

HDF5 Storage for Genotype

Dave Matthews

The main storage of the genotype matrix. This includes some C libraries written for the creation and extraction of HDF5 files for GOBii. We have yet to update its documentation in our Confluence cloud account.

2.2

Done and awaiting deployment [CIMMYT, IRRI, ICRISAT]

database container

This module is actively being developed and improved. It ships with the entire GDM stack and so versioning is the same.

Digesters

Josh Lamos-Sweeney

This is the middleware. It handles file processing and interfaces with the Data Access Layer directly for both loading and extraction of data. More information is available here: https://gobiiproject.atlassian.net/wiki/spaces/GD/pages/257589285

2.2

Done and awaiting deployment [CIMMYT, IRRI, ICRISAT]

processor container

(aka compute container)

This module is actively being developed and improved. It ships with the entire GDM stack and so versioning is the same.

LoaderUI

Yaw Nti-Addae / Angel Raquel

Desktop application for mapping different input files to GOBii data entities and for creating loading jobs. More information is available here:

2.2

Done and will be uploaded once 2.2 GDM is deployed [CIMMYT, IRRI, ICRISAT]

dowloadable, executable JAR file

This is built on an SWT version that came with Eclipse back in 2016. It has its challenges. It also doesn’t ship with our Docker containers and each user has to download it.

The Web Loader UI that PFR is working on, aims to replace this.

ExtratorUI / Web Services

Vishnu Govindaraj

Web application for creating extraction jobs. More information is available here:

2.2

Done and awaiting deployment [CIMMYT, IRRI, ICRISAT]

web container

This module ships with the web container. One ExtractorUI app = one crop, so in cases like ICRISAT having 6 crops, you will see 6 instances of this app.

Also, the web services are packaged in the same WAR as the ExtractorUI.

Timescope

Angel Raquel / Kevin Palis

Web application that can connect to any GDM backend to manage data. Currently, it covers datasets and markers searching and cleanup. You can find the main documentation space here:

2.2

Done and awaiting deployment [CIMMYT, IRRI, ICRISAT]

web container

(with plans of creating an independent container)

Currently, this application covers complex queries and cascading deletions of both dataset and marker entities (i.e. R and D of CRUD). @KevinPalis is proposing this to be a CRUD-complete software for the GDM backend and be packaged independent of the web container. This way we can spin it up anywhere as long as there’s a network route to GDM’s database port.

BeRT

Luke Cook

BeRT = Backend Regression Test

This is a framework specially built to do automated tests of the GOBii backend - from loading to extraction of different scenarios (i.e. different file formats of different vendor files)

1.0

Done and integrated to GOBii’s CI/CD pipeline

n/a

This currently doesn’t ship with GDM, but there are plans to package the binaries with one of the containers so we can do an initial automated test upon deployment to verify that the deployment is successful.

Marker Portal

JHI

(Iain Milne)

A simple web page with links to all the applications deployed to a particular server.

2.2

Done and awaiting deployment [CIMMYT, IRRI, ICRISAT]

web container

The PortalUI that PFR is working on aims to replace this and add functionalities.

KDCompute

DArT

(Andrew Kowalczyk)

Web application for data analysis that is currently integrated with our loading process (a checkbox in the LoaderUI toggles it on or off for a particular job). More information on KDCompute is available here: http://www.kddart.org/kdcompute.html

1.6.1 (server)

0.14.3 (plugin)

Deployed [CIMMYT, IRRI, ICRISAT]

KDCompute container

This ships in its own container and the last release version was May of 2019.

Portainer

n/a

(Kevin Palis / Roy Petrie for deployment)

This is a third-party open-source Docker management toolset. More information is available here: https://www.portainer.io/overview/

1.23.1

Done and awaiting deployment [CIMMYT, IRRI, ICRISAT]

Portainer container

This ships in its own container and is completely optional to clients. It merely provides a UI to manage containers - so everything this does can be done in the command line as well.

OwnCloud

n/a

(Roy Petrie for deployment)

This is a third-party client-server software for file-hosting services. We simply deploy the container they provide and link it to GDM’s filesystem. More information on OwnCloud is available here: https://owncloud.org/

10.1

Deployed [CIMMYT, IRRI, ICRISAT]

ownCloud container

This ships in its own container which we apparently re-imaged back in February 2019. We don’t make any modifications to this software and we just run our image of the owncloud server directly.

FlapJack

JHI

(Iain Milne)

Flapjack is a multi-platform application providing interactive visualizations of high-throughput genotype data, allowing for rapid navigation and comparisons between lines, markers and chromosomes. See https://ics.hutton.ac.uk/flapjack/

1.19.09.04

Available for download from JHI’s website

downloadable executables based on OS

Flapjack Bytes

JHI

(Iain Milne)

A web application subset of the Flapjack desktop app. This consumes BrAPI web services for data visualization.

n/a

In development

TBD

This is still under development by JHI. I will update this row when I have more info (such as the release date, tech stack, etc.)

Galaxy

Alexis Dereeper / Venice Juanilas

Pipeline for genomic selection and GWAS. See http://galaxy-demo.excellenceinbreeding.org/

TBD

EiB-hosted

n/a

This software is detached from the rest of GOBii’s products. We merely add a link to the Marker Portal to link to the EiB-hosted instance.

It is currently hosted in IRRI’s cloud server and funded by EiB.

DArtView

DArT

(Andrew Kowalczyk)

Desktop application that provide basic data quality stats and allow filtering of datasets (e.g. as part of a curation process to remove a bad plate, etc.).

More info TBD

n/a

In development

TBD

This is still under development by DArT and I will update this row when I have more info (such as the release date, tech stack, etc.)

PedVer

Vishnu Govindaraj

A web application that extracts data from GDM (via BrAPI calls) and generates a Flapjack project file that can be used in a Flapjack application for pedigree verification.

n/a

In development

3 containers:

• redis

• web (gunicorn)

• worker (celery)

First version will possibly be released with GDM 2.2. It uses GDM BrAPI web services and as such, requires a connection to the web container.

KeyCloak

Yaw Nti-Addae

Open-source IAM from RedHat. See https://www.keycloak.org/

n/a

In development

keyCloak container

This is still under development; the design on how it will integrate with GDM is not defined yet.

Web Configurator

Kevin Palis

This is part of Timescope and our attempt to make GDM configuration manageable (via a UI). This allows modification to LDAP and database connections, GDM email account, logging level; and allows addition/deletion of crops in a GDM instance.

n/a

In development

web container (part of Timescope)

This is dependent on the current architecture of GDM and may require significant rewrite if we are changing the architecture. It is, however, sitting on its own branch, so if we decide not to use this (in light with architectural changes), we simply do not merge it.

MonetDB

n/a

This has been long deprecated (around 2017).

n/a

Deprecated

n/a

Despite being deprecated, some of the scripts still ship with GDM. We need to completely remove it from our stack.

Web Loader UI

PFR / Yaw Nti-Addae

A web application version of the desktop LoaderUI.

n/a

In development

loaderUI container

This is currently being developed by PFR and will ship as a separate container once done. It uses GDM BrAPI web services and as such, requires a connection to the web container.

Portal UI

PFR / Yaw Nti-Addae

A web application that will replace JHI’s Marker Portal.

n/a

In development

portalUI container

This is currently being developed by PFR and will ship as a separate container once done.

JobStatus UI

PFR / Yaw Nti-Addae

A web application for tracking jobs in a GDM instance. A rewrite of the current job status module (including DB schema changes) of GDM is necessary for this application to work.

n/a

In development

jobStatusUI container

This is currently being developed by PFR and will ship as a separate container once done.