Rationale: Storing instructions in the file system as static JSONB objects moves state information away from the central repository of information about a job, confuses users, and does not play well with dynamic instances or state queries.
Overall Design:
*Decouple webservices and filesystem by having web-service talk with a scheduler
*Scheduler and database aware of ‘job’ being requested allows for more dynamic processing
*Dynamic processing and postgres storage allow more real-time status updates, facilitating updates to the Job Status redesign
Phase 1: Proof Of Concept of Direct Instruction Injection
At this phase, we look into the difficulty of modifying the webservice layer in such a way as it immediately kicks off a process thread, much the same way BERT currently directly processes jobs. This will include passing the job data directly instead of through a JSON file.
Phase 2: Scheduler/Load Balancer
Once the Phase one prototype is complete, the difficulties of adding a scheduler into the mix can be researched pretty easily. During this phase, we hope to identify a scheduler/load balancer that can work well with spot instances and other cloud improvements.
Phase 3: Instruction database table and interface
At this point, we can investigate efficient storage of the instruction data structures as well as live updates from the scheduled/balanced jobs onto this structure, as well as how this structure relates to the upcoming job table redesign.
Post-Phase 3: Finish
Interfacing with new Job Status redesign.
Testing/fixing issues found in prototype.
Integration with main codebase.