Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

Rationale: Storing instructions in the file system as static JSONB objects moves state information away from the central repository of information about a job, confuses users, and does not play well with dynamic instances or state queries.

Overall Design:

*Decouple webservices and filesystem by having web-service talk with a scheduler

*Scheduler and database aware of ‘job’ being requested allows for more dynamic processing

*Dynamic processing and postgres storage allow more real-time status updates, facilitating updates to the Job Status redesign

Phase 1: Proof Of Concept of Direct Instruction Injection

At this phase, we look into the difficulty of modifying the webservice layer in such a way as it immediately kicks off a process thread, much the same way BERT currently directly processes jobs. This will include passing the job data directly instead of through a JSON file.

Phase 2: Scheduler/Load Balancer

Once the Phase one prototype is complete, the difficulties of adding a scheduler into the mix can be researched pretty easily. During this phase, we hope to identify a scheduler/load balancer that can work well with spot instances and other cloud improvements.

Phase 3: Instruction database table and interface

At this point, we can investigate efficient storage of the instruction data structures as well as live updates from the scheduled/balanced jobs onto this structure, as well as how this structure relates to the upcoming job table redesign.

Post-Phase 3: Finish

Interfacing with new Job Status redesign.

Testing/fixing issues found in prototype.

Integration with main codebase.

  • No labels