Questions to be answered:
How does this correlate to the new architecture?
Will the calling service manage load balancing / retries
How will the calling service trigger the load? (scp/ssh, web calls, tcp, etc…)
How do instructions relate to jobs? (many jobs to one instruction?, one to one?)
...
Rationale: Storing instructions in the file system as static JSON objects moves state information away from the central repository of information about a job, confuses users, and does not play well with dynamic instances or state queries.
...
Phase 2: Scheduler/Load Balancer
(**** This phase may be marked as obsolete in new architecture ****)
Once the Phase one prototype is complete, the difficulties of adding a scheduler into the mix can be researched pretty easily. During this phase, we hope to identify a scheduler/load balancer that can work well with spot instances and other cloud improvements.
...
At this point, we can investigate efficient storage of the instruction data structures as well as live updates from the scheduled/balanced jobs onto this structure, as well as how this structure relates to the upcoming job table redesign.
Post-Phase 3: Finish
→ Interfacing with new Job Status redesign.→ Testing/fixing issues found in prototype.
...
User targets the ‘createInstruction’ end point in the web services, the same as it is done now
The serialized instruction is redirected to a digester service
(Optional) a record of the instruction is placed in the database, perhaps coupled with the job
The digester service resides on the compute node, and spins up digesters in a thread poolThe digester has callbacks built into it, updating the calling machine with changes in its status
→ Using the createInstruction end point means no changes to the loaderUI will be needed
...
→ Database schema can be a very simple file store. That is the whole table will be
‘key → instructionJson’
Callback API (Prototoype)
The Digester expect a target to an api for calling the webservice on change to status.
Here’s the API (prototype):
| Code Block |
|---|
POST .../job
Body: {Job JSON}
Creates a Job described the the given body JSON
PATCH .../job/{job_id}
Body: {Job Prototype JSON}
Updates the job with the given prototype |
...
Digester Service
The digester services acts as a mediator between the submitted jobs, and the digesting instances. It lives on the compute node, and accepts Digest instructions. It self regulates scheduling, with an abstracted regulator, allowing for configuration between systems on strategies.
When the digester service starts up, it will request all uncompleted jobs from the web service. It will execute those that are deemed ‘unsettled'. Being unsettled is intentionally vaguely stated as there are many situations where a digest job may be unfinished, but may or may not want to be executed again. The base of this requirement is all jobs with the ‘submitted’ tag should be executed. This allows the digester to hold onto instructions, without executing them and without the worry on critical failure that these jobs will never be executed.
...