...
Rationale: Storing instructions in the file system as static JSONB JSON objects moves state information away from the central repository of information about a job, confuses users, and does not play well with dynamic instances or state queries.
Furthermore, the file system is an unencapsulated stateful location, with stateful permissions and strong uniqueness constraints. All of this put together creates a volatile environment with strong imposed restrictions on the logic surrounding instructions.
What this all means is this: Instructions must be uniquely identified by a name (the file name), the running process must have permission to move the files, and there must be some guarantee that the files won’t move while the process is running (else a failure may arise when it looks for them to finish things up)the running process does not have complete ownership over its entities. All in all, having instructions on the file system imposes business logic, and creates complexity that is unneeded.
Overall Design
Goals
→ Decouple webservices and filesystem by having web-service talk with a scheduler
→ Scheduler and database aware of ‘job’ being requested allows for more dynamic processing
→ Dynamic processing and postgres storage allow more real-time status updates, facilitating updates to the Job Status redesignRemove all references to instruction files in the file system from the digester.
→ Create an opening for outside components to call the digester directly (e.g. via a webservice call)
Steps
Phase 1: Proof Of Concept of Direct Instruction Injection
...
The Digester expect a target to an api for calling the webservice on change to status.
Here’s the API (prototype):
| Code Block |
|---|
POST .../job
Body: {Job JSON}
Creates a Job described the the given body JSON
PATCH .../job/{job_id}
Body: {Job Prototype JSON}
Updates the job with the given prototype |
...