Document toolboxDocument toolbox

Vidispine

Agents [ENT 21.1 ARC]

Owned by Andreas Schmidt
May 31, 2021
3 min read

Overview

VidiFlow agents are a core concept in an Enterprise MAM Solution. An agent implements specific functionality (e.g. for copying files or accessing a third party system) which can be orchestrated by BPMN Workflows [ENT 21.1 ARC].

To make its functionality accessible to workflows, each agent implements one or more task contracts. A task contract encapsulates a specific functionality. It defines input and output parameters as well as boundary events (both interrupting and non-interrupting) for this specific functionality. An agent must provide at least one task contract. It may provide multiple task contracts to make variations of the functionality available in an easy-to-understand and consistent manner. An example would be an agent which uses a specific library to make a 3rd-party system accessible by VidiFlow. Different task contracts would make different functional aspects usable in a workflow but would still bundle the technical implementation in a single agent. Multiple task contracts per agent avoid the need for many small agents using the same library and thus reduce overhead; and they would allow writing clear task contracts without many switches for different options which would make the contracts hard to understand.

VidiFlow comes with a set of standard agents which are available in workflows out of the box. Further agents can be added to an existing system via Use-Case Packages [ENT 21.1 ARC] to add specific functionality to it.

Deployment Time

When an agent is deployed to the system the agent implementation is started in the Kubernetes cluster via a Kubernetes deployment.

Startup Time

During startup the agent does several things:

  1. All task contracts of the agent are published to ConfigPortal.

  2. If the agent is part of a Use-Case Packages [ENT 21.1 ARC] the corresponding use-case pages are published to ConfigPortal.

  3. An task-contract-specific message queue is created in the RabbitMQ Message Bus [ENT 21.1 ARC].

  4. The agents registers to all its queues for receiving messages from the Camunda Broker (see BPMN Workflows [ENT 21.1 ARC]).

Execution Time

VidiFlow agents receive their tasks via the message bus. There is one queue for each task contract. As an agent may implement multiple tasks, this agent may be responsible for multiple queues. Each agent can receive and execute up to 8 messages per task contract in parallel. If the agent supports different versions of a contract, the 8 parallel executions apply to all versions of the contract.

For all Kubernetes-based agents the messages flow is like this:

  1. Camunda executes a workflow task.

  2. The workflow task sends a message to the task-specific queue on RabbitMQ. This unprovisioned message contains the input parameters as received from the workflow. In particular, URIs are passed in as VidiFlow URIs.

  3. The agent implementing the task receives the message from the task-specific queue and performs provisioning of the message in a dedicated task provisioner which is a separate part of the agent. It:

    1. translates environment-independent Platform URIs to environment-dependent physical URIs (e.g. file or http URIs);

    2. obtains configuration parameters from ConfigPortal which are required for executing the actual task (the task itself by design must not access ConfigPortal - only the task provisioner is allowed to do so).

  4. The agent executes the task.

See BPMN Workflows [ENT 21.1 ARC] for diagrams.

Note: Although the task provisioner is part of the agent it is kept strictly separated from the code executing the task. This design decision allows automated testing of the task execution code without the need for a running workflow engine or ConfigPortal.

For some older agents still running on ServiceFabric the message flow still is like this:

  1. Camunda executes a workflow task.

  2. The workflow task sends a message to the task-specific queue on RabbitMQ. This unprovisioned message contains the input parameters as received from the workflow. In particular, URIs are passed in as Platform URIs.

  3. The agent implementing the task receives the message from the task-specific unprovisioned queue, performs provisioning of the message, and writes a new provisioned message to the tasks-specific provisioned queue.

  4. The responsible agent retrieves the message form the task-specific provisioned queue and executes the task.

Messages are marked as completed and removed from the queue only after task execution has finished (either successfully or with an error).

Upon completion the VidiFlow agent uses the message bus to communicate the task result back to the VidiFlow workflow engine. The provisioning concept also is applied to the output messages as well as for the input messages.

See BPMN Workflows [ENT 21.1 ARC] for diagrams.

Task progress updates are also sent via the message bus.