Vidispine
Release Notes [VF 21.1 RN]
- Andreas Schmidt
The following items on the list encompass breaking changes, features, and fixes that are relevant for the major release.
Release Notes are divided into the following sections:
- 1 Release 21.1.257
- 1.1 Breaking Changes
- 1.2 Docker Migration of Services and Agents
- 1.2.1 Services
- 1.2.2 VidiCore Agents
- 1.2.3 Essence Agents
- 1.2.4 Media Agents
- 1.2.5 Removed Services
- 1.3 Setup and Operations
- 1.4 Features
- 1.4.1 New and Adapted Agents and Tasks
- 1.4.1.1 List of New Tasks
- 1.4.1.2 List of Adapted Tasks
- 1.4.2 Metadata Editing
- 1.4.3 Workflow Monitor
- 1.4.4 File Upload UI
- 1.4.5 Clean up of Workflows in ConfigPortal
- 1.4.6 Redundancy of Camunda Broker and WF API
- 1.4.7 Performance and stabilization improvements
- 1.4.8 Description for Workflows in ConfigPortal
- 1.4.9 Hangfire dashboard
- 1.4.10 Deletion Monitor
- 1.4.11 Workflow Designer: input variable can be selected as "workflow object id"
- 1.4.12 RabbitMQ Connection Handling Improvements
- 1.4.13 Other fixes and changes
- 1.4.1 New and Adapted Agents and Tasks
Release 21.1.257
Breaking Changes
Docker Migration of Services and Agents
Services
The following table shows all VidiCore services which have been migrated from Service Fabric to Docker for 21.1.
It is important to note that these services are no longer available in Service Fabric and need to be replaced by the corresponding Docker-based services when updating from 20.2 to 21.1.
Service | Previous Name in Service Fabric | Name of Pod in Kubernetes |
|---|---|---|
File Notifications | fabric:/Platform.VidispineServices/FileNotifications | vidispine-agents-vidispine-agents-file-noti |
VidiCore Agents
The following table shows all VidiCore Agents which have been migrated from Service Fabric to Docker for 21.1.
It is important to note that these agents are no longer available in Service Fabric and need to be replaced by the corresponding Docker-based agent when updating from 20.2 to 21.1.
The names and contracts of the service tasks for usage in VidiFlow workflows were kept the same as in the previous release to make the migration as simple as possible.
Agent | Previous Name in Service Fabric | Name of Pod in Kubernetes |
|---|---|---|
Register Item | fabric:/Platform.VidispineServices/RegisterItemAgent | vidispine-agents-vidispine-agents-register-item |
Get Metadata | fabric:/Platform.VidispineServices/GetMetadataAgent | vidispine-agents-vidispine-agents-get-metadata |
Deletion Lock Management | fabric:/Platform.VidispineServices/DeletionLockManagementAgent | vidispine-agents-vidispine-agents-deletion-lock-manage |
Manage User Group Right | fabric:/Platform.VidispineServices/ManageUserGroupRightAgent | vidispine-agents-vidispine-agents-manage-usergroup-rig |
Set Object Metadata | fabric:/Platform.VidispineServices/SetObjectMetadataAgent | vidispine-agents-vidispine-agents-set-metadata |
Get Shape From Item | fabric:/Platform.VidispineServices/GetShapeFromItemAgent | vidispine-agents-vidispine-agents-get-shape |
Export Item | fabric:/Platform.VidispineServices/ExportItemAgent | vidispine-agents-vidispine-agents-export-item |
Transfer File Vidispine | fabric:/Platform.VidispineServices/TransferFileVidispine | vidispine-agents-vidispine-agents-transfer |
Get File Information | fabric:/Platform.VidispineServices/GetFileInformationAgent | vidispine-agents-vidispine-agents-get-file-information- |
Get File By Shape Tag | fabric:/Platform.VidispineServices/GetFileByShapeTagAgent | vidispine-agents-vidispine-agents-get-file-by-shape |
Manage Poster | fabric:/Platform.VidispineServices/ManagePoster | vidispine-agents-vidispine-agents-manage-poster |
Read File Data | fabric:/Platform.VidispineServices/ReadFileDataAgent | vidispine-agents-vidispine-agents-read-file-data |
Render Item Sequence | fabric:/Platform.VidispineServices/RenderItemSequenceAgent | vidispine-agents-vidispine-agents-render-item |
Update Metadata | fabric:/Platform.VidispineServices/UpdateMetadataAgent | vidispine-agents-vidispine-agents-update-metadata |
Find Item | fabric:/Platform.VidispineServices/FindItemAgent | vidispine-agents-vidispine-agents-find-item |
Get House Format Shapetag | fabric:/Platform.VidispineServices/GetHouseFormatShapetag | vidispine-agents-vidispine-agents-get-houseformat-shap |
Import Side Car | fabric:/Platform.VidispineServices/ImportSideCarAgent | vidispine-agents-vidispine-agents-import-side-car |
Manage Collection | fabric:/Platform.VidispineServices/ManageCollection | vidispine-agents-vidispine-agents-manage-collection |
TransferFileVidispine | fabric:/Platform.VidispineServices/WriteVidispineFileContentAgent | vidispine-agents-vidispine-agents-transfer |
GetExistingPartialOfItem | fabric:/Platform.VidispineServices/GetExistingPartialOfItem.ServiceFabricHost | vidispine-agents-vidispine-agents-get-existing-partial |
ShapeExistsOnStorage | fabric:/Platform.VidispineServices/ShapeExistsOnStorage | vidispine-agents-vidispine-agents-shape-exists-on-storage |
Test Agent | fabric:/Platform.Tools/TestAgent | tools-tools-test-agent |
Essence Agents
The following table shows all Essence Agents which have been migrated from Service Fabric to Docker for 21.1.
It is important to note that the agents in Service Fabric will still be available in release 21.1, but will be removed in the subsequent release. To access the migrated agents form a VidiFlow workflow, the service tasks must be replaced by the corresponding task with the post fix “ViaLinux”. Both measures are in place to support a step-wise migration from release 20.2 to 21.1.
VidiFlow Agents offer different task versions when adding new functionality. Newer version usually contain all existing functionality. The migrated essence agents mostly have just one task version available - when migrating, it is recommended to always use the latest version.
Agent | Previous Name in Service Fabric | Name of Pod in Kubernetes |
|---|---|---|
Create Folder | fabric:/Platform.Essences/CreateFolderAgentHost | essence-agents-essence-agents-create-folder |
Delete Folder | fabric:/Platform.Essences/DeleteFolder | essence-agents-essence-agents-delete-folder |
File Exists | fabric:/Platform.Essences/FileExists | essence-agents-essence-agents-file-exist |
Read File Content | fabric:/Platform.Essences/ReadFileContentAgent | essence-agents-essence-agents-read-file-content |
Write File Content | fabric:/Platform.Essences/WriteFileContentAgent | essence-agents-essence-agents-write-file-content |
File Delete | fabric:/Platform.Essences/FileDeleteHost | essence-agents-essence-agents-file-delete |
File Info Reader | fabric:/Platform.Essences/FileInfoReader | ssence-agents-essence-agents-file-info-reader |
Platform URI Agent | fabric:/Platform.Essences/PlatformUriAgent | essence-agents-essence-agents-platform-uri |
Timecode Management Agent | fabric:/Platform.Essences/TimecodeManagementAgent | essence-agents-essence-agents-time-code |
File Transfer Host | fabric:/Platform.Essences/FileTransferHost | essence-agents-essence-agents-file-transfer |
CreateFolderAgentHost | fabric:/Platform.Essences/CreateFolderAgentHost | essence-agents-essence-agents-create-folder |
XSL Transformation Agent | fabric:/Platform.Essences/XslTransformationService | essence-agents-essence-agents-xsl-transformation |
Workflow Management Host | fabric:/Platform.Essences/WorkflowManagementAgentHost | essence-agents-essence-agents-vpms2-workflow-managemen |
Media Agents
The following table shows all Media Agents which have been migrated from Service Fabric to Docker for 21.1.
Agent | Previous Name in Service Fabric | Name of Pod in Kubernetes |
|---|---|---|
Transcoder S4M Agent | fabric:/Platform.MediaframeworkServices/TranscoderS4MAgent | media-agents-media-agents-transcode |
TrimMxf | fabric:/Platform.MediaframeworkServices/TrimMxf | media-agents-media-agents-trimmxf |
MediaInfoAgent | fabric:/Platform.MediaframeworkServices/MediaInfoAgent | mediaanalysis-mediaanalysis-mediainfo |
NormalizeMxfAgent | fabric:/Platform.MediaframeworkServices/NormalizeMxfAgent | media-agents-media-agents-normalizemxf |
PartialRestoreMxf | fabric:/Platform.MediaframeworkServices/PartialRestoreMxf | media-agents-media-agents-partialrestore |
StitchVideoAgent | fabric:/Platform.MediaframeworkServices/StitchVideoAgent | media-agents-media-agents-stitchvideo |
CreateIndexFile | fabric:/Platform.MediaframeworkServices/CreateIndexFile | media-agents-media-agents-createindexfile |
GetPartialRestoreOffsets | fabric:/Platform.MediaframeworkServices/GetPartialRestoreOffsets | media-agents-media-agents-getpartialrestoreoffsets |
Removed Services
The following services was deleted from the VidiCore deployment:
XLS Transformation Service
The following agents were not migrated to Docker-based deployment and are expected to be deleted with the next release:
Platform.Essences/MetadataAgent
Platform.Essences/MetadataDefinitionService
Platform.Essences/SendToService
Platform.Essences/FileStateAwaiterHost
Platform.Essences/FileStateService
Setup and Operations
Component updates
The following central components were updated and will be shipped in the noted version:
Component | Version |
|---|---|
VidiCore | 5.6.1 |
ConfigPortal | 21.1.47 |
AuthService | 21.1.67 |
MediaFramework (for K8S agents) | 21.1.7 |
MediaFramework (for SF agents) | 21.1.8 |
Grafana | 7.3 |
RabbitMQ | 3.8.14 |
Prometheus | 2.22 |
Fluentd | 1.12.2 |
Nginx Ingress | 0.44.0 |
New database for Camunda Broker Redundancy
The Camunda Broker can now be operated in multiple instances for redundancy as described in Redundancy of Camunda Broker.
To allow the redundant operation, a new database “VPMS_Platform_CamundaBroker” is created in VidiFlow setup.
New database for Deletion Monitor
VidiFlow provides now a Deletion Monitor. To support this, a new database “VPMS_Platform_DelMonitor” is created in VidiFlow setup. For more details see User guide Deletion Monitor [VF 21.1 UG].
Change log-level during run-time
It is now possible to change the log-level of VidiFlow services and agents running in Kubernetes at runtime. To do so, edit the configmap for the respective service and amend the appsettings settings. An example settings looks like the following:
{
"UniqueModuleName": "WorkflowEngineAPI",
"Vpms3Logging": {
"LogLevel": {
"Default": "Debug",
"Hangfire": "Information"
}
}
}PostgreSQL support
PostgreSQL and Azure Database for PostgreSQL are now officially supported as relational database for all VidiFlow internal databases.
Grafana dashboards
A collection of useful Grafana dashboards will be deployed with VidiFlow. These are intended to serve as patterns to be able to easily map the metrics needed for the respective system in Grafana. The following screenshot show an example view:
Features
New and Adapted Agents and Tasks
All tasks can be selected via VidiFlow’s Workflow Designer. The tasks and their input and output parameters are documented in the Workflow Designer itself after choosing a task.
List of New Tasks
Detach File
The new task “Detach File” can be used to detach a file from an item in VidiCore. If the last file of a shape was detached, the shape will also be deleted. If the task is called for a file which is not attached, it will run successfully.
Trim with Vantage
A new task is implemented in the Telestream Vantage agent which allows trimming of videos.
Search for Vidicore collections
A new task was implemented to return all collections stored in VidiCore matching a given ItemSearchDocument.
List of Adapted Tasks
Trim MXF File
The task now allows one to specify sub-folder for the target file. Folders will be created if not exisiting
The task now provides an optional input parameter 'ItemToAttachPlatformUri'. If set, the task does not create an item but attach the file to the item transmitted in the new parameter.
Additionally, it provides three non-interrupting message boundary events (FileToAppear, FileOpen, FileClosed).
Remark: Most changes of Trim MXF File only apply on the Core / Docker version.
Create Image File From Poster
The Task “Create Image File From Poster” now supports TargetFileName and Subfolder as input parameters.
GetShapeFromItem
Improved support of differentiation of multiple shapes (see task description in WF Designer.
Delete
If the delete task is called for file deletion and the file is on a storage which is not accessible by VidiCore, then the file is unregistered instead.
If the delete task is called on a file and the last file of an shape is deleted, also the shape is deleted.
If the last file of a shape is detached, the shape will be deleted.
If the last file of an item was deleted and the item is not protected by other deletion locks, the item will be deleted (since task version 2.1).
Unregister File
If the last file of an item was unregistered and the item is not protected by other deletion locks, the item will be deleted (since task version 1.1).
TranscoderS4M
The task TranscoderS4MViaLinux (only Docker-based version) now support S3 and Azure blob storages as source and target. Storages need to be configured in the CP storage configuration as VidiCore storage.
Metadata Editing
Enum Value show and keep outdated values
When enum values are outdated, they now can still be shown in the UI and it is also allowed to keep their values without editing.
Auto generate values for input fields
The metadata UI builder can now access customer specific back-end services to calculate form metadata based on other metadata. After clicking an the valuation button, all form data will be send to the service. Teh service and validate the data, change one or more metadata fields and send it back to the UI which updates the data. The endpoint of the service can be configured in CP.
For detailed information please refer Create Booking Component [VF 21.1 IG] .
Workflow Monitor
VidiFlow contains a new workflow monitor UI that a new tree-list-view to monitor main- and sub-workflows in the same view and brings detailed filtering options. Role-based rights management is also available to strip down user rights for different groups of users.
For detailed information please refer to the user guide. All links are changed to the new version. The old version is deprecated, but can be reached in VidiFlow Portal named as “Workflow Monitor Classic”.
File Upload UI
Files bigger than 2 GB cannot be uploaded via the File Upload UI. Previously, the upload failed after selecting a file. Now, there will be an error message if a user selects a file bigger than 2 GB and the upload process will not initiate.
Clean up of Workflows in ConfigPortal
ConfigPortal comes with a new cleanup feature, which allows one to remove workflows with all their version or single workflow version completely from the system. Please see the Enterprise Operations Guide Clean up of Workflows in ConfigPortal [ENT 21.1 OG] .
Redundancy of Camunda Broker and WF API
Prior to 21.1, the Camunda Broker component could not be installed in multi instances for redundancy. In this version, this is finally possible and highly recommended for productive scenarios. To allow the redundant operation, a new database “VPMS_Platform_CamundaBroker” is created in VidiFlow setup.
The WF API was split into two services, one for handling all API call and the other one for handling SignalR messages for clients. Due to this, the WF API service can should should run redundantly in productive environments.
Performance and stabilization improvements
VidiFlow has experienced various performance and stabilization improvements in key components:
Item # | Item |
|---|---|
182140 | Agents perform OIDC token validation against AuthService without additional API calls |
182141 | Agenst cache OIDC tokens to reduce AuthService load |
176391 | Most VidiFlow REST clients were migrated to a new HTTP-library to avoid network socket exhaustion |
181926, 184159 | The workflow deployment behavior of the system was changed to increase performance on calling many sub-workflows. After staging a workflow from CP it will now immediately deployed |
185627 | The workflow indexing to Elastic was changed in a way that the workfow indexing is proceeded without reading from the Camunda history tables to reduce the load on those tables and due to that improve the general performance of the Camunda workflow engine |
182485, 185553 | Rebuilding the workflow index (bulk re-index) was improved performance wise |
182059 | The prefetch settings for RabbitMQ where changed in all VidiFlow agents. This leads to a better distribution of tasks among several agents |
| Several system components have better and reduced log output now |
Description for Workflows in ConfigPortal
The new column “Description” is now available for each workflow in the workflow list visible in ConfigPortal. A description can be added via a new API call for each workflow version.
In upcoming VidiFlow releases, this field will be filled with a possible description for each workflow, entered in the Workflow Designer.
Hangfire dashboard
Hangfire is used to execute recurring tasks in VidiFlow like workflow re-indexing. VidiFlow now includes a dashboard for system administrators to better monitor those jobs.
Please refer to the official Hangfire documentation for detailed usage.
Deletion Monitor
System administrators or material managers customer-side often want to determine when and why assets where deleted or will be deleted from the system’s housekeeping. The VidiFlow Deletion Monitor allows one to monitor information on the deletion of items, collections and files. Additionally, different information about deletion locks are stored.
For detailed information please refer to Deletion Monitor [VF 21.1 UG] in the VidiFlow User Guide.
Workflow Designer: input variable can be selected as "workflow object id"
The VidiFlow Workflow Monitor now supports to show the most important workflow input parameter as “Workflow Object ID”. To be able to show, this variable has to be selected as workflow object id on design time. The Workflow Designer allows to do so in the section Workflow Parameters:
The selection is limited to fields with subtype PlatformURI.
RabbitMQ Connection Handling Improvements
Apart from RabbitMQ server upgraded to version 3.8, improvements are also made for the connection handling to the RabbitMQ server to avoid connection overhead. Each module now should uses two connections to RabbitMQ (one for publishing and another one for subscription). This will reduce the connection load on the RabbitMQ server.
It is also now possible to identify which connection is used by which module. This can be seen as the ClientName in the RabbitMQ connections in the management page.
Other fixes and changes
Item # | Item |
|---|---|
179013 | WorkflowHandling and CP API calls when using "Latest" |
179512 | Additional Parameter for Vantage Agent Needed: SourceStorageUri |
179857 | Critical Bug in CP - Staging switches/sets active environment! |
180064 | S4MTranscoder 60s break |
172786 | Agent to retrieve item sequence needed to support stitching on non StationFormat Items |
179071 | Camunda: Deployment of MS SQL scripts for DB optimization |
180490 | Improve rabbitmq notification plugin handling for Vidispine |
172602 | Special character handling in Metadata definitions |
180523 | Vantage agent must use cluster-internal rabbitmq endpoint |
180875 | Vantage Agent Configurations - RetryCount, JobTimeout, WorkflowForward Timeout |
182409 | Vantage Agent Configuration: Remove option to configure multiple endpoints |
181205 | ActiveMQ memory settings is not following the MaxRAMPercentage settings |
181209 | Fluentd is not picking up new logs |
181388 | WF Monitor - Missing terminated state in Wf state filter selection |
180732 | NPM Package: ReadOnly for Edit Metadata layout for Collection |
180901 | Enable thumbnail caching in nginx ingress controller |
181165 | WF Monitor: Fix for display problems |
181215 | Logging: SF-Services: Reduce Logging of REST API in Debug Level |
181274 | ConfigPortal: Improve performance and reduce process memory |
181373 | Workflow Index Service: Revise log messages |
181466 | Add Options to delay HouseKeeping Workflow trigger |
181470 | Update UpdateMetadata and SetObjectMetadata task to support "onlyReturnChanges" vidicore metadata update |
181498 | GetMetadata agent: Reduce VidiCore load |
181409 | Special characters in password leads to script problem in db creation |
181555 | The workflow is still show running when It reach Terminated end event |
181606 | Vidispine database purging cleanup resources not sufficient for productive system |
181645 | RuleAPI randomly fails sometimes with "error executing api request" |
181893 | Booking layout not updated after changes in CP |
179514 | MediaPortal Notifier does not receive Metadata Updates when __deletion_lock_expiry is updated in VS |
181838 | CP: Editing, Workflow list stuck in loading state |
181933 | Workflow state is not updated |
182023 | TranscoderS4M: Does not shutdown MediaFramework (Service Fabric version of Agent) |
182163 | IsRedundant (for Vantage) does not work properly in CP |
182355 | WorkflowAPI: Concurrent starts for the same workflow version fail |
182413 | CamundaBroker: Improve log message |
182503 | Removing WF contract causing indexing not working properly |
183298 | Agent_GetShapeFromItem fails with two shapes of same type attached to item |
184100 | Escape spaces in filenames in URIs, Create Image from Poster (ManagePoster agent) |
184117 | TargetFileName and Subfolder as Inputparameter for "Create Image File From Poster" Task |
183247 | Agent_TrimMxf changes Audio Channels of File |
184386 | Agent_TrimMxf is not writing Subtitle to target File |
184336 | Task "Export Item" should fail if Source File does not exist |
182356 | Camunda plugin: Reduce log output |
184591 | FC: WorkflowAPI: Limit size of queues |
180742 | FileUpload: Prevent uploading of file bigger then 2GB |
181472 | Handling of DateTime Fields in WF input parameter mappings |
181868 | Possibility to change log-level per component during run-time |
182059 | Add Prefetch Settings for RabbitMq |
181490 | Research Performance Problem with Call Activity |
181872 | ConfigPortal write access to single instance only |
182124 | ConfigPortal data loss due to failed push |
169079 | MP Connector: Sync Vidispine Subtitle Metadata Fields and Groups from Vidispine |
185828 | Failed to write to VidiCore with Metadata Enum with empty values |
183747 | Workflow Cleanup Cron Job fails with DB Query |
176671 | Automatic cleanup of ACT_RU_METER_LOG table in ProcessEngine DB |
185389 | Change naming of key/value metadata for partial items in collections |
180877 | WF API accesses CP API over Ingress |
182660 | Change Camunda Link in Platform Portal to Public Ingress |
169854 | FC: Logging: Optimize fluentd logs for VidiCore and Camunda |
181533 | Camunda metrics cleanup job fails during execution |
186019 | Workflow are not set to Incident when there's a incident Task |
186028 | Special characters problem with colon in Metadata Editor |
186143 | Metadata editor: Reload of values doesn't work properly |
181472 | Handling of DateTime Fields in WF input parameter mappings |
183782 | WF API won't boot due to misconfiguration in 21.1.1-beta.280 deployment |
180388 | Update helmchart for agent/service to support auto update license to CP |
186559 | Licensing: Add packages.config for 3rd party components |
179455 | Workflow Index unable to recover when problem with database connection |
187177 | Move Platform.Portal from Platform Chart into Platform.Tools helmchart |
182620 | MFT targets and storages are not sorted |
186672 | MFT-Mask: email-field to small |
184182 | MP-Upload PROD: Metadata mask can be closed too easily |
185840 | Upload Dialog looks weird |
187013 | Vantage agent: remove obsolete helm values |
187276 | Create Image From Poster Error to trigger ImageForPosterAlreadyExists |