Transcoders [VC 21.3 GEN]

When you import items the Vidispine transcoder will be used to detect the type of media that is being imported and, of course, to transcode the media to any formats that you have requested.

The common operations performed by the transcoder are:

• Media shape deduction

• Transcoding

• Sequence rendering

• Partial file extraction

• XMP extraction and rewrite

The Vidispine transcoder has a REST API that Vidispine uses to perform the above operations. This API is not described in this document, as it typically should not be accessed directly.

Adding a transcoder

Add a transcoder by creating a new transcoder resource. The resource document should contain information on how to reach the transcoder and what storages the transcoder has direct access to.

POST /resource/
Content-Type: application/xml

<?xml version="1.0"?>
<ResourceDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <transcoder>
    <url>http://transcoder.example.com:8888/</url>
    <directAccess>
      <filter>file:/srv/media/.*</filter>
    </directAccess>
  </transcoder>
</ResourceDocument>

Vidispine checks the status of transcoders continuously in the background. As such, if the configuration is correct you will see that the transcoder shows up as online in a few seconds.

GET /resource/VX-7

<?xml version="1.0"?>
<ResourceDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <id>VX-7</id>
  <transcoder>
    <url>http://transcoder.example.com:8888/</url>
    <directAccess>
      <filter>file:/srv/media/.*</filter>
    </directAccess>
    <state>ONLINE</state>
  </transcoder>
</ResourceDocument>

The Vidispine installer will by default install and configure a transcoder in Vidispine for you, so this step is typically not needed.

Using multiple transcoders

Depending on your license, you may be allowed to use more than one transcoder. To do so, simply add additional transcoders as explained above. Vidispine will submit transcode jobs to the transcoder based on the current number of jobs being processed by the transcoder.

Vidispine will use the transcoder with the least amount of work. If a transcoder goes offline then any transcode job steps using that transcoder will fail and be retried using one of the online transcoders. If all transcoders are offline then jobs will wait for one to become available.

How transcoders perform jobs

A transcoder will perform a job as soon as it is received, and will not schedule jobs for later execution. Vidispine, that is, the user of the transcoder is responsible for scheduling which jobs a transcoder should execute and when they should be executed.

Transcoder job limit

The maxJob setting can be used to limit the number of Vidispine jobs that may use a specific transcoder at the same time. If all transcoders are busy, jobs will be put on WAITING state, with a TranscoderBusy problem. The jobs will restart as soon as any qualified transcoder becomes available again.

 <ResourceDocument xmlns="http://xml.vidispine.com/schema/vidispine">
   <id>VX-7</id>
   <transcoder>
     <url>http://transcoder.example.com:8888/</url>
     <maxJob>5<maxJob>
   </transcoder>
 </ResourceDocument>

A Vidispine job typically triggers multiple transcoder jobs, e.g. shape deductions and transcodes, so in the above case there may still be more than 5 running jobs on a transcoder. They will however all belong to at most 5 jobs in Vidispine.

The above setting works for VSA transcoders as well. VSA users can also add transcoder.maxJob to one of the agent configuration files. For example:

$ cat /etc/vidispine/agent.conf.d/transcoder.conf

transcoder.maxJob=5

The transcoder’s configuration file

The transcoder configuration file config.xml contains default settings for the transcoder and need typically not be modified, as the settings can instead by configured in Vidispine.

Modify the transcoder file

On a Linux system, copy the file /opt/vidispine/transcoder/config.xml to /etc/transcoder-config.xml. Then edit /etc/transcoder-config.xml. The file in /etc takes precedence over the file in /opt/vidispine.

Modify the transcoder resource

On all operating systems, the transcoder configuration can be changed by adding configuration to the resource definition of the transcoder (Adding a transcoder).

Note that port of the transcoder cannot be changed in this fashion.

Modifying the transcoder configuration is this fashion takes precedence over the local configuration file and the global transcoder configuration, see below.

Modify all transcoders

It is also possible to change the configuration of all transcoders, by setting the configuration property transcoderDefaultConfiguration to the XML representation of the transcoder configuration.

Thumbnail settings

The thumbnailResolution element contains the default resolution of the thumbnails produced by the transcoder.

<a:thumbnailResolution>
   <a:width>320</a:width>
   <a:height>240</a:height>
</a:thumbnailResolution>

You can also change the thumbnailing frequency by changing thumbnailPeriod. For example, to thumbnail every 3 seconds:

<a:thumbnailPeriod>
   <a:samples>3</a:samples>
   <a:timeBase>
      <a:numerator>1</a:numerator>
      <a:denominator>1</a:denominator>
   </a:timeBase>
</a:thumbnailPeriod>

If the transcoder does not use a scene change detection plugin, the frequency defaults to once every 10 seconds.

StatsD settings

To have the transcoder send metrics to a StatsD server you can either:

• Enable StatsD using the API, see StatsD

• Update the transcoder configuration with the address and port of the StatsD server:

<a:statsd>
  <a:destination>
    <a:address>127.0.0.1</a:address>
    <a:port>8125</a:port>
  </a:destination>
  <a:prefix>t1</a:prefix>
</a:statsd>

The prefix element configures the prefix to use for each metric. By default this is the transcoder.

Transcoder resources settings

Path to temporary storage

Controls where temporary files are stored. Default is /tmp on UNIX-like systems, or %TEMP% on Windows.

<a:tempPath>/mnt/largetemparea<a:tempPath>

Number of decoding threads

Controls the number of decoding threads. Defaults to 4 for I-frame-only formats. The actual number of threads used depends on codec.

<a:decoderOfferThreads>8<a:decoderOfferThreads>

Number of encoding threads

Controls the number of encoding threads. Defaults to automatic setting. The actual number of threads used depends on codec.

<a:encoderThreads>8<a:encoderThreads>

HTTP buffer sizes

Controls the size of HTTP reads and writes of the transcoder. dataBufferSize controls the maximum number of read bytes in memory. Default is 100 MB. dataBufferWriteSize controls the maximum number of write bytes in memory. Default is 100 MB. dataBufferFlushTime controls the number of seconds written bytes are stored in memory before it is flushed. Default is 4 seconds.

For system that uses segment files and where the transcoder has enough memory, it is recommended to increase these numbers, up to 10 times.

Image processing

To control the memory and disk usage used by the transcoder for image processing, use the <imagemagick> element in the transcoder configuration. The most important settings are listed below, for a complete list, see http://www.imagemagick.org/script/resources.php (under environment variables, used without the MAGICK_ prefix in the transcoder configuration).

Maximum heap usage

<a:imagemagick>
  <a:key>MEMORY_LIMIT</a:key>
  <a:value>1GB</a:value>
</a:imagemagick>

Temporary work area

<a:imagemagick>
  <a:key>TEMPORARY_PATH</a:key>
  <a:value>/var/tmp</a:value>
</a:imagemagick>

The default value is the value set by the general transcoder temporary path, see above. It is recommended that the tempPath setting is used, rather than the imagemagick one.

Operations overview

Zeroconf transcoders

The following is done to remove the need to configure the transcoders directly:

• Vidispine pushes its own license to the transcoder, so that each transcoder does not need a license file of their own.

• The transcoder returns the IP address from where the license was pushed, that is, the IP address of the application server, removing the need for explicitly configuring the reverse address, that is, where the transcoder can reach Vidispine, in most cases.

• In addition, Vidispine generates temporary pre-authorized URIs that are used by the transcoder. This removes the need for entering any application server information in the transcoder configuration file.

Reverse address and NAT

The reverse address does not work if there is NAT or other port forwarding mechanisms between the application server and the transcoder. If so, the address to VS-EA can be overridden in the definition for the transcoder by setting the <reverseAddress> element.

<?xml version="1.0"?>
<ResourceDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <transcoder>
    <url>http://transcoder.example.com:8888/</url>
    <reverseAddress>vs.example.com</reverseAddress>
  </transcoder>
</ResourceDocument>

The rules for how the address to Vidispine is determined are as follows:

1. If the configuration property apiNoauthUri is set, it is used for all transcoders.

2. If the configuration property apiNoauthPort is set, it is used for together with the detected or manually set reverse address.

Transcoder’s access to media

By default, the transcoder accesses non-file-schema media through the application server. This has several advantages:

• The same user is used for all file access.

• Possibility for support for extended file attributes and permissions.

• Support for other file systems (URI schemes).

Streaming the media puts some extra load on the application server. Some tuning might be necessary.

The transcoder resource in Vidispine can be set up to access files directly. By adding a directAccess element to the transcoder resource, Vidispine will let the transcoder access the media directly. If no directAccess elements are present, an implicit

<directAccess>
   <filter>file:.*</filter>
</directAccess>

is added. In order to tell Vidispine that all files should go via the application server, add an

<directAccess>
   <filter>NO_MATCH</filter> <!-- dummy regular expression that does not match anything -->
</directAccess>

Growing files

For all file systems that supports read-while-write, and for container formats that are built for streaming (e.g. MXF), growing file is supported when streamed through the application server. If growing files is required to local files with the file scheme, a directAccess/NO_MATCH element as per above must be added to the resource configuration.