Versions Compared

Old Version 1

changes.mady.by.user Andreas Schmidt

Saved on

compared with

New Version Current

changes.mady.by.user Andreas Schmidt

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents


Storages are where Vidispine will store any files that are ingested/created in the system. All files on a storage location will get an entry in the Vidispine database, containing state, file size, hash etc. This is to keep track of any file changes.

...

This can be controlled, either on system level or on storage level, by the storage metadata scanMethodAlgorithm. Possible values are:

  • VISITOR - use file visitors if possible, otherwise iterator. This is the default.

  • ITERATOR - use file iterators

  • LEGACY

Files

When are files scanned?

...

The following URI schemes are defined.

file

Syntax:

file:///{path}

Example:

file:///mnt/storage/, file:///C:/mystorage/

Note:

The URI file://mnt/storage/ is not valid! (But file:/mnt/storage/ is.)

ftp

Syntax:

ftp://{user}:{password}@{host}/{path}

Example:

ftp://johndoe:secr3t@example.com/mystorage/

Add query parameter passive=false to force active mode. To set the client side ports used in active mode, set the configuration property ftpActiveModePortRange, the value should be a range, e.g. 42100-42200.

...

Info

New in version 4.17: For some servers using a basic implementation of ftp and which does not support some of the commands often found, e.g. listing a directory without having to step into it first, the query parameter serverType=basic can be used if issues with connecting and listing files are experienced. This will in some cases provide a better compatibility.

sftp

Syntax:

sftp://{user}:{password}@{host}/{path}

Example:

sftp://johndoe:secr3t@example.com/mystorage/

When using a private key to authenticate:

Syntax:

sftp://{alias}@{host}/{path}

Example:

sftp://examplesftp@example.com/mystorage/

Note:

Currently only PKCS#1 keys are supported; using vault or local secrets.

http

Syntax:

http://{user}:{password}@{host}/{path}

Example:

http://johndoe:secr3t@example.com/mystorage/

Note:

Requires WebDAV support in host.

https

Syntax:

https://{user}:{password}@{host}/{path}

Example:

https://johndoe:secr3t@example.com/mystorage/

Note:

Requires WebDAV support in host.

omms

Syntax:

omms://{userId}:{userKey}@{hostList}/{clusterId}/{vaultId}/

Example:

omms://c2f6a2f4-6927-11e1-cc94-ab94bd11183f:some%20secret@10.0.0.3,10.0.0.4/4255378f-dc73-fca3-e40d-5726008b3dac/0a49472d-15d4-12f1-862e-f9708d49267e/

Note:

Object Matrix Matrix Store.

s3

Syntax:

s3://{accessKey}:{secretKey}@{bucket}/{path}

Example:

s3://KDASODSALSDI8U:RxZYlu23NDSIN293002WdlNyq@mystore/storage1/

If no access key is provided, then the credentials will be read from the AwsCredentials.properties file in the

Anchor
index-2
index-2
credentials directory, if one exists. Else, credentials will be read from the default locations used by the AWS SDK.

...

The following query parameters are supported:

Anchor
s3-uri-params
s3-uri-params

endpoint

The endpoint that the S3 requests will be sent to.

See Regions and Endpoints in the Amazon documentation for more information.

region

The region that will be used in the S3 requests.

See Regions and Endpoints in the Amazon documentation for more information.

signer

The algorithm to use to signing requests. Valid values include S3SignerType for AWS signature v2, and AWSS3V4SignerType for AWS signature v4.

Default:

Signature algorithm will be selected by region.


Info

For Version 4 Signature only regions (Beijing and Frankfurt) to work, the endpoint or region parameter must be set. Example:

  • s3://frankfurt-bucket/?endpoint=s3.eu-central-1.amazonaws.com

  • s3://frankfurt-bucket/?region=eu-central-1


...

storageClass

The default Amazon S3 storage class that will be used for new files created on an Amazon S3 storage. Can be either standard, infrequent or reduced

Default:

standard


sseAlgorithm

The encryption used to encrypt data on the server side. See Server-Side Encryption. By default no encryption will be performed.

This sets the x-amz-server-side-encryption header on PUT Object S3 requests.

Example:

AES256


sseKeyId

The encryption used to encrypt data on the server side. See Server-Side Encryption. By default no encryption will be performed.

This sets the x-amz-server-side-encryption-aws-kms-key-id header on PUT Object S3 requests.

If the sseAlgorithm is present and has the value of aws:kms, this indicates the ID of the AWS Key Management Service (AWS KMS) master encryption key that was used for the object.

The KMS KEY you specify in the policy must use the arn:aws:kms:region:acct-id:key/key-id format.

Example:

arn:aws:kms:us-west-2:360379543683:key/071a86ff-8881-4ba0-9230-95af6d01ca01


accelerate

Enable S3 Transfer Acceleration.

Default:

false


Info

For S3 Transfer Acceleration to work, the endpoint or region parameter must be set. Also make sure that transfer acceleration is enabled on the bucket.

Other S3 compatible endpoints may not support transfer acceleration.


Anchor
s3-retrievaltier
s3-retrievaltier

retrievalTier

The default Glacier retrieval tier to use when restoring the file. Can be set to either Expedited, Standard or Bulk. See Restoring Archived Objects for more information.

Anchor
s3-ssl
s3-ssl

ssl

Vidispine is by default using SSL when communicating with S3. Set to false to disable SSL support.

Default:

true


Anchor
s3-role-arn
s3-role-arn

Info

New in version 21.3.

roleArn

The role ARN to try to assume to access the content of the bucket.

In order to be able to access buckets and content across accounts, it is now possible to supply a role ARN that VidiCore will try to assume to access the data.

roleExternalId

The (optional) external id attached to the role specified as roleArn

stsRegion

(optional) The region to where calls to assume role are made (AWS STS). This should be set to something as close to your system as possible to reduce latency and get better reponse times (example: eu-west-1, us-east-2).

Info

  • When a role is being assumed VidiCore will need to contact AWS Security Token Service (STS) in order to complete the request. Unless the system is running on EC2/ECS the best practice when using role ARN for S3 storages would be to make sure the stsRegion parameter is being used. If this is not supplied, VidiCore will take more time trying to figure out which region to call (see below).

  • If no region is specified OR VidiCore is NOT running on EC2/ECS, VidiCore will fallback to the AWS default region which would be us-west-2. This is not recommended for optimal performance.


ds3

Syntax:

ds3://{accessKey}:{secretKey}@{bucket}/{path}

Example:

ds3://KDASODSALSDI8U:RxZYlu23NDSIN2Nyq@bucketname/?endpoint=http://blackpearl-endpoint

Note:

Spectra BlackPearl Deep Storage Gateway.

The following query parameters are supported:

Anchor
ds3-uri-params
ds3-uri-params

endpoint

The endpoint of the BlackPearl service. This is mandatory.

chunkReadyTimeout

The maximum time (in seconds) of waiting for BlackPearl to prepare the target data chunk, or an EOF will be returned.

Default:

1800


checksumType

If set, a client-side checksum will be computed and sent to BlackPearl gateway for data integrity verification. Supported checksum types are: md5, crc32 and crc32c.

Default:

Empty, no checksum will be sent.


azure

Syntax:

azure://:{accessKey}@{accountName}/{containerName}

Example:

azure://:KLKau23dEE02WdlLiO@companyname/container1/

gs

Google Cloud Storage.

Using a P12 private key:

Syntax:

gs://{privateKeyAlias}@{bucket}/?project={project}&account={account}

Example:

gs://test-key-p12@test-bucket/?project=12345&account=67890

Using a JSON private key:

Syntax:

gs://{privateKeyAlias}@{bucket}/?project={project}

Example:

gs://test-key-json@test-bucket/?project=12345

Using an OAuth2 access token:

Syntax:

gs://:{accessToken}@{bucket}/?project={project}

Example:

gs://:abc123@test-bucket/?project=12345

Using the credentials file specified in the GOOGLE_APPLICATION_CREDENTIALS environmental variable:

Syntax:

gs://{bucket}/

Example:

gs://test-bucket/

universal

A universal URI is used to create a universal storage method. A universal storage method does not have a root URI, instead all files contain their own absolute URI.

...