Generic Video Digitisation Workflow

Last updated: 27 January 2009
Published in: Digitising analogue media
Tags: analogue collections | digitisation | video

Comments (0)

Summary

This introductory document is aimed at smaller video archives or paper-based collections which have a video element. This document may also prove beneficial to specially funded video digitisation projects. We shall look at ways in which best practice can be observed and incorporated into each stage of even the smallest video digitisation project, from selecting an initial digital format to delivery.

Background

For some time it has been possible to identify a ‘best practice’ for the digitisation of photographs and much of this advice is just as applicable to the digitisation of video materials. Until recently though, approaches to the digitisation of video were largely in response to the deterioration of analogue originals. With resources tied up in a kind of ‘digital triage’ it was not possible to proactively establish an efficient workflow specifically for digitising moving images.
Now though, several influential documents such as Prestospace’s Preservation Guide, the AHDS (Arts and Humanities Data Service) document Moving Image and Sound Archiving Survey, and the IASA (International Association of Sound and Audiovisual Archives) publication Guidelines on the Production and Preservation of Digital Objects have set precedents from which we can begin to identify best practice for the digitisation of analogue moving images. Large or small, the major investment of a video digitisation project is likely to be in the form of staff time. Because analogue videotape has to be captured in real time, re-capturing should be avoided. This demands an effective and streamlined workflow.

Establishing a workflow

Design of a workflow begins with reference to well-defined project aims. Defining those aims is the joint responsibility of project managers and key stakeholders. Of course, hands-on digitisation staff also have a vital role to play in defining project aims by offering advice and undertaking ‘feasibility’ studies. Results from these trial runs (for instance, the number of videotapes which can be digitised in a day) allow project managers to realistically shape the expectations of stakeholders into achievable project deliverables.
It is important though, to delineate between the feasibility study phase and initiation of the digitisation phase proper. Digitisation is a resource intensive process which cannot begin without first clearly established required project outcomes.
Your day-to-day workflow should be robust enough to allow most materials to be handled in a standardised way while allowing time to be spent on exceptions. It should be flexible enough to allow future refinement as experience grows. Crucially, your working processes should be recorded in detail in a ‘workflow manual’ which acts as a clear guide to all members of the team.

Precepts

With several hundred different formats of analogue video and an infinite amount of possible project aims it is not possible to offer a highly detailed one-size-fits-all workflow for the digitisation of analogue video. Each project will have its own unique aspects. It is possible however, to offer a generic workflow (see below) for the digitisation of analogue video to computer hard drive. It is also possible to identify a number of precepts which can be used to inform the design of each step in your own project.

Capture at best quality possible

Each digitisation project must choose an initial format in which to digitise or ‘capture’ analogue video. This initial format can later be converted or ‘transcoded’ into the multitude of formats likely to be required for delivery (see Guide to Choosing a File Format). However, each of these subsequent versions is necessarily of an equal or lesser ‘quality’ than the initial capture. It is not possible to reintroduce into future generations, quality which was not captured at the time of initial digitisation.
The way to proceed may then seem obvious; digitise analogue video in the highest quality ‘uncompressed’ format and then generate lower quality versions as required. Indeed, as digital storage costs fall, this approach is becoming more common. But at around 70GB per hour, uncompressed video data is still difficult for some projects to accommodate. When deciding on an initial capture format, the larger aims and scope of the project should be taken into account. This is perhaps the single most important decision of any video digitisation project. If video is captured with lossy compression, lost information cannot be recovered without repeating the costly digitisation process. If uncompressed (or even ‘losslesly compressed’) video is captured, storage costs for the resultant huge files can become a seriously limiting factor. Prior to digitisation all stakeholders must agree on a format of digital video which, for their project is ‘fit for purpose’.

Collect required technical metadata as work progresses

Metadata or ‘structured data about data’ needs should be identified in the initial phase of the project (for more information see Metadata for Digital Video) technical metadata handling can then be embedded into the digitisation workflow. This might mean simply filling out a small number of Dublin Core fields as the work progresses. Some metadata, such as original tape format or original label information may be vital to future use of the collection and risks being lost if not recorded at the time of digitisation. It is recommended that at least a basic list of analogue tapes to be digitised is compiled into a skeleton database before digitisation begins. This list then becomes the basis for metadata collected throughout the project. In-depth collection of metadata, such as the construction of complex descriptive records, is best carried out parallel to but outside of the digitisation process by a subject specialist.

Archive original capture files

As mentioned above, the initial capture of the analogue video signal will be of the highest ‘quality’ (which is taken here to mean most similar to the content of the original analogue recording) and so, resources permitting, should be archived as is, as part of a preservation master file. If no re-mastering is required then these files are write-protected and used in order to generate lower-resolution delivery copies.
In an ideal situation, two identical and geographically separate generations of these mater files would be created. While this tactic does offer protection against disaster for unique or valuable materials, the related costs may prove prohibitive.

Re-master only copies

Videotapes may have deteriorated and image problems may be digitised along with the images themselves. Many effective and exciting digital restoration techniques are possible once video is in the digital realm. A simple technique such as an audio boost for example, can make problem footage usable. But these techniques are not always successful. For this reason restoration techniques are best thought of as potentially destructive. Therefore, such techniques should only ever be carried out on a copy rather than a unique file.

Archive re-mastered generation

Once footage has been re-mastered or digital restored these files too should be archived. Restorative techniques represent a significant investment of time and resources. Once successfully completed these files should be of similar status to preservation masters. Re-mastered files (if created) will generally be used as the source of delivery or ‘access’ copies.

Create access generation versions for delivery

Copies of video intended for delivery will typically be of lower-resolution than mater copies, perhaps in a propriety format. Although the creation of these can be largely automated, access copies also represent an investment of time and resources and so should also be archived safely.

Plan for sustainability

Even in the initial stages your project’s output should be viewed as valuable and so deserving of a long lifespan. Simple steps such as generating a checksum value (see FAQ) for each captured file will assist in future error checking.

Use open technologies, avoid propriety archival formats

Some software, hardware and file formats are the property of a single company or institution; these are described as ‘propriety’ technologies. Others are developed and supported by a group of several companies or institutions; these are known as ‘open’ technologies. An advantage of using an open technology is the support of a user community which often accompanies it. There is also a perceived reduction in the risk of obsolescence; support for a propriety format can disappear along with the single company who developed it but this is not so for open technologies. Proprietary file formats can safely be used for delivery but never for long-term preservation purposes.

Example workflow

Presented below is a suggested workflow, in this case for the digitisation of a modest archive of U-matic videotapes. The format of your original analogue videotapes may be different and this will impact on your workflow but general principles of best practice remain the same, regardless of videotape type.

Figure 1.1 U-matic videotape circa 1986

Check capture equipment

At the core of any digitisation project is the playback equipment. Formats such as VHS are still in production and so new players are available. Unfortunately the majority of videotape formats such as U-matic are obsolete and so players will typically be older, used models. Players should be carefully monitored and frequently serviced. Playback heads will require periodic cleaning.

Test capture

A tape containing test footage such as colour bars with a test tone should be run through the system daily to check playback and capture is working correctly. This test should include the digital capture of a potion of the test tape and playback of the captured file.

Check tapes

Videotapes are working objects with complex moving parts, only attempt to play tapes which appear to be in good condition. Tapes in poor condition should be treated as exceptions and may require external conservation treatments.

Check playback

Unlike contemporary digital videotape recorders, players of older analogue formats such as U-matic cannot be automatically controlled by your computer. Playback is initiated manually and should be closely monitored throughout. Your tapes may begin with colour bars or other calibration measures. If this is the case an inline colour corrector can be used to bring the image within appropriate parameters. If colour bars are not present on the tape, your system should be balanced for a tape which contains generic test footage (see ‘Test capture’ above). Audio test tones should be handled similarly using an inline amplifier. Settings on your player such as tracking may need to be finely adjusted for individual videotapes.

Playback

Certain types of deterioration present on the analogue videotape can cause the image and/or audio signal to be poor or intermittent. In this case, capture should be halted and the tape treated as an exception. It may be that the condition of the tape cannot be improved in which case it may be desirable to capture unique content ‘as is’ and attempt digital restoration later.

Trim clips

Trim and/or divide video clips if required, in a video editing application. It is vital that the output format of the editing program be identical to the original capture format. Otherwise quality may be lost before initial archiving has taken place.

Prepare for archiving

Prepare captured files (or trimmed equivalents exported from your editing program) for archiving. Files will typically need renaming before they can be archived, perhaps with a unique identifier. This will depend on the file system or the digital content management system you are using. If footage was captured in a propriety file format, an extra step may be required in order to transcode video to an open format of equal quality for long-term storage.

Archive master generation copy

Once in an open format, with no digital errors a file may be given the status of ‘preservation master’ and archived into your file management structure or digital asset management system. These may be sited on peripheral devices, removable media or networked storage. Both digital tape (e.g. Linear Tape-Open or LTO) and online hard drive-based systems are commonly used for large files. Online systems facilitate automated processes such as error checking and their costs are now falling dramatically. If resources permit, preservation master copies should be written in duplicate to separate locations. Any codecs or other files required to playback the video files should also be written to the archive.

Collect master generation metadata

It is now possible to add information relating to your captured preservation masters to your skeleton record. This should include file name, location, duration, size (in bytes) a checksum value and any observed image or sound problems. This last step is important as later checks may detect problems and erroneously attribute these to poor capture rather than tape deterioration.

Re-master

Improving the audiovisual file, if required will almost certainly be best left for a dedicated post-digitisation phase of the project as it is potential a very slow process. Commonly, only a few techniques are desirable or indeed practical in order to improve video material. These include; audio boost, equalising audio, noise reduction, colour balancing, contrast adjustment and of course, editing.

Archive re-mastered files

Once these techniques have been satisfactorily carried out on a copy of a preservation master file the newly restored file can be archived as part of a re-mastered generation and used to create all subsequent versions of videos suited to delivery.

Collect re-mastered generation metadata

Metadata should be made to clearly reflect the fact that a re-mastered version of a video exists. This will prevent original preservation masters being erroneously used to create future delivery versions. It may be desirable to create an entirely new record for the re-mastered version of the file.

Transcode for delivery

Once digital video is deemed fit for purpose, work can progress with the creation of surrogate ‘access status’ video files in lower resolution formats suited to delivery. This stage might be left until after digitisation, or could be integrated into it, perhaps by trancoding files overnight. Delivery formats are often proprietary and change very rapidly, currently Flash video and MPEG-2 are commonly used although MPEG-4 is rapidly growing in popularity.

Archive access generation

These ‘access status’ video files should be suitably named and integrating into the file management structure. Although they may be seen as having a lesser status than formats intended for long-term preservation, they too represent a significant time investment and should also be carefully archived.

Collect access generation metadata

Some metadata will almost certainly be required for access versions and, as with other generations, this may mean the creation of a dedicated catalogue record.

Sign-off

The process of ‘signing off’ indicates that a metadata record or digital video file has been checked and deemed fit for purpose. This may be as simple as noting date of completion and name of operator in a metadata record. For details of implementing quality assurance within a workflow see Quality Assurance for Digitisation Projects.

Conclusion

Assembling an effective workflow isn’t an easy task. The type, age and condition of videos can vary widely even within a single collection. As noted above, videotapes are infinitely variable and complex working objects. Through research, planning and well documented implementation the task can be made a lot easier. Feasibility studies, though crucial, are only an indication of what is possible; your workflow can only be truly refined through practice. For this reason, every workflow should be an iterative process. You can obtain advice relating specifically to your collection by calling JISC Digital Media’s helpdesk service or by emailing your query.

Figure 1.2 Summary of a generic video digitisation workflow

Last updated: 27 January 2009
Published in: Digitising analogue media
Tags: analogue collections | digitisation | video

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Comments (0)