-
-
Notifications
You must be signed in to change notification settings - Fork 325
Reading and Writing Sequences
HEIF files can contain image sequences. These sequences may consist of several media tracks. A media track is either a visual track or a metadata track. Other track types (e.g. audio) are currently not supported.
A track contains timed samples. Metadata samples can contain any arbitrary data and the track header specifies what kind of data is contained through an URI-based naming scheme. Samples from different tracks do not necessarily come in pairs. Instead, they can be timed independently. For example: while the visual track may carry a 30 fps video stream, the metadata track may contain acceleration sensor data, sampled at a higher rate.
Additionally, each sample (image or metadata) can contain sample auxiliary information (SAI). This is a usually small data package with auxiliary information that is directly attached to the sample. Libheif currently supports TAI timestamps and textual content IDs as defined in the GIMI standard.
Finally, metadata can also be attached to a track as a whole. Currently, libheif support the TAI timestamp configuration block and GIMI track content IDs.
Relations between tracks are defined by track references. For example, a cdsc (Content Description) reference from the metadata track to the visual track denotes that this metadata track is further describing the referenced visual track.
The API specific to sequence processing can be found in the libheif/heif_sequences.h header file.
This document only describes the basic steps to decode and encode HEIF sequences. For more specific functionality, the reader is referred to that header file.
After opening the HEIF file as usual and obtaining a heif_context object, the application can check with
int heif_context_has_sequence(heif_context*)whether there is an image sequence in the file. Note that a file can contain still images and a sequence at the same time.
The application can use the functions
int heif_context_number_of_sequence_tracks(const struct heif_context*);
void heif_context_get_track_ids(const struct heif_context* ctx, uint32_t out_track_id_array[]);to get an overview what tracks are contained in the HEIF file and then get access to a specific track with
struct heif_track* heif_context_get_track(const struct heif_context*, uint32_t id);Alternatively, for convenience, it can call heif_context_get_track(ctx, 0) with a zero ID to get the first (and usually only) visual track in the file.
It can check with
heif_track_type heif_track_get_track_handler_type(struct heif_track*);what kind of data the track contains. This can be an image sequence, a video, or metadata. Other track types (e.g. audio) are currently not supported by libheif, but the raw data can still be read to some extent with the same API as for metadata tracks.
Each track has a timescale, which specifies the clock ticks per second which are the basis for all timing values given in this track. Note that the timescale can be different for each track and that there is also a timescale assigned to the total sequence, which can also be different than the track timescales. When the application gets the total track duration or a sample duration, it has to divide the duration by the timescale to get the duration in seconds.
Decoding can optionally start by querying the track resolution with
struct heif_error heif_track_get_image_resolution(heif_track*, uint16_t* out_width, uint16_t* out_height);Then images can be decoded in temporal order with
struct heif_error heif_track_decode_next_image(struct heif_track* track,
struct heif_image** out_img,
enum heif_colorspace colorspace,
enum heif_chroma chroma,
const struct heif_decoding_options* options);The decoded image is stored in out_img. When the last image has been read and we are reading past the end of the sequence, the error heif_error_End_of_sequence is returned.
The colorspace and chroma parameters can be used to force a specific output color format. If the sequence is coded in a different color format, libheif will convert the image accordingly. When specifying heif_colorspace_undefined and heif_chroma_undefined, libheif will not convert the image but use the most convenient format. Usually, this will be the input video format, but it can also deviate from the codec format because of necessary internal processing.
The image display duration is attached to the image and can be obtained with
uint32_t heif_image_get_duration(const heif_image*);Remember that the image duration is given in timescale units of the track.
Reading metadata tracks is similar to reading visual tracks, but instead of images, the raw data is read into heif_raw_sequence_sample objects with
struct heif_error heif_track_get_next_raw_sequence_sample(struct heif_track*,
heif_raw_sequence_sample** out_sample);The raw data can then be retrieved from this object with
const uint8_t* heif_raw_sequence_sample_get_data(const heif_raw_sequence_sample*, size_t* out_array_size);Similar to reading sequence images, the sample duration is attached to the heif_raw_sequence_sample and can be obtained with
uint32_t heif_raw_sequence_sample_get_duration(const heif_raw_sequence_sample*);In order to write an image sequence, the application first has to add a visual track with
struct heif_error heif_context_add_visual_sequence_track(heif_context*,
uint16_t width, uint16_t height,
struct heif_track_info* info,
heif_track_type track_type,
heif_track** out_track);The track type can be set to heif_track_type_video or heif_track_type_image_sequence.
More information about the track storage configuration is contained in the heif_track_info structure.
The easiest way to fill heif_track_info is to allocate a new structure with
struct heif_track_info* heif_track_info_alloc();as this will fill the structure with the default values and we only need to set the values we are interested in.
The only field that we are currently interested in is uint32_t heif_track_info::track_timescale defining this track's clock ticks per second.
The other fields will be covered further down.
Now that we have a visual track, we can encode images with
struct heif_error heif_track_encode_sequence_image(struct heif_track*,
const struct heif_image* image,
struct heif_encoder* encoder,
const struct heif_encoding_options* options);The encoder and options parameters are set similar as for regular HEIF images. They define which codec to use and further encoding options.
Don't forget to assign the image duration to the image with
void heif_image_set_duration(heif_image*, uint32_t duration);before encoding the image.
Libheif currently uses the "URI Meta Sample Entry" format for metadata tracks. This contains an URI field in the track header that specifies what kind of data is stored in the metadata track, i.e. the binary format of the contained samples. With this information, we can create a new metadata track:
struct heif_error heif_context_add_uri_metadata_sequence_track(heif_context*,
struct heif_track_info* info,
const char* uri,
heif_track** out_track);We then allocate raw sequence sample objects, copy the raw metadata into it and assign the duration:
heif_raw_sequence_sample* heif_raw_sequence_sample_alloc();
heif_error heif_raw_sequence_sample_set_data(heif_raw_sequence_sample*, const uint8_t* data, size_t size);
void heif_raw_sequence_sample_set_duration(heif_raw_sequence_sample*, uint32_t duration);Now the sample can be stored in the metadata track:
struct heif_error heif_track_add_raw_sequence_sample(struct heif_track*,
const heif_raw_sequence_sample*);As mentioned in the introduction, we can attach Sample Auxiliary Information (SAI) to samples in the track.
The type of the SAI is specified with a four-character-code. Libheif currently supports TAI timestamps and GIMI content IDs as SAIs.
The SAI data is assigned to the sample objects, i.e. to the heif_image in the case of visual tracks, or heif_raw_sequence_sample in the case of metadata tracks.
When reading a file these can be retrieved from the samples with
const char* heif_image_get_gimi_sample_content_id(const heif_image*);
const char* heif_raw_sequence_sample_get_gimi_sample_content_id(const heif_raw_sequence_sample*);for content IDs and
const struct heif_tai_timestamp_packet* heif_raw_sequence_sample_get_tai_timestamp(const struct heif_raw_sequence_sample*);
struct heif_error heif_image_get_tai_timestamp(const struct heif_image* img,
struct heif_tai_timestamp_packet* timestamp);for TAI timestamps.
There exist similar functions to attach the SAIs to the samples before encoding the samples into the track.
There are also functions to get the list of SAIs that are available in a read HEIF file:
int heif_track_get_number_of_sample_aux_infos(struct heif_track*);
void heif_track_get_sample_aux_info_types(struct heif_track*, struct heif_sample_aux_info_type out_types[]);A track that contains TAI timestamp SAIs also needs a TAIC configuration box. This is assigned to the track as a whole. (More correctly, it is assigned to a cluster of samples, but since there is usually only one cluster of metadata samples, the TAIC is assigned to the whole track.) It can be retrieved using
const struct heif_tai_clock_info* heif_track_get_tai_clock_info_of_first_cluster(struct heif_track*);When writing a metadata track, the heif_tai_clock_info can be specified in the above mentioned heif_track_info.
It is also possible to assign a track-level GIMI content ID to the track. This can be obtained with
const char* heif_track_get_gimi_track_content_id(const struct heif_track*);and when writing the track, it is also specified in heif_track_info.
When writing tracks with SAIs attached to the samples, libheif offers two possibilities how the data is laid out in the file:
- The SAIs can be stored right after the image/metadata sample. This is advantageous if the file should be streamed, as all sample data is then clustered together.
- All SAIs can be stored in one contiguous block. This means, the file contains a block with all sample data, and one block for each of the SAI types. This results in slightly smaller file sizes since libheif does not need to save file offsets to the individual sample SAIs as it can make use of a special storage mode when all SAIs are stored contiguously.
Which mode to choose can be specified in the heif_track_info structure.
Usually, metadata tracks provide additional information related to a visual track.
This can be indicated in the HEIF file by setting a track reference.
A track reference has a type, which in this case is cdsc (Content Description).
When writing a file, the track references can be added with
void heif_track_add_reference_to_track(heif_track*, uint32_t reference_type, heif_track* to_track);In order to indicate that a metadata track is describing a visual track, we would call this as:
heif_track_add_reference_to_track(metadata_track, heif_track_reference_type_description_of, visual_track);For the corresponding functions to list the track references when reading files, consult the libheif/heif_sequences.h header file.