diff --git a/api/TimeAddressableMediaStore.yaml b/api/TimeAddressableMediaStore.yaml index e5fd5788..c2f2b8e9 100644 --- a/api/TimeAddressableMediaStore.yaml +++ b/api/TimeAddressableMediaStore.yaml @@ -865,6 +865,11 @@ paths: description: Filter on video Flows that have the given frame height. schema: type: integer + - name: init_segments + in: query + description: Filter Flows on the value of `init_segments`. + schema: + type: boolean - $ref: '#/components/parameters/trait_resource_paged_key' - $ref: '#/components/parameters/trait_paged_limit' responses: @@ -952,6 +957,11 @@ paths: description: Filter on video Flows that have the given frame height. schema: type: integer + - name: init_segments + in: query + description: Filter Flows on the value of `init_segments`. + schema: + type: boolean - $ref: '#/components/parameters/trait_resource_paged_key' - $ref: '#/components/parameters/trait_paged_limit' responses: @@ -1053,6 +1063,9 @@ paths: mxf_h264: summary: Video Flow - H.264 (MXF container) externalValue: examples/flow-get-200-video-h264-mxf.json + fmp4_h264: + summary: Video Flow - H.264 (fmp4 container) + externalValue: examples/flow-get-200-video-h264-fmp4.json jp2_jpeg2k_singleframe: summary: Video Flow - JPEG-2000 (JP2 container, single frame segments) externalValue: examples/flow-get-200-video-jpeg-jp2.json @@ -1143,7 +1156,9 @@ paths: summary: Delete Flow description: | Deletes the Flow and associated Segments. - If Flow Segment deletion takes too long then this request will return 202 Accepted and the `Location` header will point to a Flow Delete Request to monitor deletion progress + If Flow Segment deletion takes too long then this request will return 202 Accepted and the `Location` header will point to a Flow Delete Request to monitor deletion progress. + + Services SHALL only delete Media Objects when they are no longer referenced by any Flow Segments or other Media Objects (i.e. as init Objects). operationId: DELETE_flows-flowId tags: - Flows @@ -1754,39 +1769,39 @@ paths: in: query description: | Include storage metadata in `get_urls` in the response. - When `verbose_storage` is `false` only `url`, `presigned`, and `label` will be included in `get_urls`. + When `verbose_storage` is `false` only `url`, `presigned`, and `label` will be included in `get_urls` and `init_object.get_urls`. schema: default: false type: boolean - name: accept_get_urls in: query description: | - A comma separated list of labels of Flow Segment `get_urls` to include in the response. - Omitting `accept_get_urls` will result in no filtering of `get_urls`. - An empty `accept_get_urls` results in an empty or no `get_urls` in the response. - Flow Segment `get_urls` with no label will only be returned if `accept_get_urls` is omitted. + A comma separated list of labels of Flow Segment `get_urls` and `init_object.get_urls` to include in the response. + Omitting `accept_get_urls` will result in no filtering of `get_urls` or `init_object.get_urls`. + An empty `accept_get_urls` results in `get_urls` and `init_object.get_urls` being empty or omitted in the response. + Flow Segment `get_urls` and `init_object.get_urls` with no label will only be returned if `accept_get_urls` is omitted. Without `get_urls`, the response from the service could be substantially faster if it is not required to generate a large number of pre-signed URLs for example. - Where multiple filter query parameters are provided, the returned `get_urls` will match all filters. + Where multiple filter query parameters are provided, the returned `get_urls` and `init_object.get_urls` will match all filters. schema: $ref: 'schemas/url-label-list.json' - name: accept_storage_ids in: query description: | - A comma separated list of `storage_id`s of Flow Segment `get_urls` to include in the response. - Omitting `accept_storage_ids`, or providing an empty `accept_storage_ids` will result in no filtering of `get_urls`. - Flow Segment `get_urls` with no storage ID will only be returned if `accept_storage_ids` is omitted or empty. + A comma separated list of `storage_id`s of Flow Segment `get_urls` and `init_object.get_urls` to include in the response. + Omitting `accept_storage_ids`, or providing an empty `accept_storage_ids` will result in no filtering of `get_urls` or `init_object.get_urls`. + Flow Segment `get_urls` and `init_object.get_urls` with no storage ID will only be returned if `accept_storage_ids` is omitted or empty. A full list of available `storage_id`s may be found at the [/service/storage-backends](#/operations/GET_storage-backends) endpoint. - Where multiple filter query parameters are provided, the returned `get_urls` will match all filters. + Where multiple filter query parameters are provided, the returned `get_urls` and `init_object.get_urls` will match all filters. schema: $ref: 'schemas/uuid-list.json' - name: presigned in: query description: | - If set to `true`, only presigned URLs (i.e. those whos `presigned` property is `true`) will be returned in `get_urls` in the response. - If set to `false`, only non-presigned URLs (i.e. those whos `presigned` property is `false`) will be returned in `get_urls`. + If set to `true`, only presigned URLs (i.e. those whos `presigned` property is `true`) will be returned in `get_urls` and `init_object.get_urls` in the response. + If set to `false`, only non-presigned URLs (i.e. those whos `presigned` property is `false`) will be returned in `get_urls` and `init_object.get_urls`. If omitted, both presigned and non-presigned URLs will be returned. If `presigned` is set to `false`, the response from the service could be substantially faster if it is not required to generate a large number of pre-signed URLs. - Where multiple filter query parameters are provided, the returned `get_urls` will match all filters. + Where multiple filter query parameters are provided, the returned `get_urls` and `init_object.get_urls` will match all filters. schema: type: boolean - name: include_object_timerange @@ -1856,6 +1871,9 @@ paths: Note that for codecs with temporal re-ordering, the timerange representes the _presentation_ timeline, and clients may need to check the `key_frame_count` property and/or read backwards from the start of the requested timerange to retrieve enough reference material to start decoding. + Where Flow Segments reference initialisation segment Objects with the same ID, the initialisation segment is the same. + Consuming clients may choose to ignore initialisation segment Objects that haven't changed in subsequent Flow Segments. + When making requests to the provided `get_urls`, clients should include credentials if the provided URL is on the same origin as the API itself, akin to the `same-origin` mode in the [WhatWG Fetch Standard](https://fetch.spec.whatwg.org/#concept-request-credentials-mode). operationId: GET_flows-flowId-segments tags: @@ -1881,39 +1899,39 @@ paths: in: query description: | Include storage metadata in `get_urls` in the response. - When `verbose_storage` is `false` only `url`, `presigned`, and `label` will be included in `get_urls`. + When `verbose_storage` is `false` only `url`, `presigned`, and `label` will be included in `get_urls` and `init_object.get_urls`. schema: default: false type: boolean - name: accept_get_urls in: query description: | - A comma separated list of labels of Flow Segment `get_urls` to include in the response. - Omitting `accept_get_urls` will result in no filtering of `get_urls`. - An empty `accept_get_urls` results in an empty or no `get_urls` in the response. - Flow Segment `get_urls` with no label will only be returned if `accept_get_urls` is omitted. + A comma separated list of labels of Flow Segment `get_urls` and `init_object.get_urls` to include in the response. + Omitting `accept_get_urls` will result in no filtering of `get_urls` or `init_object.get_urls`. + An empty `accept_get_urls` results in `get_urls` and `init_object.get_urls` being empty or omitted in the response. + Flow Segment `get_urls` and `init_object.get_urls` with no label will only be returned if `accept_get_urls` is omitted. Without `get_urls`, the response from the service could be substantially faster if it is not required to generate a large number of pre-signed URLs for example. - Where multiple filter query parameters are provided, the returned `get_urls` will match all filters. + Where multiple filter query parameters are provided, the returned `get_urls` and `init_object.get_urls` will match all filters. schema: $ref: 'schemas/url-label-list.json' - name: accept_storage_ids in: query description: | - A comma separated list of `storage_id`s of Flow Segment `get_urls` to include in the response. - Omitting `accept_storage_ids`, or providing an empty `accept_storage_ids` will result in no filtering of `get_urls`. - Flow Segment `get_urls` with no storage ID will only be returned if `accept_storage_ids` is omitted or empty. + A comma separated list of `storage_id`s of Flow Segment `get_urls` and `init_object.get_urls` to include in the response. + Omitting `accept_storage_ids`, or providing an empty `accept_storage_ids` will result in no filtering of `get_urls` or `init_object.get_urls`. + Flow Segment `get_urls` and `init_object.get_urls` with no storage ID will only be returned if `accept_storage_ids` is omitted or empty. A full list of available `storage_id`s may be found at the [/service/storage-backends](#/operations/GET_storage-backends) endpoint. - Where multiple filter query parameters are provided, the returned `get_urls` will match all filters. + Where multiple filter query parameters are provided, the returned `get_urls` and `init_object.get_urls` will match all filters. schema: $ref: 'schemas/uuid-list.json' - name: presigned in: query description: | - If set to `true`, only presigned URLs (i.e. those whos `presigned` property is `true`) will be returned in `get_urls`. - If set to `false`, only non-presigned URLs (i.e. those whos `presigned` property is `false`) will be returned in `get_urls`. + If set to `true`, only presigned URLs (i.e. those whos `presigned` property is `true`) will be returned in `get_urls` and `init_object.get_urls`. + If set to `false`, only non-presigned URLs (i.e. those whos `presigned` property is `false`) will be returned in `get_urls` and `init_object.get_urls`. If omitted, both presigned and non-presigned URLs will be returned. If `presigned` is set to `false`, the response from the service could be substantially faster if it is not required to generate a large number of pre-signed URLs. - Where multiple filter query parameters are provided, the returned `get_urls` will match all filters. + Where multiple filter query parameters are provided, the returned `get_urls` and `init_object.get_urls` will match all filters. schema: type: boolean - name: include_object_timerange @@ -1974,6 +1992,12 @@ paths: This example shows how the Media Object for a Flow Segment could be accessed using multiple URLs. Clients are free to choose any of the URLs given (which return identical Media Objects), however the service has expressed a preference for the "pipeline-a" URLs by putting them first in the list. externalValue: examples/flow-segments-get-200-multiple-urls.json + init_segments: + summary: Objects with initialisation segments + description: | + This example shows how Media Objects signal the use of initialisation segments. + Note the re-use of the same initialisation segment in some consecutive Flow Segments. + externalValue: examples/flow-segments-get-200-init.json "400": description: Bad request. Invalid query options. "404": @@ -2014,7 +2038,10 @@ paths: - The timerange of the Segment MUST NOT overlap any other Segment in the same Flow. - The Flow Segment's `timerange` start and end, once offset by `ts_offset`, MUST be contained entirely within the Media Object's `timerange` - When re-using Media Objects, requests which change object properties (e.g. `key_frame_count` or `object_timerange`) SHOULD be rejected. + When re-using Media Objects, requests which change object properties (e.g. `key_frame_count`, `object_timerange`, or `init_object_id`) SHOULD be rejected. + + If an Object has previously been registered as an initialisation segment (i.e. via `init_object_id`), Service implementations SHOULD reject its use as a media segment (i.e. via `object_id`). + If an Object has previously been registered as a media segment (i.e. via `object_id`), Service implementations SHOULD reject its use as an initialisation segment (i.e. via `init_object_id`). operationId: POST_flows-flowId-segments tags: - FlowSegments @@ -2050,7 +2077,9 @@ paths: delete: summary: Delete Flow Segment description: | - Deletes the Flow Segments. If the deletion takes too long then this request will return 202 Accepted and the `Location` header will point to a Flow Delete Request to monitor deletion progress + Deletes the Flow Segments. If the deletion takes too long then this request will return 202 Accepted and the `Location` header will point to a Flow Delete Request to monitor deletion progress. + + Services SHALL only delete Media Objects when they are no longer referenced by any Flow Segments or other Media Objects (i.e. as init Objects). operationId: DELETE_flows-flowId-segments tags: - FlowSegments @@ -2113,6 +2142,12 @@ paths: Objects will likely go unused in cases such as shutdown of ingesting clients, the end of ingested live streams, and unexpected network congestion. Clients SHOULD, however, adapt the number of Objects they request such that they may reasonably expect to use them before the timeout advertised in [`min_object_timeout` at the `/service`](#/operations/GET_service) endpoint, which is subject to a specified minimum (see service endpoint schema). + Where media format make use of initialisation segments, the mime-type of those initialisation segments may differ from that of the media segments. + In such cases, the `container` on the Flow will be set to the mime-type of the media segments and is the assumed default. + A separate request to this endpoint should be made to allocate storage for the initialisation segment(s) with `content-type` set to the mime type of the initialisation segment(s). + `content-type` MUST NOT be set in the request body of this end point at any other time. + Initialisation segment Objects SHOULD be re-used wherever possible. + Service implementations need to handle situations where Objects are not used, and where content is were uploaded but no Flow Segment was registered successfully. In these circumstances, Services should garbage collect Objects after the timeout advertised in [`min_object_timeout` at the `/service`](#/operations/GET_service) endpoint. @@ -2172,42 +2207,42 @@ paths: in: query description: | Include storage metadata in `get_urls`. - When `verbose_storage` is `false` only `url`, `presigned`, and `label` will be included in `get_urls`. + When `verbose_storage` is `false` only `url`, `presigned`, and `label` will be included in `get_urls` and `init_object.get_urls`. schema: default: false type: boolean - name: accept_get_urls in: query description: | - A comma separated list of labels of Media Object `get_urls` to include in the response. - Omitting `accept_get_urls` will result in no filtering of `get_urls`. - An empty `accept_get_urls` results in an empty or no `get_urls` in the response. - Media Object `get_urls` with no label will only be returned if `accept_get_urls` is omitted. - Without `get_urls`, the response from the service could be substantially faster if it is not required to + A comma separated list of labels of Media Object `get_urls` and `init_object.get_urls` to include in the response. + Omitting `accept_get_urls` will result in no filtering of `get_urls` or `init_object.get_urls`. + An empty `accept_get_urls` results in `get_urls` and `init_object.get_urls` being empty or omitted in the response. + Media Object `get_urls` and `init_object.get_urls` with no label will only be returned if `accept_get_urls` is omitted. + Without `get_urls` or `init_object.get_urls`, the response from the service could be substantially faster if it is not required to generate a large number of pre-signed URLs for example. - Where multiple filter query parameters are provided, the returned `get_urls` will match all filters. + Where multiple filter query parameters are provided, the returned `get_urls` and `init_object.get_urls` will match all filters. schema: $ref: 'schemas/url-label-list.json' - name: accept_storage_ids in: query description: | - A comma separated list of `storage_id`s of Media Object `get_urls` to include in the response. - Omitting `accept_storage_ids`, or providing an empty `accept_storage_ids` will result in no filtering of `get_urls`. - Media Object `get_urls` with no storage ID will only be returned if `accept_storage_ids` is omitted or empty. + A comma separated list of `storage_id`s of Media Object `get_urls` and `init_object.get_urls` to include in the response. + Omitting `accept_storage_ids`, or providing an empty `accept_storage_ids` will result in no filtering of `get_urls` or `init_object.get_urls`. + Media Object `get_urls` and `init_object.get_urls` with no storage ID will only be returned if `accept_storage_ids` is omitted or empty. A full list of available `storage_id`s may be found at the `service/storage-backends` endpoint. - Where multiple filter query parameters are provided, the returned `get_urls` will match all filters. + Where multiple filter query parameters are provided, the returned `get_urls` and `init_object.get_urls` will match all filters. schema: type: string pattern: ^([0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12})(,[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12})*$ - name: presigned in: query description: | - If set to `true`, only presigned URLs (i.e. those whos `presigned` property is `true`) will be returned in `get_urls`. - If set to `false`, only non-presigned URLs (i.e. those whos `presigned` property is `false`) will be returned in `get_urls`. + If set to `true`, only presigned URLs (i.e. those whos `presigned` property is `true`) will be returned in `get_urls` and `init_object.get_urls`. + If set to `false`, only non-presigned URLs (i.e. those whos `presigned` property is `false`) will be returned in `get_urls` and `init_object.get_urls`. If omitted, both presigned and non-presigned URLs will be returned. If `presigned` is set to `false`, the response from the service could be substantially faster if it is not required to generate a large number of pre-signed URLs. - Where multiple filter query parameters are provided, the returned `get_urls` will match all filters. + Where multiple filter query parameters are provided, the returned `get_urls` and `init_object.get_urls` will match all filters. schema: type: boolean - name: flow_tag.{name} @@ -2269,42 +2304,42 @@ paths: in: query description: | Include storage metadata in `get_urls`. - When `verbose_storage` is `false` only `url`, `presigned`, and `label` will be included in `get_urls`. + When `verbose_storage` is `false` only `url`, `presigned`, and `label` will be included in `get_urls` and `init_object.get_urls`. schema: default: false type: boolean - name: accept_get_urls in: query description: | - A comma separated list of labels of Media Object `get_urls` to include in the response. - Omitting `accept_get_urls` will result in no filtering of `get_urls`. - An empty `accept_get_urls` results in an empty or no `get_urls` in the response. - Media Object `get_urls` with no label will only be returned if `accept_get_urls` is omitted. - Without `get_urls`, the response from the service could be substantially faster if it is not required to + A comma separated list of labels of Media Object `get_urls` and `init_object.get_urls` to include in the response. + Omitting `accept_get_urls` will result in no filtering of `get_urls` or `init_object.get_urls`. + An empty `accept_get_urls` results in `get_urls` and `init_object.get_urls` being empty or omitted in the response. + Media Object `get_urls` and `init_object.get_urls` with no label will only be returned if `accept_get_urls` is omitted. + Without `get_urls` and `init_object.get_urls`, the response from the service could be substantially faster if it is not required to generate a large number of pre-signed URLs for example. - Where multiple filter query parameters are provided, the returned `get_urls` will match all filters. + Where multiple filter query parameters are provided, the returned `get_urls` and `init_object.get_urls` will match all filters. schema: $ref: 'schemas/url-label-list.json' - name: accept_storage_ids in: query description: | - A comma separated list of `storage_id`s of Media Object `get_urls` to include in the response. - Omitting `accept_storage_ids`, or providing an empty `accept_storage_ids` will result in no filtering of `get_urls`. - Media Object `get_urls` with no storage ID will only be returned if `accept_storage_ids` is omitted or empty. + A comma separated list of `storage_id`s of Media Object `get_urls` and `init_object.get_urls` to include in the response. + Omitting `accept_storage_ids`, or providing an empty `accept_storage_ids` will result in no filtering of `get_urls` or `init_object.get_urls`. + Media Object `get_urls` and `init_object.get_urls` with no storage ID will only be returned if `accept_storage_ids` is omitted or empty. A full list of available `storage_id`s may be found at the `service/storage-backends` endpoint. - Where multiple filter query parameters are provided, the returned `get_urls` will match all filters. + Where multiple filter query parameters are provided, the returned `get_urls` and `init_object.get_urls` will match all filters. schema: type: string pattern: ^([0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12})(,[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12})*$ - name: presigned in: query description: | - If set to `true`, only presigned URLs (i.e. those whos `presigned` property is `true`) will be returned in `get_urls`. - If set to `false`, only non-presigned URLs (i.e. those whos `presigned` property is `false`) will be returned in `get_urls`. + If set to `true`, only presigned URLs (i.e. those whos `presigned` property is `true`) will be returned in `get_urls` and `init_object.get_urls`. + If set to `false`, only non-presigned URLs (i.e. those whos `presigned` property is `false`) will be returned in `get_urls` and `init_object.get_urls`. If omitted, both presigned and non-presigned URLs will be returned. If `presigned` is set to `false`, the response from the service could be substantially faster if it is not required to generate a large number of pre-signed URLs. - Where multiple filter query parameters are provided, the returned `get_urls` will match all filters. + Where multiple filter query parameters are provided, the returned `get_urls` and `init_object.get_urls` will match all filters. schema: type: boolean - name: flow_tag.{name} @@ -2339,8 +2374,13 @@ paths: type: string content: application/json: - example: - $ref: examples/objects-get-200.json + examples: + basic: + summary: Basic example + externalValue: examples/objects-get-200.json + init_segment: + summary: Object that makes use of an initialisation segment + externalValue: examples/objects-get-200-init.json schema: $ref: "schemas/object.json" "400": @@ -2361,6 +2401,9 @@ paths: The API instances SHOULD be capable of handling the case where the only existant instances are uncontrolled. + API instances SHALL NOT cascade the creation of new Object instances to initialisation Objects. + Clients MUST initiate the creation of new Object instances of initialisation Objects directly. + Where a client has written a new uncontrolled Object instance, the client is responsible for ensuring that the Object written is complete and correct before registering it with this method. All instances of an Object MUST be identical. @@ -2406,6 +2449,9 @@ paths: API instances should remove the Media Object instance from the `get_urls` list and then, if the instance is controlled, delete the Object instance from storage. API instances SHOULD prevent clients from deleting all Object instances. Additionally, API instances MAY prevent clients from deleting all controlled Object instances. Where clients wish to remove all copies of an Object from the store, they should do so by deleting all Flows or Flow Segments which reference the Object. + + API instances SHALL NOT cascade the deletion of Object instances to initialisation Objects. + Clients MUST initiate the deletion of Object instances of initialisation Objects directly. operationId: DELETE_objects-instances tags: - Objects diff --git a/api/examples/flow-get-200-video-h264-fmp4.json b/api/examples/flow-get-200-video-h264-fmp4.json new file mode 100644 index 00000000..1de5e6d1 --- /dev/null +++ b/api/examples/flow-get-200-video-h264-fmp4.json @@ -0,0 +1,54 @@ +{ + "id": "4f79cfd1-c057-47f4-8e4d-1b126ca7bf34", + "source_id": "2aa143ac-0ab7-4d75-bc32-5c00c13d186f", + "generation": 0, + "created": "2008-05-27T18:51:00Z", + "metadata_updated": "2023-09-14T09:45:26Z", + "segments_updated": "2023-09-14T09:45:26Z", + "description": "Big Buck Bunny", + "label": "bbb", + "format": "urn:x-nmos:format:video", + "created_by": "tams-dev", + "updated_by": "tams-dev", + "tags": { + "input_quality": "contribution" + }, + "codec": "video/h264", + "container": "video/iso.segment", + "avg_bit_rate": 2479, + "segment_duration": { + "numerator": 10 + }, + "essence_parameters": { + "init_segments": true, + "frame_rate": { + "numerator": 24, + "denominator": 1 + }, + "frame_width": 1280, + "frame_height": 720, + "bit_depth": 8, + "interlace_mode": "progressive", + "colorspace": "BT709", + "transfer_characteristic": "SDR", + "aspect_ratio": { + "numerator": 16, + "denominator": 9 + }, + "pixel_aspect_ratio": { + "numerator": 1, + "denominator": 1 + }, + "component_type": "YCbCr", + "vert_chroma_subs": 2, + "horiz_chroma_subs": 2, + "avc_parameters": { + "profile": 100, + "level": 31, + "flags": 0 + } + }, + "collected_by": [ + "e85efab4-993b-4ad6-9af3-4cd8d0d38860" + ] +} diff --git a/api/examples/flow-segments-get-200-init.json b/api/examples/flow-segments-get-200-init.json new file mode 100644 index 00000000..eb95f4ef --- /dev/null +++ b/api/examples/flow-segments-get-200-init.json @@ -0,0 +1,53 @@ +[ + { + "object_id": "846023d3-612d-5014-bc47-88f6eb2d04bb", + "timerange": "[0:0_10:0)", + "get_urls": [ + { + "url": "https://store.example.com/tams-e2b89b02-21e7-5f9d-aa2d-db38b01453c9/846023d3-612d-5014-bc47-88f6eb2d04bb" + } + ], + "init_object": { + "object_id": "9ef06e23-b882-4a93-b005-480de42ef4e9", + "get_urls": [ + { + "url": "https://store.example.com/tams-e2b89b02-21e7-5f9d-aa2d-db38b01453c9/9ef06e23-b882-4a93-b005-480de42ef4e9" + } + ] + } + }, + { + "object_id": "25be83fc-11d1-5743-9d47-6865cef5ea35", + "timerange": "[10:0_20:0)", + "get_urls": [ + { + "url": "https://store.example.com/tams-e2b89b02-21e7-5f9d-aa2d-db38b01453c9/25be83fc-11d1-5743-9d47-6865cef5ea35" + } + ], + "init_object": { + "object_id": "9ef06e23-b882-4a93-b005-480de42ef4e9", + "get_urls": [ + { + "url": "https://store.example.com/tams-e2b89b02-21e7-5f9d-aa2d-db38b01453c9/9ef06e23-b882-4a93-b005-480de42ef4e9" + } + ] + } + }, + { + "object_id": "8b785422-6a82-5d60-b25a-f77e0a748321", + "timerange": "[20:0_30:0)", + "get_urls": [ + { + "url": "https://store.example.com/tams-e2b89b02-21e7-5f9d-aa2d-db38b01453c9/8b785422-6a82-5d60-b25a-f77e0a748321" + } + ], + "init_object": { + "object_id": "192472f1-55fd-45dc-b355-e6833a3a140c", + "get_urls": [ + { + "url": "https://store.example.com/tams-e2b89b02-21e7-5f9d-aa2d-db38b01453c9/192472f1-55fd-45dc-b355-e6833a3a140c" + } + ] + } + } +] diff --git a/api/examples/objects-get-200-init.json b/api/examples/objects-get-200-init.json new file mode 100644 index 00000000..3c7c4a0b --- /dev/null +++ b/api/examples/objects-get-200-init.json @@ -0,0 +1,22 @@ +{ + "id": "846023d3-612d-5014-bc47-88f6eb2d04bb", + "referenced_by_flows": [ + "4f79cfd1-c057-47f4-8e4d-1b126ca7bf34", + "0fde9c11-da9d-434a-a113-d3b20a2cf251" + ], + "first_referenced_by_flow": "4f79cfd1-c057-47f4-8e4d-1b126ca7bf34", + "timerange": "[150:0_200:0)", + "get_urls": [ + { + "url": "https://store.example.com/tams-e2b89b02-21e7-5f9d-aa2d-db38b01453c9/846023d3-612d-5014-bc47-88f6eb2d04bb" + } + ], + "init_object": { + "id": "9ef06e23-b882-4a93-b005-480de42ef4e9", + "get_urls": [ + { + "url": "https://store.example.com/tams-e2b89b02-21e7-5f9d-aa2d-db38b01453c9/9ef06e23-b882-4a93-b005-480de42ef4e9" + } + ] + } +} diff --git a/api/schemas/flow-audio.json b/api/schemas/flow-audio.json index b0632d3c..4f6a2a76 100644 --- a/api/schemas/flow-audio.json +++ b/api/schemas/flow-audio.json @@ -78,6 +78,10 @@ ] } } + }, + "init_segments": { + "description": "Whether the Flow makes use of initialisation segments. This parameter MUST be set to `true` if Media Objects have `init_object` populated. If set to `true`, all Media Objects MUST have `init_object` populated. Assume `false` if omitted.", + "type": "boolean" } } } diff --git a/api/schemas/flow-core.json b/api/schemas/flow-core.json index 05328f60..66e3c6da 100644 --- a/api/schemas/flow-core.json +++ b/api/schemas/flow-core.json @@ -68,7 +68,7 @@ "$ref": "mime-type.json" }, "container": { - "description": "The container MIME type for Flow Segments. Note that the `type` component of the container MIME type (i.e. the component before the `/`) may be different to the `type` component of the codec MIME type. e.g. An audio Flow may have `audio/aac` coded content may be wrapped in a `video/mp2t` container. Where multiple types exist for a subtype (e.g. `video/mp4`, `audio/mp4`, `application/mp4`), the closest MIME type to the Flow `format` should be used (e.g. `audio/mp4` for a Flow `format` of `urn:x-nmos:format:audio`). Mime types from the [IANA registry](https://www.iana.org/assignments/media-types/media-types.xhtml) should be preferred. Where multiple MIME types are possible, the most common should be preferred. Where this is insufficient, the maintainers of the TAMS repository may create an application note advising which MIME type to use. Where the Flow does not reference any Media Object(s) directly (e.g. an empty Multi Flow that serves only to collect related mono-essence Flows that do reference Media Objects), this property MUST NOT be set.", + "description": "The container MIME type for Flow Segments. Where the media format employs initialisation segments, this is the mime type of the media segments and NOT the initialisation segment(s). Note that the `type` component of the container MIME type (i.e. the component before the `/`) may be different to the `type` component of the codec MIME type. e.g. An audio Flow may have `audio/aac` coded content may be wrapped in a `video/mp2t` container. Where multiple types exist for a subtype (e.g. `video/mp4`, `audio/mp4`, `application/mp4`), the closest MIME type to the Flow `format` should be used (e.g. `audio/mp4` for a Flow `format` of `urn:x-nmos:format:audio`). Mime types from the [IANA registry](https://www.iana.org/assignments/media-types/media-types.xhtml) should be preferred. Where multiple MIME types are possible, the most common should be preferred. Where this is insufficient, the maintainers of the TAMS repository may create an application note advising which MIME type to use. Where the Flow does not reference any Media Object(s) directly (e.g. an empty Multi Flow that serves only to collect related mono-essence Flows that do reference Media Objects), this property MUST NOT be set.", "$ref": "mime-type.json" }, "avg_bit_rate": { diff --git a/api/schemas/flow-data.json b/api/schemas/flow-data.json index d8d4ab7c..8d2deb4f 100644 --- a/api/schemas/flow-data.json +++ b/api/schemas/flow-data.json @@ -30,6 +30,10 @@ "data_type": { "description": "The type of information encoded in the Flow, identified using a URN. e.g. The data_type may be urn:x-tams:data:bounding-box, and the codec `application/json`.", "type": "string" + }, + "init_segments": { + "description": "Whether the Flow makes use of initialisation segments. This parameter MUST be set to `true` if Media Objects have `init_object` populated. If set to `true`, all Media Objects MUST have `init_object` populated. Assume `false` if omitted.", + "type": "boolean" } } } diff --git a/api/schemas/flow-multi.json b/api/schemas/flow-multi.json index 88e283a7..c82891a8 100644 --- a/api/schemas/flow-multi.json +++ b/api/schemas/flow-multi.json @@ -18,6 +18,18 @@ "enum": [ "urn:x-nmos:format:multi" ] + }, + "essence_parameters": { + "title": "Multi Flow Essence Parameters", + "description": "Describes the parameters of the essence inside this multi Flow", + "type": "object", + "additionalProperties": false, + "properties": { + "init_segments": { + "description": "Whether the Flow makes use of initialisation segments. This parameter MUST be set to `true` if Media Objects have `init_object` populated. If set to `true`, all Media Objects MUST have `init_object` populated. Assume `false` if omitted.", + "type": "boolean" + } + } } } } diff --git a/api/schemas/flow-segment-post.json b/api/schemas/flow-segment-post.json index bb7d26dc..5f5e60fc 100644 --- a/api/schemas/flow-segment-post.json +++ b/api/schemas/flow-segment-post.json @@ -8,7 +8,11 @@ ], "properties": { "object_id": { - "description": "The Object identifier for the Media Object.", + "description": "The Object identifier for the Media Object. The `content-type` of the Media Object MUST match the `container` mime-type of the Flow. Service implementations SHOULD reject Objects IDs which have previously been registered as an `init_object_id` on other Flow Segments. i.e. init segments may not be used as media segments.", + "type": "string" + }, + "init_object_id": { + "description": "The Object identifier for the initialisation segment Object required to decode the Media Object. The `content-type` of the initialisation segment Object MAY differ from the `container` mime-type of the Flow. This parameter MUST only be set where the media format makes use of initialisation segments. Initialisation Objects SHOULD be re-used where possible. This parameter SHOULD be omitted where the Object `object_id` already exists and is being re-used. Service implementations SHOULD reject Objects IDs which have previously been registered as an `id` on other Flow Segments. i.e. media segments may not be used as init segments.", "type": "string" }, "ts_offset": { diff --git a/api/schemas/flow-segment.json b/api/schemas/flow-segment.json index df0244d6..7226f4e9 100644 --- a/api/schemas/flow-segment.json +++ b/api/schemas/flow-segment.json @@ -39,11 +39,33 @@ "description": "The count of samples in the Segment (which may be fewer than in the Media Object). The count could be less than expected given the Segment duration and rate if there are gaps. If not set, every sample from sample_offset onwards is used. Note that a sample is a video frame or audio sample. A (coded) audio frame has multiple audio samples. DEPRECATED: Use object_timerange instead - see AppNote 0036. Service implementations SHOULD continue to store and return it if set.", "type": "integer", "deprecated": true + }, + "init_object": { + "title": "Initialisation Object", + "description": "The Object containing the initialisation segment required to decode the parent Media Object.", + "unevaluatedProperties": false, + "allOf": [ + { + "type": "object", + "required": [ + "object_id" + ], + "properties": { + "object_id": { + "description": "The identifier of the initialisation Object.", + "type": "string" + } + } + }, + { + "$ref": "object-core.json" + } + ] } } }, { - "$ref": "object-core.json" + "$ref": "object-media-core.json" } ] } diff --git a/api/schemas/flow-storage-post.json b/api/schemas/flow-storage-post.json index df85e17a..b51b4f75 100644 --- a/api/schemas/flow-storage-post.json +++ b/api/schemas/flow-storage-post.json @@ -17,6 +17,10 @@ "storage_id": { "description": "The Storage Backend to allocate storage in. A Storage Backend identifier as advertised at the [/service/storage-backends](#/operations/GET_storage-backends) endpoint. If not set the default, as advertised at the [/service/storage-backends](#/operations/GET_storage-backends) endpoint, will be used if available. An invalid Storage Backend identifier will result in a 400 error.", "$ref": "uuid.json" + }, + "content_type": { + "description": "The `content_type` to use for the Objects. This parameter MUST only be set where requesting storage for initialisation segments in media formats which require them, and where the mime-type of those initialisation segments differs to that of the media segments. Assumed to be the `container` type of the Flow if not set.", + "$ref": "mime-type.json" } }, "not": { diff --git a/api/schemas/flow-storage.json b/api/schemas/flow-storage.json index 2614b27a..807256a3 100644 --- a/api/schemas/flow-storage.json +++ b/api/schemas/flow-storage.json @@ -1,7 +1,7 @@ { "type": "object", "title": "Media Bucket Object Store", - "description": "Gives information on storage for Media Objects. This schema is for the `http_object_store` Storage Backend type which provides URLs for storing Media Objects in object store buckets, and is the only Storage Backend type currently implemented. URLs SHOULD support the inclusion of checksums in headers as supported by advertised Storage Backend product. See AppNote 0048 for more details.", + "description": "Gives information on storage for Media Objects. This schema is for the `http_object_store` Storage Backend type which provides URLs for storing Media Objects in object store buckets, and is the only Storage Backend type currently implemented. URLs SHOULD support the inclusion of checksums in headers as supported by advertised Storage Backend product. See AppNote 0048 for more details. Where included in the response, `content-type` MUST match the corresponding value in the request.", "properties": { "media_objects": { "type": "array", diff --git a/api/schemas/flow-video.json b/api/schemas/flow-video.json index 4a7aa0ac..a971b789 100644 --- a/api/schemas/flow-video.json +++ b/api/schemas/flow-video.json @@ -212,6 +212,10 @@ "description": "If `true`, the frame rate of the Flow is variable and `frame_rate` MUST NOT be set. If `false` or omitted, the frame rate of the Flow is fixed and `frame_rate` MUST be set.", "type": "boolean", "default": false + }, + "init_segments": { + "description": "Whether the Flow makes use of initialisation segments. This parameter MUST be set to `true` if Media Objects have `init_object` populated. If set to `true`, all Media Objects MUST have `init_object` populated. Assume `false` if omitted.", + "type": "boolean" } }, "if": { diff --git a/api/schemas/object-core.json b/api/schemas/object-core.json index c22779e2..e7fdb373 100644 --- a/api/schemas/object-core.json +++ b/api/schemas/object-core.json @@ -43,10 +43,6 @@ } ] } - }, - "key_frame_count": { - "description": "The number of key frames in the Media Object. This should be set greater than zero when the Media Object contains key frames that serve as a stream access point", - "type": "integer" } } } diff --git a/api/schemas/object-media-core.json b/api/schemas/object-media-core.json new file mode 100644 index 00000000..ab0e7d74 --- /dev/null +++ b/api/schemas/object-media-core.json @@ -0,0 +1,18 @@ +{ + "type": "object", + "description": "Provides the location and metadata of the media files corresponding to a Media Object.", + "title": "Object", + "allOf": [ + { + "properties": { + "key_frame_count": { + "description": "The number of key frames in the Media Object. This should be set greater than zero when the Media Object contains key frames that serve as a stream access point", + "type": "integer" + } + } + }, + { + "$ref": "object-core.json" + } + ] +} diff --git a/api/schemas/object.json b/api/schemas/object.json index 5eba2810..252684bd 100644 --- a/api/schemas/object.json +++ b/api/schemas/object.json @@ -6,8 +6,7 @@ "type": "object", "required": [ "id", - "referenced_by_flows", - "timerange" + "referenced_by_flows" ], "properties": { "id": { @@ -26,13 +25,35 @@ "$ref": "uuid.json" }, "timerange": { - "description": "The timerange covering the sample timestamps embedded in or derived from the Media Object itself, on the Media Object's timeline.", + "description": "The timerange covering the sample timestamps embedded in or derived from the Media Object itself, on the Media Object's timeline. This parameter MUST be set where the Object contains media. It MUST NOT be set where the Object contains an init segment.", "$ref": "timerange.json" + }, + "init_object": { + "title": "Initialisation Object", + "description": "The Object containing the initialisation segment required to decode the parent Media Object.", + "unevaluatedProperties": false, + "allOf": [ + { + "type": "object", + "required": [ + "id" + ], + "properties": { + "id": { + "description": "The identifier of the initialisation Object.", + "type": "string" + } + } + }, + { + "$ref": "object-core.json" + } + ] } } }, { - "$ref": "object-core.json" + "$ref": "object-media-core.json" } ] } diff --git a/docs/README.md b/docs/README.md index 739a9b5d..d1f34ef9 100644 --- a/docs/README.md +++ b/docs/README.md @@ -27,6 +27,7 @@ For more information on how we use application notes, see [here](./appnotes/READ | [0018](./appnotes/0018-managing-multiple-object-instances.md) | Managing Multiple Object Instances | | [0019](./appnotes/0019-implementing-retention-management.md) | Methods of implementing retention management | | [0021](./appnotes/0021-media-integrity.md) | Integrity model for media in TAMS, and when interacting with other systems | +| [0024](./appnotes/0024-using-init-segments.md) | Using Media with Initialisation Segments in TAMS | ## ADRs @@ -77,12 +78,13 @@ For more information on how we use ADRs, see [here](./adr/README.md). | [0037](./adr/0037-improve-webhooks.md) | Proposal for improvements to the Webhooks endpoints | | [0038](./adr/0038-improved-storage-management.md) | Improved Storage Management | | [0039](./adr/0039-remove-pre-actions.md) | Proposal to remove pre-actions from storage allocation response | -| [0040](./adr/0040-tag-usability-enhancements.md) | Tag Usability Enhancements | -| [0041](./adr/0041-require-explicit-framerate.md) | Requiring explicit frame rates | -| [0042](./adr/0042-uncontrolled-object-instance-labels.md) | Make `label` Mandatory for Uncontrolled Object Instances | -| [0043](./adr/0043-signalling-retention-time.md) | Signalling retention time | -| [0044](./adr/0044-signalling-timeouts.md) | Signalling timeout periods | -| [0046](./adr/0046-governance.md) | Governance | +| [0040](./adr/0040-tag-usability-enhancements.md) | Tag Usability Enhancements | +| [0041](./adr/0041-require-explicit-framerate.md) | Requiring explicit frame rates | +| [0042](./adr/0042-uncontrolled-object-instance-labels.md) | Make `label` Mandatory for Uncontrolled Object Instances | +| [0043](./adr/0043-signalling-retention-time.md) | Signalling retention time | +| [0044](./adr/0044-signalling-timeouts.md) | Signalling timeout periods | +| [0045](./adr/0045-flow-init-segments.md) | Support for init Segments in Flows | +| [0046](./adr/0046-governance.md) | Governance | | [0048](./adr/0048-media-integrity.md) | Integrity model for media in TAMS, and when interacting with other systems | \* Note: ADR 0004a was the unintended result of a number clash in the early development of TAMS which wasn't caught before publication diff --git a/docs/adr/0045-flow-init-segments.md b/docs/adr/0045-flow-init-segments.md new file mode 100644 index 00000000..3a8c97b9 --- /dev/null +++ b/docs/adr/0045-flow-init-segments.md @@ -0,0 +1,204 @@ +--- +status: "proposed" +--- +# Support for init Segments in Flows + +## Context and Problem Statement + +Various media formats employ initialisation segments (aka init segments) for communicating parameters required to configure decoders. +Notably, CMAF-compatible formats such as MPEG DASH and Fragmented MP4 (fMP4). +Init segments are normally communicated via a manifest, alongside media segments. +The decoder will fetch and process the init segment, before identifying the required media segments (which may be mid-stream) and fetching and processing those. + +From a TAMS perspective, init segments are not normally associated with any particular point of time. +A client must be able to write/read, or construct, an init segment independently of the timeline. + +## Decision Drivers + +We would like to: + +* support TAMS features (e.g. edit-by-reference) as fully as possible +* be compatible with as many decoders/players as possible +* place as little burden on implementations as possible with TAMS-specific functionality +* have no/minimal impact on existing (non-init segment based) implementations +* re-use existing patterns/functionality where possible/practical +* maintain, as far as possible, the "independently decodable Objects" property of TAMS +* allow the use of existing init segment extensions (e.g. C2PA) +* avoid specifying Flow parameters that are format-specific + +## Considered Options + +* Option 1: Generate init segments from essence parameters +* Option 2: Significantly expand essence parameters to fully capture everything +* Option 3: Create a Flow Segment with a never timerange for the init segment +* Option 4: Make the init segment a Flow-level property for which a URL can be generated +* Option 5: Make the init segment an object-level property, which can be uploaded and have a URL generated +* Option 6: Have a Flow that is the init segment Flow +* Option 7: Put the init segment somewhere on the TAMS API instead of as an object (e.g. as a base64 blob) +* Option 8: Embed the init segment in every Object + +## Decision Outcome + +Chosen option: Option 5: Make the init segment an object-level property, which can be uploaded and have a URL generated. + +Options 1, 2, 3, and 4 would not allow for Flows which have multiple init segments. +This would place additional restrictions on workflows that TAMS interfaces with, or would require transformation to/from a single-init segment rendition where multiple init segment renditions are required elsewhere. +Examples of such workflows are those which use multi-period manifests, and those which use certain profiles of C2PA. +Additionally, edit-by-reference workflows would be unduly restricted by the need for a single init segment per Flow. + +Option 1, 2, 3, and 7 were considered particularly undesirable in terms of how they interacted with the TAMS data model, clashed with existing patterns, or would have a poor developer experience. + +Option 6 would require implementations generic operations (e.g. store transfer) to have specific understanding/behaviour of media types to operate correctly. +It may also break existing implementations of such functionality silently in some cases (e.g. media segments copied without init segments), or result in unexpected behaviour. + +Option 8 would again place restrictions on workflows that TAMS interfaces with, or would require transformation to/from an embedded-segment rendition. +Such an approach was also identified as being expressly prohibited by some media formats. +Real-world issues were also identified regarding processing load, and smoothness of playback with some player/decoder implementations where superfluous init segments are provided. + +Option 5 provides little/no restrictions on broader workflows. +It provides good backwards compatibility. +It provides good feature compatibility with the likes of edit-by-reference. +It provides minimal burden on client implementations. +It re-uses existing patterns, keeping the burden on service implementations as low as possible. + +### Implementation + +Implemented by . + +## Pros and Cons of the Options + +### Option 1: Generate init segments from essence parameters + +This option would see reading clients (or the store) generate their own init segments based on the existing technical metadata in the Flow metadata. +This is likely not possible with TAMS currently. +The TAMS API likely does not capture all metadata media formats may need to include in init segments. + +* Good, because it requires no changes to the API specification +* Good, because it supports features like edit-by-reference without any modification +* Bad, because initial research indicates this will not be possible due to missing metadata +* Bad, because it requires a potentially complex implementation that requires in-depth knowledge of the structure of init segments and significant domain expertise +* Bad, because extensions such as C2PA metadata may need to be re-created on ingest due to the sequence of init and media segments changing + +### Option 2: Significantly expand essence parameters to fully capture everything + +This option would see Option 1 extended to include the addition of required parameters to Flow metadata. + +* Good, because it supports features like edit-by-reference without any modification +* Bad, because it will likely require significant additions to Flow technical metadata +* Bad, because it will likely require the addition of format specific parameters to Flow technical metadata +* Bad, because it requires a potentially complex implementation that requires in-depth knowledge of the structure of init segments and significant domain expertise +* Bad, because extensions such as C2PA metadata may need to be re-created on ingest due to the sequence of init and media segments changing + +### Option 3: Create a Flow Segment with a never timerange for the init segment + +The TAMS TimeRange format has a special "never" value - `()`. +This currently has no meaning for Flow Segments, as media segments always have a duration. +Init segments can be considered to have no duration. +This option would see the never TimeRange used to represent init segments. + +* Good, because it requires little/no spec changes +* Good, because it doesn't require format-specific Flow parameters +* Neutral, because extensions such as C2PA metadata should be supported as the sequence of init and media segments isn't changed +* Bad, because it features like edit-by-reference would require a different workflow to non-init-segment Flows +* Bad, because it doesn't allow for different init segments for different parts of the Flow + * Relevant formats support multiple init-segments via features such as "multi-Period" manifests + * Some relevant formats may use init segments to signal minor changes to encoding parameters +* Bad, because the approach is somewhat unintuitive +* Bad, because it may result in unexpected behaviour with existing TAMS implementations + +### Option 4: Make the init segment a Flow-level property for which a URL can be generated + +This option would see the init segment managed via the existing Object CRUD mechanisms. +But rather than being registered against a Flow Segment, it would be registered against a new Flow property. +Existing Object patterns would be used for upload, URL generation, re-use, delete, etc. +Like Flow Segments, different Flows could re-use the same init Object. +Implementations might use the init segment's Object ID to determine the compatibility of Flow Segments in edit-by-reference workflows. + +* Good, because it doesn't require format-specific Flow parameters +* Good, because it re-uses existing patterns/mechanisms +* Good, because it supports features like edit-by-reference without any modification + * Though additional checks for Flow compatibility may be required +* Neutral, because it requires a backwards-compatible addition to the spec +* Neutral, because extensions such as C2PA metadata should be supported as the sequence of init and media segments isn't changed +* Bad, because it doesn't allow for different init segments for different parts of the Flow + * Relevant formats support multiple init-segments via features such as "multi-Period" manifests + * Some relevant formats may use init segments to signal minor changes to encoding parameters + +### Option 5: Make the init segment an object-level property, which can be uploaded and have a URL generated + +As with Option 4, this option would see the init segment managed via the existing Object CRUD mechanisms. +Rather than associating the init Object with the Flow, this option would see it associated with each Media Object. +A change in the init segment's Object ID may be used to determine a need to re-provision the init segment in the consuming client. + +* Good, because it doesn't require format-specific Flow parameters +* Good, because it re-uses existing patterns/mechanisms +* Good, because it allows for different init segments for different parts of the Flow + * Relevant formats support multiple init-segments via features such as "multi-Period" manifests + * Some relevant formats may use init segments to signal minor changes to encoding parameters +* Good, because it supports features like edit-by-reference without any modification +* Neutral, because it requires a backwards-compatible addition to the spec +* Neutral, because extensions such as C2PA metadata should be supported as the sequence of init and media segments isn't changed +* Bad, because it will result in a significant increase in Object metadata which makes up a large proportion of data stored in TAMS + +### Option 6: Have a Flow that is the init segment Flow + +This option would see a new init segment Flow type. +An init Flow would be associated with the media Flow. +Init segments would be assigned the TimeRange over which they are valid in the associated media Flow. +This would allow for multiple init segments to be applied to different parts of the media Flow's timeline. + +* Good, because it doesn't require format-specific Flow parameters +* Good, because it re-uses existing patterns/mechanisms +* Good, because it allows for different init segments for different parts of the Flow + * Relevant formats support multiple init-segments via features such as "multi-Period" manifests + * Some relevant formats may use init segments to signal minor changes to encoding parameters +* Neutral, because it requires a backwards-compatible addition to the spec +* Neutral, because it would require open-ended Flow Segments to support live Flows +* Neutral, because extensions such as C2PA metadata should be supported as the sequence of init and media segments isn't changed +* Bad, because it features like edit-by-reference would require a different workflow to non-init-segment Flows +* Bad, because it may be unintuitive +* Bad, because it requires Objects from multiple Flows (init segment Flow, and media segment Flow) to decode a give piece of media + +### Option 7: Put the init segment somewhere on the TAMS API instead of as an object (e.g. as a base64 blob) + +As with Options 4, 5, or 6. +But the init segment would be stored as a base 64 encoded blob (or similar) that may be held in the backend database, instead of as file stored in an object store. + +In addition to the base options: + +* Bad, because it would represent a new pattern where one isn't strictly required +* Bad, because it would require a different GET path to media segments +* Bad, because it would likely require additional work on the part of clients to construct a virtual file for the init segment to pass to decoders/players + +### Option 8: Embed the init segment in every Object + +As with Option 5, but with the init segment embedded in the Object file itself, rather than as a property against the Object. +This would make each Object entirely independently decodable, maintaining this important property of TAMS. +Relevant formats have a concept of "self-initialising" media segments. +But these are, in at least some formats, explicitly not allowed for segmented media such as is used in TAMS. +And are only permitted for playlist-style use cases where a media segment is used to contain a full file. + +* Good, because it doesn't require format-specific Flow parameters +* Good, because it re-uses existing patterns/mechanisms +* Good, because it requires no changes to the API specification +* Good, because it allows for different init segments for different parts of the Flow + * Relevant formats support multiple init-segments via features such as "multi-Period" manifests + * Some relevant formats may use init segments to signal minor changes to encoding parameters +* Good, because it supports features like edit-by-reference without any modification +* Bad, because media using extensions such as C2PA metadata may need to be re-encoded/re-packaged on ingest due to the sequence of init and media segments changing +* Bad, because it will result in an increase in the size of Objects, which makes up a large proportion of data stored in TAMS + * Such an increase is likely to be minor for individual Objects, but significant in larger deployments with relatively small Segment sizes + * Note: This point is in comparison to the other Options in the ADR, rather than to the current situation in the TAMS ecosystem where equivalent metadata is already stored in Objects +* Bad, because this approach may result in un-needed overheads or even interrupted playback in some player implementations + * Some relevant implementations of player technologies, such as Media Source Extensions, are known to re-initialise decoders on (superfluous) new init segments which may result in disruption to playback + +### Option 9: Do not add support for init segments + +Many/all of the above options require significant changes to how the TAMS API is used, or requires formats such as fMP4 to be used in non-standard ways. +This may lead to fragmentation of the TAMS ecosystem, or require fMP4 codecs to be modified in a manner that requires significant in-depth domain knowledge. +We have, however, seen multiple off-spec extensions to the TAMS specification to add init-segment support which itself exacerbates the issue of fragmentation of the TAMS ecosystem. + +* Good, because it encourages TAMS interoperability by maintaining a small number of workflow variations. +* Neutral, because we would explicitly not support a format commonly used in the media industry + * If the way the format would be supported would be non-standard from an fMP4 point of view, TAMS support may bring its own issues - hence "neutral" +* Bad, because this option may persist the existing situation with off-spec extensions diff --git a/docs/appnotes/0024-using-init-segments.md b/docs/appnotes/0024-using-init-segments.md new file mode 100644 index 00000000..83b2b1c7 --- /dev/null +++ b/docs/appnotes/0024-using-init-segments.md @@ -0,0 +1,76 @@ +# 0024: Using Media with Initialisation Segments in TAMS + +> [!NOTE] +> There is an unfortunate overlap in terminology between TAMS and CMAF/DASH/HLS/fMP4. +> As such, where Segments or Flow Segments are referred to capitalised this document is referring to the TAMS resource types. +> Where init segments or media segments are referred to in lower case, this document is referring to the CMAF resource types. +> Where this document refers to Media Objects or Objects, it is referring to the TAMS Media Object resource type. +> Objects (without Media prepended) has been preferred when referring Objects containing initialisation segments to avoid potential confusion "init Media Object" may cause. +> There is no implied distinction between Media Object or Objects in the TAMS API. + +>[!NOTE] +> This Application Note refers to CMAF, MPEG-DASH, HLS, and fMP4. +> It also uses terms of art from those formats. +> It should, however, be read as applying to the general case of any format that uses equivalent concepts of initialisation segments. + +> [!IMPORTANT] +> This Application Note assumes basic knowledge of how to write/read media to/from a TAMS service. + +Some media formats, such as [CMAF](https://www.iso.org/standard/85623.html) compatible MPEG-DASH and fMP4, split content into initialisation (init) segments and media segments. +The init segments contain parameters used to configure decoders common across the stream of media +The media segments generally contain the media itself with additional configuration parameters specific to that media segment. +The split of configuration parameters between init and media segments varies according to the profile used. +When the configuration changes, a new init segment may be required. +While a change in technical parameters would generally require a new Flow in TAMS, some init segment changes may be minor codec parameter changes which would not warrant a new Flow. +A change in init-segment will normally result in a new "period" in DASH, or a "discontinuity" in HLS. +Other formats may have similar concepts. + +Additionally, technologies implemented as extensions to init segments may result in a new init segment for non-technical changes. +One example being C2PA signing of live media (specifically fragmented MP4, applicable in segmented TAMS use cases). +C2PA uses cryptography to verify the provenance of media. +One of the two C2PA live profiles places cryptographic keys in the init segment which may be used to verify media segments and their sequencing. +When that cryptographic key is rotated, a new init segment will be published. +See [C2PA Section 19](https://spec.c2pa.org/specifications/specifications/2.4/specs/C2PA_Specification.html#live-video) for more information on using C2PA with live media. + +TAMS support for init segments makes use of our existing Media Object resource to store and re-use init segments. +These are then associated via their ID to media segment Objects. +This allows existing Media Object re-use mechanisms (i.e. edit-by-reference) to continue to function. +Init segments follow the media. +A client may then detect if the init segment has changed by observing if the init Object ID remains the same or changes between subsequent Flow Segments. + +> [!IMPORTANT] +> Encoder implementations/configurations should minimise the creation of new init segments as much possible. +> Over-use of init segments may result in un-needed re-initialisation of decoders which may adversely affect performance and/or playback. + +## General Workflows + +The general workflows described here refer to init and media segments in isolation from manifests used by specific media formats. +The approaches have, however, been designed to map to manifests where required. + +### Write + +1. [Register the Flow](https://bbc.github.io/tams/main/index.html#/operations/PUT_flows-flowId) + * The `init_segments` essence parameter MUST be set to `true` + * The `container` parameter MUST be set to the mime type of the media segments +2. [Request storage for the init segment(s)](https://bbc.github.io/tams/main/index.html#/operations/POST_flows-flowId-storage) + * The `content-type` MUST be set to the mime type of the init segment(s) + * If the mime type of the init segment(s) matches that of the media segments, this step may be skipped and storage for media and init segments requested together +3. Upload the init segment(s) +4. [Request storage for the media segments](https://bbc.github.io/tams/main/index.html#/operations/POST_flows-flowId-storage) + * `content-type` MUST NOT be set +5. Upload the media segments +6. [Register the media segments as Flow Segments](https://bbc.github.io/tams/main/index.html#/operations/POST_flows-flowId-segments) + * `object_id` must be set to the Object ID of the media segment + * `init_object_id` must be set to the Object ID of the init segment + * Init segment objects should be reused wherever possible + +### Read + +1. [Get the Flow Segments listing for the Flow](https://bbc.github.io/tams/main/index.html#/operations/GET_flows-flowId-segments) +2. For each Flow Segment + 1. If init segment Object ID has changed + 1. Get init segment Object + 2. (Re)initialise decoder with init segment + 2. Get media segment Object + 3. Perform [media timeline mapping](https://github.com/bbc/tams/#flow-and-media-timelines) as usual in TAMS + 4. Play media segment