Trev 301 Middleware
Trev 301 Middleware
Architectures for
System integration
Broadcast production systems typically were built as a collection of silo or island devices. However, with the increasing use of IT in production, and the sharing of common resources such as storage and connectivity, the question becomes: how best can we glue all this equipment together into an integrated system? This is exactly what EBU Project Group P/MDP has been studying.
It is a fact of life that there will rarely, if ever, be global agreement on common standards in some fields. Which nation will give up their national language for a universal Esperanto? Technology is no different. Broadcasters will remember the camps of PAL and NTSC. Accepting this fact leads us to look for solutions that allow interchange between different standards what we choose to call Middleware. In the case of PAL/NTSC, the answer was the standards converter an early example of middleware. At a more basic level, consider the variety of different mains power connectors which exist throughout the world. To enable your equipment to work anywhere, it is necessary to have a different mains cable for each type of connector. And then came the middleware the universal mains adapter.
System 1 System 2
Figure 1 A familiar problem with a well-known solution
System . n
In broadcasting these days, every system tends to be a framework; with endless possibilities, but only working if tightly integrated with all of the other systems (frameworks) of the organization. This can be very hard to achieve. Even if you do reach some level of integration, full interoperability is seldom achieved. It tends to be expensive, takes a lot of time to configure and implement, and fails to fully realize the opportunities for flexibility. On top of that, the demands to constantly change and optimize our workflows using these systems are increasing. We are missing the universal adapter between our systems!
1 / 13
System 4
System 3
DISTRIBUTED PRODUCTION
Buy programmes
Produce programmes
Development, service, support & operation of: Archive Acquire & manage: Marketing & promotion of: Rights Programmes, services & brand
The integration architecture System operations ensures that information and Administration functions are accessible from all the process stages from Figure 3 early season planning to playTypical broadcaster's value chain (source: DR) out via production, distribution and transmission. It is even relevant in the delivery platform where some data reaches the set-top box (e.g. the EPG).
Broadcast-specific scope
In many parts of our operations, we can benefit from integration techniques commonly used in other industries. But some areas are more critical to broadcasters than to other industries. This is because broadcasting deals with real-time rich media and not just common data in the form of text and numbers: 1) Media asset management Storage and archiving of our products: video, audio and interactive services. This is about the general integration architecture between our assets and all of our other systems. 2) The production environment Where can benefits and new ways of working be realized using integration technologies? 3) The play-out environment We need to ensure data and content integrity from planning and scheduling, the production, through play-out, to the end user. 4) Workflow management We need to connect the workflows which relate to the processes in the whole value chain across systems or modules of functionality.
1. P/MDP stands for Middleware in Distributed programme Production. The Group was created in autumn 2003 and consisted of various EBU members. The main contributions to its report and this article came from: the BBC, BR, DR, the IRT, NRK, RAI, RTR, SVT and VRT.
EBU TECHNICAL REVIEW January 2005 EBU Project Group P/MDP 2 / 13
DISTRIBUTED PRODUCTION
Examples
Several EBU members have already gained experience with system integration architectures: Bayerischer Rundfunk is implementing an IT-based play-out and production centre. It encompasses a messaging-based system for system integration. Danish Radio is in the process of moving from a vertically to a horizontally integrated system architecture. Standards and policies are established and both a broker and a concept for service-oriented integration are in operation. This has been done in close connection with the implementation of a DR metadata specification.
EBU TECHNICAL REVIEW January 2005 EBU Project Group P/MDP 3 / 13
DISTRIBUTED PRODUCTION
BBC Research & Development is studying a completely horizontally integrated production architecture. This project can be regarded as having the luxury of a green field situation, trying to utilise the benefits of horizontal system integration to its fullest extent.
Key concepts
We are used to working with isolated systems, interfaced by people. These are systems in co-existence. Today this is changing. The main task of system integration today is to turn co-existing systems into co-operating systems, to collect systems into super-systems, when appropriate and efficient, and to optimize workflows.
Introducing layers
In the late nineties the primary view was that one had to connect all systems or modules to each other directly. However, for a large number of modules, this becomes almost unmanageable. The alternative is to connect the modules via some common glue. But this glue needs to be well-defined and widely accepted. The development of a layered system architecture was a good first step. A system can be seen as split into layers in a horizontal architecture where each layer is independently accessible. Systems without this approach are referred to as vertical systems.
-> provides input & output for the system; -> the brains of the system; -> the memory or content of the system (databases and fileservers).
In many situations the concept can benefit from introducing a layer under the databases and fileservers, taking into account that physical storage can differ. This then allows you to change or scale the storage without changing the data layer. So the fourth layer would be: 4) Storage -> the physical place where data are put.
DISTRIBUTED PRODUCTION
On the other hand, some older and non-layered systems have quite open interfaces, although there is a tendency to refer to non-layered systems as vertical and closed. In the real world, we are dealing with a complex mixture of scenarios, and the solutions tend to be equally diverse.
Component-based systems
The layers mentioned are an abstraction, introduced to illustrate logical groups within systems, which in development and daily work are useful to treat separately. We can go one step further and describe systems which are component-based 3. A component is a black box with known and described interfaces which can be used without knowing anything about the internal structure or internal functionality of the box. These components can be grouped inside the above-mentioned layers, although some components work across layers as well. As long as they stick to the rules in the above definition, that should not be a problem.
Some of the components' interfaces should be openly accessible, but not necessarily all. This can be achieved in many ways and is one area where the need for standardization arises. The main point to remember is that systems should be component-based.
1 2 N
Hub
3 ...
Spokes (adapters)
3
...
N x (N-1) / 2 connections
Cheap to do with few modules, but complex and expensive to maintain.
N connections
Greater effort to start with, but easier to maintain and scale. Cheap to connect to.
One of the early techniques to act as glue was the data-broker Figure 6 (a hub and some adapters to Example of glue the systems). Without going in to technical details, the concept and expected benefits are illustrated in Fig. 6.
Fig. 7 combines the concept of middleware (glue) with the layers introduced earlier. The way the middleware interconnects the data layer with the application (logic) and the presentation to the user, is by using communication via interfaces. These interfaces form the boundaries of
3. The term component-based is preferred to object-oriented because object-oriented is just one example of a component-based design, e.g. a procedural design can also be component-based.
5 / 13
DISTRIBUTED PRODUCTION
System 1
the puzzle pieces in Fig. 7. It is important that the interfaces allow for communication in a common, flexible, structured, efficient and re-useable way. Where do components fit into this? As we discussed before, systems are nowadays more and more component-based. Fig. 8 extends our layered model with this notion. The components are drawn as little puzzle pieces, indicating they have their own interfaces. This means we can speak about interfaces at multiple levels: for example, the interface to the data-layer or the interface to a component within that layer. Components could be system components or middleware components. Working at component level is leading us to another kind of glue indeed another integration technique: Web services.
Application logic
Middleware
Application logic
Application logic
Middleware
Application logic
= component
Figure 8 Systems can be built out of many small components
The resulting system integration architecture depends on your particular business model (the products and operations) and your particular IT environment. Here we describe four common integration types. Message-based integration (copying data between systems) The same data is needed in different systems and an automated exchange mechanism provides copies of data in these systems and ensures data-integrity while doing so. It can be set up as rule-based or event-based. Functionality is made available by an integration broker.
System 1 System 2
Data-carrying integration (Different systems access the same data) Data exists as a single copy in a single location. All systems work real-time on the same instance of data. Integration is achieved by agreement between the systems about the data model they use, and about a common logic for reading and updating data. This is provided by a data-carrying integration platform, primarily consisting of an application server and a database.
EBU TECHNICAL REVIEW January 2005 EBU Project Group P/MDP 6 / 13
DISTRIBUTED PRODUCTION
System 1
System 2
Portal integration (Wrapping of multiple user interfaces into a single one) Employed where users require a more integrated user dialogue or interface than is provided by the separate applications. Where individual customizing of the user interface is required, or where certain functions should be automated, or where single logon is needed ... an integration of the user interface is necessary. Web-based user interfaces provide integration via a portal.
System 1 System 2
Workflow (Mechanisms to co-ordinate events in systems) Refers to automating or semi-automating of workflows, by integrating functionality and data. To support tightly connected or integrated business processes it should be possible in advance to define which actions need to take place when certain data are exchanged or updated. Workflow mechanisms can be found in each of the three former described integration types.
Process Process 1 2
Process Process 3 4
Video
Data Data
Workstation Server
Laptop computer
The above integration architecture concepts are related to planning, production, and play-out systems, as well as the administrative back-office, financial and accounting systems. When it comes to executing play-out or transfer of real-time, rich multimedia files (e.g. a 50 Mbit/s video file) other tools and technologies than described above are used. However, the management, control and reporting of this can be handled like any other kind of data in any other industry.
DISTRIBUTED PRODUCTION
Abbreviations
AAF API MXF NTSC Advanced Authoring Format Application Programming Interface Material eXchange Format National Television System Committee (USA) PAL Phase Alternation Line SMPTE Society of Motion Picture and Television Engineers UML Unified Modelling Language VTR Video Tape Recorder
Understanding services
A service is basically a special system component which can be invoked by other components wanting a certain task to be performed on certain data, e.g. an authentication service which is granting or denying access to resources. To be able to do this, it is necessary for the invoking components to know what operations are available and how to ask for these operations. The specification of these is called a service interface specification.
Pervasive services
Pervasive services provide functionality which is not specific to the broadcasting needs. Pervasive services are reusable by other services and applications. The use of pervasive services allows us to benefit from solutions worked out in other industries, for example:
EBU TECHNICAL REVIEW January 2005 EBU Project Group P/MDP 8 / 13
DISTRIBUTED PRODUCTION
Authentication; Session management; Time references; Workflow management. For the pervasive services, the users should select available services and implementing technologies which are appropriate to support their business. The proper requirements have to be applied to the services (e.g. the bandwidth needed and the desired network characteristics), to guarantee they meet the users' specific operating conditions.
Different views
There are different types of players involved in the development of IT-based TV programme production systems. These different roles require different views of the system which is modelled, for example: Planning Engineer View a set of service definitions and high-level descriptions of their interactions. System Integrator View a description of how to use a service. The System Integrator View has to represent a high-level view of the interfaces of the services and their dependencies on other services (e.g. the pervasive services). System Developer View all the detailed information about the components to be implemented.
DISTRIBUTED PRODUCTION
2) A formalised text part defining the actors, the preconditions and the business rules relevant for the Use Case. Several Use Cases can be collected into logical groups to form a drawing of a Use Case scenario, including text parts for each Use Case. There are frameworks which give some guidance in how to deal with system analysis and description. For more information look for MDA in [1] and Zachmann in [2].
VTR
Play
Record
Stop
Play-out system
Key observations
Vendors are opening up their systems
Openness is the norm in IT, but is only slowly becoming practice in the broadcast domain. One reason for this is the large influence of traditional wholesale providers. The main driver to open up is, quite simply, pressure from customers demanding openness.
DISTRIBUTED PRODUCTION
Standardization
There are two areas where standardization efforts would be welcome: 1) A common (core) metadata model There is a lack of a common (core) metadata model. This is the top priority issue for most vendors. Existing metadata specifications have not been very successful and are seen as too large/vague to apply. There is a strong demand for a more modular and business-oriented approach: e.g. a simple subset with a coherent structure. This must include the semantics (= unambiguous meaning) of the elements it provides. 2) Bread & butter services The business services commonly used by broadcasters, such as ingest, play-out and scheduling are the broadcasters' bread & butter services. It seems that standardization of middleware / system integration architectures could best take place at this level. These primary services could all be broken down into non-competitive and common technical services (play, record, transfer, transcode, etc.) and all have a need to interface. Besides services, relevant business objects should be specified as well. The specifications should not dictate technology,
EBU TECHNICAL REVIEW January 2005 EBU Project Group P/MDP 11 / 13
DISTRIBUTED PRODUCTION
but allow for competition. Once the services are defined, it should be investigated if toolkits, APIs, etc., would be of value.
Golden Rules
The experiences shared by the P/MDP Members resulted in a list of Golden Rules for broadcasters, vendors and standards organizations. Ten are listed below but see the full report [3] for the complete list.
Table 1 Ten golden rules for system integration
#
1 2 3 4 5 6 7 8 9 10
Golden Rule
Middleware is infrastructure Don't talk systems, talk services or functions Clearly define your process workflows Know your world Own your world Partner with your vendor(s) Demand open modularity Demand open standards Demand free choice of storage Take real-time seriously
Notes
Treat it as such: e.g. financially, management Otherwise you will see a misleading picture Look at middleware from the business perspective Set up a central information repository Define your vital business (objects) yourself Make sure it is clear where the risk is borne Proprietary combinations will lock you in Proprietary interfaces are too limited Not one which is tied in with other components Plan for it from the beginning, don't use quick fixes
12 / 13
DISTRIBUTED PRODUCTION
Chris Chambers joined the BBC as an Engineering trainee and this year is his 37th with the Corporation! Over the years, he has held a number of posts covering the installation of broadcast facilities in Outside Broadcasts, Studios, Broadcast Infrastructure and Data Services. He has also been involved with the strategic development of broadcasting systems, looking into ATM- and IP-based networking developments as well as file standards for broadcasting use. He has been a participant on a number of internal BBC groups covering these topics and has produced a number of papers on these subjects. Currently in his post at BBC R&D, Mr Chambers runs a small team investigating networks and storage. He has been heavily involved in developments using ATM network structures to support live broadcasting within production areas, along with working on new standards within the AES and the IEC to support the audio on such systems. A part of the work of his team is to support the Desktop Production project within BBC R&D that aims to develop systems for complete broadcast and production structures using standard IT components. Chris Chambers continues to represent the BBC in EBU project groups and currently chairs two of these, one working on the development of middleware technologies (P/MDP) and the other covering common control strategies for live structures (N/CNCS) both of these areas can be applied to production and broadcast systems. He also chairs an AES standard working group on developing metadata structures and is currently working on two IEC standardization projects.
References
[1] http://www.omg.org [2] http://www.zifa.com [3] EBU doc. Tech 3300: The Middleware Report EBU doc. Tech 3300-s: Supplement to the Middleware Report Available via http://www.ebu.ch/en/technical/publications/tech3000_series/
13 / 13