Professional Documents
Culture Documents
Designing A Distributed Multimedia Synchronization Scheduler
Designing A Distributed Multimedia Synchronization Scheduler
Abstract are due to all the mechanisms wich get the data from the
Distributed multimedia applications need an effective storage medium to the display devices at the user’s end of
synchronization strategy to deal with the temporal the application. Briefly, these mechanisms are: the
disturbances introduced into the data streams by the retrieval of data from the storage medium, the transport
transport and operating systems. An essential part of this service between the storage medium and the displaying
strategy is scheduling - i.e. determining optimal times for application, the displaying application itself, and the
the retrieval of data from the data servers. The CITR underlying operating systems and protocols which run
News-On-Demand project prototype currently uses a the other mechanisms. Synchronization methods can be
centralized synchronization system which lacks effective either preventive or corrective. Preventive mechanisms
high-level scheduling. This paper looks at how to add this are designed to minimize synchronization errors as data
functionality without seriously disturbing the components move through the system to the user. These include
already in place. The proposed solution is a distributed scheduling, which is the process of organizing and
synchronization scheduler, designed and implemented in a coordinating the retrieval of data from the storage
way which makes use of the current prototype’s medium, and efficient transport mechanisms and
components. operating systems, which allow for the precise control of
timing constraints. Corrective methods involve adjusting
the presentation of the data once they are received by the
1. Introduction displaying application so that the appearance of
synchrony is maintained. Synchronization strategies in
Multimedia technology - still one of the primary areas multimedia applications use these methods in varying
of research in the computing world - is the ability to degrees.
integrate the traditional media formats, i.e. text, graphics, An example of such an application is the News-on-
sound and moving images, into a single document. The Demand system being developed as part of the research in
media which were first used in computer applications - the Broadband Services major project of the Canadian
text and graphics - are relatively simple in that they have Institute for Telecommunications Research (CITR). This
no implicit time semantics or constraints; only their project is made up of many sub-projects being carried out
spatial semantics (e.g., where a paragraph should be by different research teams. The sub-projects include a
placed on a screen, or how big the font should be) are file system for time-dependent media (the Continuous
relevant to the user. Audio and video, however, are so- Media File Server, or CMFS), a Quality-of-Service
called time-continuous or time-dependent media: they negotiation and monitoring system, database technology
only make sense if they are played back in the right order and synchronization of data.
and at the right speed. Displaying and integrating these The synchronization research is being done by a team
media types, therefore, requires some kind of temporal from the Multimedia Communications Research
coordination. This means that a multimedia application Laboratory (MCRLab), of the University of Ottawa’s
cannot be designed without taking the temporal Electrical and Computer Engineering department. The
relationships between its components into account. MCRLab’s synchronization strategy looks at both the
Multimedia synchronization usually refers to these preventive and corrective aspects of synchronization.
temporal relationships among the different media [1]. Prevention is mainly at the scheduling level. Due to
Synchronization errors - disturbances in the temporal technical considerations, the current overall project
semantics of data, such as a varying frame rate for video - prototype uses a centralized design for the scheduler [2].
This means that only one module, located at the client, - the object layer, which integrates streams and time-
organizes and coordinates the retrieval of the data form independent data such as text and still images. The
the storage medium - in this case, the CMFS and the difference between the two types of objects are hidden
text database. However, there is research [3] which here. Synchronization at this level is called inter-
suggests that a distributed scheduler, where the servers object synchronization.
participate in the scheduling of the multimedia data, These levels allow us to organize and study the types
would be more efficient, particularly for the client. The of temporal constrains between multimedia data, and the
MCRLab has therefore committed itself to re-designing synchronization errors that occur during the playback of a
the scheduler in the current prototype to make it multimedia document.
distributed.
This paper will therefore study the issues involved in
changing from a centralized to a distributed scheduler and 2.2 System components and how they affect
the implementation of such a scheduler. The rest of the synchronization.
paper is organized as follows. Section two studies
synchronization issues. Section three presents and As it was said before, there are many links in the
compares different scheduler architectures. Section four chain which moves data from its storage site to the user.
discusses what needs to be changed in the current Each of these links performs a certain task and affects the
prototype to move to a distributed scheduler. Section five data in a different way. They all tend to introduce
discusses the actual implementation of the scheduler. synchronization errors, unless care is taken to avoid this.
Section six points to further research and concludes the Servers store data and put that data into the network; the
paper. network transports the data to the client; the client reads
the data from the network and presents it to the user;
operating systems and protocols allow these systems to
2. Synchronization Concepts run and do their work.
Obviously, given the special temporal constraints of
multimedia data, as well as their resource requirements
2.1 Media granularity and semantics (high bandwidth and high resolution for good video), the
systems described above have to be designed or chosen
There can be many different types of temporal with care. In particular, the servers must be able to store
relations between the components of multimedia data. A large amounts of data in such a way that retrieval is
good way to classify them is to look at the different quick and efficient; networks must provide high
semantic levels of multimedia data. A reference model for bandwidth and low transmission rate variations;
organizing these levels has been given in [1]. Briefly operating systems and applications must be able to
stated, this model recognizes three levels of organization provide real-time data processing (retrieval, re-
inside multimedia documents, and one level “above” the synchronization, display) [4].
document, which allows the author of the document to Synchronization methods usually aim at making the
specify temporal relations. The levels inside a document best of the properties and functionalities of the
are: “infrastructure” an application is using, not to change
- the media layer, which looks at the semantics inside them; the synchronization research at the MCRLab thus
a single stream of time-dependent data. The stream is concentrates mainly on server- and client- side
viewed as a succession of Logical Data Units, or application-level strategies.
LDUs, which denote the lowest level of data
granularity seen by the application. This could be
2.3 Synchronization strategies
frames, groups of frames, data blocks, etc.
Synchronization at this level is referred to as intra-
stream synchronization. 2.3.1 Prevention. Preventing synchronization errors
- the stream layer, which looks at the relations involves minimizing these errors. This requires
between whole streams, and groups of streams. LDUs eliminating any inefficiencies and latencies in the
are hidden at this level, streams are seen as a whole. mechanisms which move data from the storage medium to
Synchronization at this level is known as inter- the user. These mainly involve disk-reading scheduling
stream synchronization. policies, network transport protocols and operating
systems which allow for real-time computing and for
temporal semantics of data to be taken into account.
Looking at the data semantics introduced earlier, it can be The client decides when it wants to present data and when
said that “infrastructure” systems take care of intra-stream it wants to request the data from the servers. The servers
synchronization, while applications usually use this take care of low-level (i.e. intra-stream) scheduling only.
“infrastructure” to ensure inter-stream and inter-object The client then takes care of high-level and low-level
synchronization, although some operating systems might synchronization with the SSP. Figure 1 shows a diagram
provide facilities for inter-stream synchronization [1] and of the current system prototype that has been
some applications might delve into intra-stream implemented using a centralized scheduler [7].
synchronization. At any rate, the prevention methods The user interacts with the application through a
developed at the MCRLab concentrate on providing inter- Graphical User Interface (GUI). The user first chooses an
object scheduling using lower-level mechanisms and a article from the list of available articles that is presented
temporal notation called Time Flow Graphs (TFG) [5]. A to him/her by the GUI. This request is then sent to the
scheduler uses the specifications for a presentation Database Server. The Database Server returns the article
described with this notation to create a schedule for the scenario to the scheduler, which takes care of all
delivery of the objects to the client by the servers scheduling tasks (starting data readers, computing
(delivery schedule) and the presentation of these objects to schedules, and so on) in the client application. The
the user by the client application (presentation schedule) scheduler uses the scenario to compute an acceptable
[6]. This scheduler can be centralized - entirely located at presentation schedule. Following this schedule, the client
the client - or distributed - the delivery scheduling
LEGEND
functionalities are shared among the servers and the client. CMFS
COMPONENTS:
So far research into the distributed scheduler has been LLSC
CMFS
TEXT
DBS: Database Server
CMFS: Continuous Media File Sever (A/V)
LLSC TEXT: Text Server
mainly theoretical; later sections of this paper will discuss DBS
LLSC NW: Transport Network
CLIENT: Client Application
the actual implementation of such a scheduler. CM
DS
CM
GUI:
SCH:
Graphical User Interface
Scheduler Process
DR: Data Reader Process
NW MPB: Media PlayBack Process
(Audio, Video, Text)
2.3.2 Correction. If errors can’t be prevented - and MI DS
CM
DS
COMMUNICATIONS:
this being the real world, they can’t - they must be GUI DR CM Control Messages
DR
rectified. This is to be done by the client application CM
MPB
C MPB
DR DS Data Streams
C MPB
receiving the data. The method used by the MCRLab is SCH HLSC C
Meta-Information
(Scenario & Scheduling)
LLSC
the Stream Synchronization Protocol (SSP)[2]: units CM
SYNCHRONIZATION FUNCTIONALITIES:
which control and monitor the client-end of the data CLIENT HLSC
High-Level Scheduling (Streams, Objects)
LLSC Low-Level Scheduling (Inside Streams)
connections compare the real arrival times of data with C Correction (SSP Protocol)
the ones predicted by the presentation schedule and notify
the scheduler of any discrepancies. These discrepancies Figure 1: System prototype with a
are then compensated for by the scheduler, which delays centralized scheduler
the display of data that are “ahead” of other data, allowing
the late data to “catch up.” Since this is done entirely at
opens connections between itself and the servers and
the client side, the architecture of the scheduler does not
requests the different media that needs to present the
affect the SSP, and the SSP does not affect the servers.
article. There is no explicit delivery schedule in this
model. When the user gives the command to start
displaying the article, the client starts reading data from
3. Scheduler Models
the connections and displaying them. The data streams
are constantly monitored by the data readers, which have
a Media Synchronization Controller (MSC); this MSC
3.1 The centralized scheduler knows the presentation schedule and notifies the
scheduler when synchronization errors (with regards to
As has been previously stated, in this model all of the the schedule) are detected. The scheduler then re-schedules
object-level scheduling is performed at the client by a the reading or the display of certain streams, using the
process called a scheduler. When a user requests an Stream Synchronization Protocol [7].
article, the scheduler receives from the database server the It is clear from the figure that control over the system
scenario for the article. The scheduler then computes a is centralized at the client: the client determines the
presentation schedule which fits the scenario, as well as a schedules, opens the connections and requests the data.
delivery schedule for the servers, which is basically the This type of architecture requires the client application to
presentation scenario adjusted for network conditions. be quite intelligent. Since it is desirable that the client
application be kept simple, so that it will easily run on a scheduler is notified of any errors and re-schedules the
wide variety of platforms, this should be avoided. rest of the presentation accordingly. For example, if an
Another drawback is that there is no proper high-level object arrives late, the scheduler delays the presentation
(i.e. stream and object layer) scheduling in this model, of objects which are to be presented either in parallel
only correction measures. Servers with some kind of with, or after, the late object. This co-operation between
high-level scheduling capability would help reduce the scheduler and the CMSCs implements the SSP
synchronization errors, thus lightening the workload of correction protocol alluded to in section 2.3.2 and
the client. As it stands, the client does all of the explained in more detail in [7].
synchronization work, except for media layer delivery The overall architecture of this system is illustrated in
scheduling. figure 2, with a particular emphasis on control messages
between the components .
server to the client and to the TSCs. The TSCs use the
scenario to open the appropriate connections to the Figure 2: System architecture with a
client; in doing so, they obtain the end-to-end network distributed scheduler
delay on each connection. The TSCs then exchange
information about the different objects in the Clearly, the servers have a more complicated
presentation and network conditions to derive a structure in this model, since they have an extra
presentation schedule and a delivery schedule. The scheduler module, effectively giving them a two-tiered
delivery schedule, i.e. the optimal times for transmitting architecture. Communications between the components
data to the client, is derived from the presentation are also more complicated, as all the servers must
schedule by subtracting the total delay that each object exchange information. But while the servers must now
encounters from the times in the presentation schedule. perform more tasks, the client does not have to derive a
The total delay for each object is the sum of the end-to- delivery schedule any more, and the system has stream
end network delay on its connection, the buffer delay and object layer scheduling capabilities, which are absent
upon reception by the client, and the decoding time for from the centralized scheduler architecture seen in the
the object [6] [7]. The presentation schedule is also previous section. This should improve performance by
derived at the client from the scenario and the TFG for reducing synchronization errors, therefore making
the presentation. recovery operations more efficient, and by simplifying
The connection-endpoints are controlled by Media the client application.
Synchronization Controllers (MSCs), which ensure
intra-stream synchronization. The server MSCs
(SMSCs) are directly responsible for opening the 4. Re-engineering the Current Model
connections requested by the TSCs and for ensuring
media layer scheduling for each stream. The client MSCs 4.1 Assessing the centralized model
(CMSCs) are in charge of monitoring synchronization
errors upon reception of the streams by the client. Errors After deciding what the requirements of the new
are detected by comparing the actual arrival time of an architecture are to be, i.e. a distributed scheduler shared
object or an LDU with its projected arrival time based on between the client and the servers, the centralized model
the schedule provided by the client scheduler. The
CMI
has to be assessed to see how similar it is to the CMI LEGEND
COMMUNICATIONS:
NW
figure 1. The client appears to have all it needs, since it CM Control Messages
MPB
CMSC
CMI Control & Meta-Infornation Messages
C
more complicated: they include TSC modules for high- SCH LLSC
C MPB
C SYNCHRONIZATION FUNCTIONALITIES:
HLSC HLSC
level (inter-object and inter-stream) scheduling, absent CM
High-Level Scheduling (Streams, Objects)
LLSC Low-Level Scheduling (Inside Streams)
from the centralized model, which control the CLIENT
C Correction (SSP Protocol)
HLSC
TSC
HLSC
TEXT: Text Server
TSC: Temporal Scheduler Controller
CM
SMSC: Server Media Synchronization Controller
-report the status of the request to the client AVS CM
TEXT
CMFS: Continuous Media File Server
NW: Transport Network
CMFS SMCS
-The TSC goes back to waiting for a new message LLSC LLSC
CLIENT: Client Application
GUI: Graphical User Interface
SCH: Scheduler Process
MI
CMI DS
CMSC: Client Media Synchronization Controller
DS MPB: Media PlayBack Process
(Audio, Video, Text)
The CMFS consists of two types of processes: the DS
COMMUNICATIONS:
NW
cmfsadm and the cmfsnode. The cmfsnode manages the CM Control Messages
MPB
CMSC
CMI Control & Meta-Infornation Messages
C
the appropriate cmfsnode process. There can be many SCH LLSC
C MPB
C SYNCHRONIZATION FUNCTIONALITIES:
CM
HLSC High-Level Scheduling (Streams, Objects)
LLSC Low-Level Scheduling (Inside Streams)
only contains one cmfsadm. The CMFS API has been CLIENT
C Correction (SSP Protocol)