Abstractions From Multimedia Hardware

Lehrstuhl für Informatik 4 Lehrstuhl für Informatik 4
Kommunikation und verteilte Systeme Kommunikation und verteilte Systeme
Abstractions from Multimedia Hardware

State of the art of programming
Chapter 2: Basics
• Application code is highly dependent on hardware
Chapter 3: Multimedia Systems – Communication Aspects and Services • Change of multimedia devices still often requires re-implementation
Chapter 4: Multimedia Systems – Storage Aspects • Example: sequence of (very) specific instructions:
Chapter 5: Multimedia Usage and Applications static unsigned long read_timer(void)
• Documents and Hypermedia {
/* example taken from Linux kernel code - i386 architecture */
• Multimedia User Interfaces
unsigned long t, flags;
• Abstractions for Programming 5.3: Abstractions for Programming int i;
• Multimedia Applications • Abstraction Levels __save_flags(flags);
__cli();
• System Software and t = jiffies * 11932;
Libraries outb_p(0, 0x43); Strong hardware dependency may cause
• Frameworks i = inb_p(0x40); problems with:
i |= inb(0x40) << 8;
• Higher Programming • Portability
__restore_flags(flags);
Languages return (t - i); • Reusability
• Object-Oriented Approaches } • Coding efficiency
Chapter 5.3: Abstractions for Programming Page 1 Chapter 5.3: Abstractions for Programming Page 2

Abstraction Levels Libraries

• Common operating system extensions try to solve this problem
• Different programming possibilities for accessing and representing multimedia data: • Processing of continuous media based on functions embedded in libraries
• Libraries differ in their degree of abstraction
Multimedia application • Example from IBM’s early Audio Visual Connection (AVC):
acb.channel = AAPI_CHNA
Object-oriented Higher acb.mode = AAPI_PLAY
programming programming ...
languages languages
aud_init(&acb) /* acb is the audio control block */
...
Frameworks audrc = fab_open(AudioFullFileName,AAFB_OPEN,AAFB_EXNO, 0,&fab,0,0,0,0);
System software fork(START in PARALLEL)
aud_strt(&acb)
Libraries displayPosition(RelativeStarttime, Duration)
...
Device drivers for continuous media
Device
Libraries - OpenGL System Software
2D and 3D graphics API developed by Silicon Graphics Device access becomes part of the operating system:
• Basic idea: “write applications once, deploy across many platforms”: • Data as time capsules (file extensions)
PCs Each Logical Data Unit (LDU) carries in its time capsule its data type, actual
Workstations value and valid life span
Super Computers Useful concept for video, where each frame has a valid life span of 40ms (rate
of read access during a normal presentation)
• Benefits:
Presentation rate is changed for VCR (Video Cassette Recorder) functions like
Stable fast forward, slow forward or fast rewind by
Reliable and Portable • Changing the presentation life span of a LDU
Evolving • Skipping of LDUs or repetition of LDUs
Scalable (Features like Zoom, Rectangle handling ...)
• Data as streams
Well documented and easy to use
A stream denotes the continuous flow of audio and video data between a
• Integrated with: source and a sink
Windows 95/NT/2000/XP Prior to the flow the stream is established equivalent to the setup of a
UNIX X Window System connection in a networked environment

System Software - MCI System Software - DirectX
Windows Media Control Interface (MCI): Low-level APIs and libraries for high-performance applications
• Especially games - formerly known as the "Game SDK"
Direct access to hardware services
• E.g. audio & video cards, hardware accelerators
• “DirectX” = “direct access”
MMSYSTEM library for • Strong relationship/interaction with ActiveX/DCOM
extensibility and device
independence
System Software - DirectX Frameworks
Additional level of abstraction:

Components:
• DirectDraw - 2 dimensional graphics capabilities • Combination of different media
• Direct3D - extensively functional 3D graphics programming API • New data types
• DirectSound - (3D) sound, mixing and playback of multiple streams E.g. mixed.video := orig.video + subtitle
• DirectPlay - for network multiplayer game development Granularity problematic (pixel, frame, image group, sequence, video, ...)
• DirectInput - input from various peripherals, e.g. joysticks, data gloves • Reuse of modules:
Implementation Strategy: Synchronization
• Hardware Abstraction Layer (HAL) Scaling
• Hardware Emulation Layer (HEL) (De)compression
• Media Layer (for aggregated “high level” functionality) File formats
Animations …
Media streaming
Application frame given: generic problem solution
Synchronization
Creation of specialized application components: media containers, media processors

Frameworks: JMF Higher Programming Languages

Framework controls execution Media as data types:
• E.g. myApplicationComponent.init(); myApplication.start(); • Definition of appropriate data types (e.g. for video and audio)
• Example: Java Media Framework (JMF) • Smallest unit can be a LDU
• Example of merging a text and a motion picture:
JMF supports platform-neutral synchronized playback of multimedia data
subtitle TEXT_STRING PROCESS_2
• Local or via network: mixed.video, ldu.video VIDEO_LDU; output(video_window,mixed.video)
Pull protocols - e.g. HTTP ... ...
Push protocols - e.g. RTP WHILE END_WHILE
Easy integration into platforms native environment and core packages; supported media COBEGIN ...
formats: PROCESS_1
• MPEG, QuickTime, AVI input(av_filehandle,ldu.video)
IF new_video_scene
• WAV, AU, MIDI
input(subtitle_filehandle,subtitle)
Extensible: mixed.video := ldu.video + subtitle
• New Data Sources (e.g. for FTP or Video on Demand (VOD) protocols)
• New Players In the above example, implicit type conversion must occur
Real-time processing of LDUs depend on their granularity and can be problematic in a
Chapter 5.3: Abstractions for Programming Page 11 Chapter 5.3: Abstractions
high-level language for Programming Page 12
Higher Programming Languages Higher Programming Languages
Media as files: instead of considering continuous media as data types they can be
considered as files: Media as processes: it is possible to map continuous media to processes and
file_h1 = open(MICROPHONE_1,...) integrate them into a high-level language:
file_h2 = open(MICROPHONE_2,...)
file_h3 = open(SPEAKER, ...)
PROCESS cont_process_a; This process implements a set of actions
... (“set-volume”, “set-loudness”)
...
on_message_do
read(file_h1)
set_volume ...
read(file_h2)
set_loudness ...
mix(file_h3, file_h1, file_h2) ...
activate(file_h1, file_h2, file_h3) ...
... [main]
deactivate(file_h1, file_h2, file_h3) pid = create(cont_process_a)
... send(pid, set_volume, 3)
rc1 = close(file_h1) send(pid, set_loudness)
rc2 = close(file_h2) ...
rc3 = close(file_h3)

Programming Language Requirements Object-Oriented Approaches
The high-level language should support parallel processing, because the processing
of continuous data is Basic ideas of object-oriented programming is data encapsulation in connection with
class and object definitions
• controlled by the language through pure asynchronous instructions
• Abstract Type Definition (definition of data types through abstract interfaces)
• an integral part of a program through the identification of media
• Class (implementation of a abstract data type)
Different processes must be able to communicate through an inter-process
• Object (instance of a class)
communication mechanism, which must be able to:
• Understand a priori and/or implicitly specified time requirements (QoS parameters or Other important properties of object-oriented systems are:
extracted from the data type) • Inheritance
• Transmit the continuous data according to the requirements • Polymorphism
• Initiate the processing of the received continuous process on time
Object-oriented Approaches Object-Oriented Approaches

Devices as classes: devices are assigned to objects which represent their behavior
and interface Processing units as classes:
class media_device { • Three main objects:
char *name; Source objects
public: Destination objects
void on(), off();
}; Combined source-destination objects allows the creation of data flow paths
through connection of objects
• Multimedia object
Basic Multimedia Classes (BMCs) /
Basic Multimedia Objects (BMOs)
class media_in_device: class media_out_device: Compound Multimedia Classes (CMCs) /
public media_device { public media_device { Compound Multimedia Objects (CMO), which are compound of BMCs / BMOs
private: public: and other CMCs/CMOs
DATA data;
public:
void put_data(refDATA dat); BMOs and CMOs can be distributed over different computer nodes
refDATA get_data(); };
};

Object-oriented Approaches Object-Oriented Approaches

Media as classes:
Example of CMC (simplified notation): • Media Class Hierarchies define hierarchical relations for different media
Lexicon: compound_class; • Different class hierarchies are better suited for different applications
DATA: Explain external; Example:
Animation external; Medium Video
Acoustic_Medium Video_Scene
Music
ACCESS_POINTS: VIDEO_SOURCE Animation.VIDEO_SOURCE; Image
Opus
AUDIO_SOURCE Animation.AUDIO_SOURCE; Score Image_Segment
TEXT_SOURCE Explain.TEXT_SOURCE; Audio_Block Pixel
Sample_Value Line
METHODS: Speech Pixel
...
start: display(Explain); Column
...
play: play(Animation); Opitcal_Medium Pixel
pause: pause(Animation); Video Animation
stop: stop(Animation); Video_Scene ...
... Text
...

Abstractions From Multimedia Hardware

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Abstractions From Multimedia Hardware

Uploaded by

Copyright:

Available Formats

Lehrstuhl für Informatik 4 Lehrstuhl für Informatik 4

Kommunikation und verteilte Systeme Kommunikation und verteilte Systeme