Anatomy of a Mobile Audio Streaming Application

Adam Fleming
CTO
Apadmi
Apadmi Background

Small Team of Mobile Experts

Developing Mobile Apps for over 10 Years

Developing Platform Technology for Symbian

Mobile Consultancy & Development Service

Technical Workshops & Training

Feasibility Studies, Prototyping

Requirements Analysis

Application Development, Test

See www.apadmi.com
NPR Player Implementation

Implemented as Native C++ Application

Background playback support required

Low-level access into streaming APIs

Performance critical

Integration with other parts of system

Notably telephony
A Streaming Media Application

Content UI
Browsing

Stream Metadata
Control
Playback Status

Audio
Streaming
Network API Streaming
Engine
API
Network Audio
Stream Data
Content Provider

NPR – National Public Radio

(www.npr.org)

Radio Program Syndication

Mostly spoken word (News, Talk etc)

Also Print Format News

Mix of local and wider-interest items

Already have a API providing text,
graphical and audio content
(www.npr.org/api/index)
Streaming Service Interfacing

NPR supports a simple RESTful HTTP-

based interface

Static content is provided as XML

parsed into HTML for display in embedded
browser

Streams either live or fixed-length

MP3 data stream

Will look at the network classes a little
later
Data Bearers

Cellular Data/WiFi

Speed

Wide range of actually achievable speeds

Differing degrade behaviour

Access

Operator networks, Private networks, Firewalls

Access Point Behaviour

Connection persistence

Authentication

Cost
 System Provided API - Network

Socket RSocketServ
Server

Cellular Shortlink RConnection RHostResolver

WiFi Data Data

RHttpSession RSocket

http://developer.symbian.org/wiki/index.php/Sockets_(Fundamentals_of_Symbian_C++)
http://developer.symbian.org/wiki/index.php/Symbian_OS_Communications_Programming/11._HTTP
Network – Connecting to an IAP
RSocketServ iSocketServ;
RConnection iConnection;
TUint32 iIap;
…

// Set up the RConnection

User::LeaveIfError(iSocketServ.Connect());
User::LeaveIfError(iConnection.Open(iSocketServ));
…

// Set up the Iap

TCommDbConnPref connPref;
connPref.SetIapId(iIap);
connPref.SetDialogPreference(ECommDbDialogPrefDoNotPrompt);
connPref.SetDirection(ECommDbConnectionDirectionOutgoing);
…

// Try to connect
iConnection.Start( connPref, iStatus );
Network – HTTP Connection

RHttpSession RHttpTransaction

MHTTPSessionEventCallback MHTTPTransactionCallback

Refer to more detailed explanations on

Wiki
http://developer.symbian.org/wiki/index.php/Symbian_OS_Communications_Programming/11._HTTP
System Provided API - Network

Open RHttpSession

Request a transaction from the session

Set the properties, filters, headers and
body of the request

Submit the request

Receive callback through
MHTTPTransactionCallback::MHFRunL()

THTTPEvent contains state of transaction
A Streaming Media Application

Content UI
Browsing

Stream Metadata
Control
Playback Status

Audio
Streaming
Network API Streaming
Engine
API
Network Audio
Stream Data
Playback Mechanism Considerations

User-interface requirements

Media Playback

“trick modes”

Background Playback

Degradation Behaviour

Requirements for platform integration

Incoming/Outgoing calls
Playback Control API
class CNPRPlaybackControl
{
…
void OpenUrlL(CNPRPlaylist* aPlaylist,
TBool aFixedLengthStream);
void SetPlayList(const CNPRPlaylist* aPlaylist);
void CloseUrl();
void PlayL();
void PauseL();
void StopL();
void TogglePlayPause();
virtual TInt SupportMetadata() const;
virtual void SetVolume(TInt aVolume);
virtual TInt Volume();
…
}

class CNPRPlayList
{
…
const TPtrC8 URL(TInt aIndex) const;
…
}
A Streaming Media Application

Content UI
Browsing

Stream Metadata
Control
Playback Status

Audio
Streaming
Network API Streaming
Engine
API
Network Audio
Stream Data
Content Browsing API

Provided by NPR

HTTP Requests which return fixed-form
XML

Responses parsed and used to generate
HTML

Displayed in embedded browser control

Includes an API to allow user to specify
current location to retrieve local stories,
stations etc
Location - Classes
class Location Classes

RPositionServ er
CActiv e

Position Serv er

RPositioner CPositionGetter

TPosition

http://developer.symbian.org/wiki/index.php/Creating_Location-
Aware_Applications#Positioning_methods_supported_by_the_S60_platform
Location - Setup
#include <lbs.h>
#include <lbspositioninfo.h>

RPositionServer iLocationServer;
RPositioner iPositioner;
TPositionInfo iPositionInfo;

{
// open the server sessions
User::LeaveIfError(iLocationServer.Connect());

// open positioner using default module

User::LeaveIfError(iPositioner.Open(iLocationServer));

// set our application as location requestor

User::LeaveIfError(iPositioner.SetRequestor(CRequestor::ERequestorService,
CRequestor::EFormatApplication,
*iAppName));
}
Location - Use
void CGpsPositionRetriever::GetPosition()
{
…
// request position updates
iPositioner.NotifyPositionUpdate(iPositionInfo, iStatus);
SetActive();
…
}

void CGpsPositionRetriever::RunL()
{
…
TInt error = iStatus.Int();
if (error == KErrNone)
{
TPosition pos;
iPositionInfo.GetPosition(pos);
iObserver.HandleGPSRetrievalL(pos);
}
}
A Streaming Media Application

Content UI
Browsing

Stream Metadata
Control
Playback Status

Audio
Streaming
Network API Streaming
Engine
API
Network Audio
Stream Data
Audio Streaming

The most complex part of the project

Timeliness is critical

Need to smooth out changes in network
speed

Must be able to run in background whilst
user doing something else

Must not interfere with UI responsiveness
System Provided API – Audio
Streaming

MMF Application

MDF CMdaAudio*Stream

Hardware
Device Drivers Codec
Implementation
Audio Streaming - Classes

CMdaAudioOutputStream

MMdaAudioOutputStreamCallback

TMdaAudioDataSettings

Also

CMdaAudioInputStream

MMdaAudioInputStreamCallback
CMdaAudioOutputStream
class CMdaAudioOutputStream : public CBase,
public MMMFClientUtility
{
public:
IMPORT_C static CMdaAudioOutputStream* NewL(MMdaAudioOutputStreamCallback& aCallBack,

TInt aPriority,

TMdaPriorityPreference aPref = EMdaPriorityPreferenceTimeAndQuality);

virtual void SetAudioPropertiesL(TInt aSampleRate, TInt aChannels);
virtual void Open(TMdaPackage* aSettings);
virtual TInt MaxVolume();
virtual TInt Volume();
virtual void SetVolume(const TInt aNewVolume);
virtual void WriteL(const TDesC8& aData);
virtual void Stop();
virtual const TTimeIntervalMicroSeconds& Position();
IMPORT_C void SetBalanceL(TInt aBalance = KMMFBalanceCenter);
IMPORT_C TInt GetBalanceL() const;
IMPORT_C TInt GetBytes();
}
MMdaAudioOutputStreamCallback
class MMdaAudioOutputStreamCallback
{
public:
virtual void MaoscOpenComplete(TInt aError) = 0;
virtual void MaoscBufferCopied(TInt aError, const TDesC8& aBuffer) = 0;
virtual void MaoscPlayComplete(TInt aError) = 0;
}
System Provided API – Audio
Streaming
sd Class M odel

Pl ayer CMdaAudi oOutputStream

MMF

Open()

Set Up Stream()

MaoscOpenCompl ete()

Wri te()

Wri teData()

MaoscBufferCopi ed()

Wri te()

Wri teData()

MaoscBufferCopi ed()

Stop()

Stop Pl ayback()

Done()

MaoscPl aybackCompl ete()

A Streaming Media Application

Content UI
Browsing

Stream Metadata
Control
Playback Status

Audio
Streaming
Network API Streaming
Engine
API
Network Audio
Stream Data
Audio Engine – Architecture

Main decision – single threaded, or multi-

threaded

Consider implementations of both

Highlight strengths and weaknesses
Important Point

The choice between single or multiple

threads is not a choice between active
objects or threads

Active objects are fundamental to the way
in which clients and servers communicate

Callbacks are just wrapped up active
objects

If a thread uses any API which uses
callbacks, it will need an active scheduler
Single Threaded Architecture

“Simple”

Lowest use of system resources

Prone to delays caused by system loading

Possible for UI to interfere with audio
playback
Architectural Description

Conceptually very simple, everything is

built into the same application and
application logic simply responds to
requests and notifications

Single thread semaphore used by all
servers which app has a session with –
Windows, Socket, Location, etc

Need to ensure that all event handling
code completes quickly

Need to consider priority of events
Downfalls of Single-threading

Unbounded time from event completion to RunL

Long running event handlers

Some of these are out of your control

Displaying softkey menu for example

Very frequent events

UI events when a key is held down

Concrete example

Scrolling through a news story

Either scrolling is not smooth or playback stutters
Multi-Threaded Architecture

Dual-threaded

More complex

Heavier loading on system

Less susceptible to delays due to system
loading

UI actions should not impact playback
performance

Requires mechanism for inter-thread
communication
Architectural Description

UI Thread looks after the display and

interactions with the user

Audio Streaming Thread responsible for
reading data from the network, and
sending it on to the Audio Streaming APIs

Need a way to allow the UI thread to
control the Audio Streaming Thread

Need a way to allow the Audio Streaming
Thread to communicate information back
to the UI
Message Queues

Lightweight way to allow small amounts of

data to be passed safely between threads

Simpler to use than setting up shared
memory, mutexes, signalling semaphores
etc

Fire-and-Forget style messaging, ideal for
event notifications

Can carry small data types of fixed length
http://developer.symbian.org/wiki/index.php/Threads,_Processes,_and_IPC_(Fundament
als_of_Symbian_C++)#Message_Queues
Message Queues
// Creation :
RMsgQueue<TMsgType> iStreamMsgQueue;
User::LeaveIfError(iStreamMsgQueue->CreateLocal(5));

// Request Notification
iStreamMsgQueue->NotifyDataAvailable(iStatus);
SetActive();

// retreive data and do something with it.

TMsgType msg;
iStreamMsgQueue->Receive(msg);
Architecture Continued

Threads communicate by passing

messages through the message queue

Both threads in same process, so can pass
local pointers as well as TClass instances

Clearer design

Each thread contains a single state-machine
Dual Thread Example
sd Class Model

UI Thread Playback Thread

User

Browse Stations()

Download Station Info()

Display Stations()

Select Station()
Select Station()

Startup()
Notify Startup()

Download Packet()
Browse News Stories()

Play Packet()

Download Story()

Read Story()

View Story()

Download Packet()
Advantages

Playback thread is now allocated time-

slices independent of the UI

App is still susceptible to system loading,
but not to the same degree

Threads are handled independently

UI and playback priorities explicitly set
Gotchas…

Need to know which thread you’re in

Server handles are thread-local

Callbacks are active-objects too

Still need an AS in your thread

Passing data between threads needs
protection

Not an issue here, but needs to be
remembered
A final word of warning…

Streaming Audio API is well established

and understood

But it doesn’t always behave in the way that
you would expect

For MP3, MaoscPlaybackComplete does not get
called with KErrUnderflow at the end of
playback

But it does for all other media types

Probably a defect in MP3 codec implementation
A Streaming Media Application

Content UI
Browsing

Stream Metadata
Control
Playback Status

Audio
Streaming
Network API Streaming
Engine
API
Network Audio
Stream Data
Questions?