Anatomy of a Mobile Audio Streaming Application

Adam Fleming CTO Apadmi

Apadmi Background
Small Team of Mobile Experts Developing Mobile Apps for over 10 Years Developing Platform Technology for Symbian Mobile Consultancy & Development Service
Technical Workshops & Training Feasibility Studies, Prototyping Requirements Analysis Application Development, Test

See www.apadmi.com

NPR Player Implementation
Implemented as Native C++ Application
Background playback support required Low-level access into streaming APIs Performance critical Integration with other parts of system
Notably telephony

A Streaming Media Application

Content Browsing

UI

Control

Stream Metadata Playback Status

Network API Network Stream

Streaming Engine Audio Data

Audio Streaming API

Content Provider
NPR – National Public Radio
(www.npr.org)

Radio Program Syndication
Mostly spoken word (News, Talk etc)

Also Print Format News Mix of local and wider-interest items Already have a API providing text, graphical and audio content (www.npr.org/api/index)

Streaming Service Interfacing
NPR supports a simple RESTful HTTPbased interface Static content is provided as XML
parsed into HTML for display in embedded browser

Streams either live or fixed-length
MP3 data stream

Will look at the network classes a little later

Data Bearers
Cellular Data/WiFi
Speed
Wide range of actually achievable speeds Differing degrade behaviour

Access
Operator networks, Private networks, Firewalls

Access Point Behaviour
Connection persistence Authentication

Cost

System Provided API - Network
Socket Server
RSocketServ

WiFi

Cellular Data

Shortlink Data

RConnection

RHostResolver

RHttpSession

RSocket

http://developer.symbian.org/wiki/index.php/Sockets_(Fundamentals_of_Symbian_C++) http://developer.symbian.org/wiki/index.php/Symbian_OS_Communications_Programming/11._HTTP

Network – Connecting to an IAP
RSocketServ iSocketServ; RConnection iConnection; TUint32 iIap; … // Set up the RConnection User::LeaveIfError(iSocketServ.Connect()); User::LeaveIfError(iConnection.Open(iSocketServ)); … // Set up the Iap TCommDbConnPref connPref; connPref.SetIapId(iIap); connPref.SetDialogPreference(ECommDbDialogPrefDoNotPrompt); connPref.SetDirection(ECommDbConnectionDirectionOutgoing); … // Try to connect iConnection.Start( connPref, iStatus );

Network – HTTP Connection

RHttpSession

RHttpTransaction

MHTTPSessionEventCallback

MHTTPTransactionCallback

Refer to more detailed explanations on Wiki
http://developer.symbian.org/wiki/index.php/Symbian_OS_Communications_Programming/11._HTTP

System Provided API - Network
Open RHttpSession Request a transaction from the session Set the properties, filters, headers and body of the request Submit the request Receive callback through MHTTPTransactionCallback::MHFRunL()
THTTPEvent contains state of transaction

A Streaming Media Application

Content Browsing

UI

Control

Stream Metadata Playback Status

Network API Network Stream

Streaming Engine Audio Data

Audio Streaming API

Playback Mechanism Considerations
User-interface requirements Media Playback
“trick modes” Background Playback Degradation Behaviour

Requirements for platform integration
Incoming/Outgoing calls

Playback Control API
class CNPRPlaybackControl { … void OpenUrlL(CNPRPlaylist* aPlaylist, TBool aFixedLengthStream); void SetPlayList(const CNPRPlaylist* aPlaylist); void CloseUrl(); void PlayL(); void PauseL(); void StopL(); void TogglePlayPause(); virtual TInt SupportMetadata() const; virtual void SetVolume(TInt aVolume); virtual TInt Volume(); … } class CNPRPlayList { … const TPtrC8 URL(TInt aIndex) const; … }

A Streaming Media Application

Content Browsing

UI

Control

Stream Metadata Playback Status

Network API Network Stream

Streaming Engine Audio Data

Audio Streaming API

Content Browsing API
Provided by NPR HTTP Requests which return fixed-form XML Responses parsed and used to generate HTML Displayed in embedded browser control Includes an API to allow user to specify current location to retrieve local stories, stations etc

Location - Classes
class Location Classes RPositionServ er CActiv e Position Serv er

RPositioner

CPositionGetter

TPosition

http://developer.symbian.org/wiki/index.php/Creating_LocationAware_Applications#Positioning_methods_supported_by_the_S60_platform

Location - Setup
#include <lbs.h> #include <lbspositioninfo.h> … RPositionServer RPositioner TPositionInfo … { // open the server sessions User::LeaveIfError(iLocationServer.Connect()); // open positioner using default module User::LeaveIfError(iPositioner.Open(iLocationServer)); // set our application as location requestor User::LeaveIfError(iPositioner.SetRequestor(CRequestor::ERequestorService, CRequestor::EFormatApplication, *iAppName)); } iLocationServer; iPositioner; iPositionInfo;

Location - Use
void CGpsPositionRetriever::GetPosition() { … // request position updates iPositioner.NotifyPositionUpdate(iPositionInfo, iStatus); SetActive(); … } void CGpsPositionRetriever::RunL() { … TInt error = iStatus.Int(); if (error == KErrNone) { TPosition pos; iPositionInfo.GetPosition(pos); iObserver.HandleGPSRetrievalL(pos); } }

A Streaming Media Application

Content Browsing

UI

Control

Stream Metadata Playback Status

Network API Network Stream

Streaming Engine Audio Data

Audio Streaming API

Audio Streaming
The most complex part of the project Timeliness is critical Need to smooth out changes in network speed Must be able to run in background whilst user doing something else Must not interfere with UI responsiveness

System Provided API – Audio Streaming

MMF

Application

MDF

CMdaAudio*Stream

Device Drivers

Hardware Codec Implementation

Audio Streaming - Classes
CMdaAudioOutputStream MMdaAudioOutputStreamCallback TMdaAudioDataSettings Also CMdaAudioInputStream MMdaAudioInputStreamCallback

CMdaAudioOutputStream
class CMdaAudioOutputStream : public CBase, public MMMFClientUtility { public: IMPORT_C static CMdaAudioOutputStream* NewL(MMdaAudioOutputStreamCallback& aCallBack, TInt aPriority, TMdaPriorityPreference aPref = EMdaPriorityPreferenceTimeAndQuality); virtual void SetAudioPropertiesL(TInt aSampleRate, TInt aChannels); virtual void Open(TMdaPackage* aSettings); virtual TInt MaxVolume(); virtual TInt Volume(); virtual void SetVolume(const TInt aNewVolume); virtual void WriteL(const TDesC8& aData); virtual void Stop(); virtual const TTimeIntervalMicroSeconds& Position(); IMPORT_C void SetBalanceL(TInt aBalance = KMMFBalanceCenter); IMPORT_C TInt GetBalanceL() const; IMPORT_C TInt GetBytes(); }

MMdaAudioOutputStreamCallback
class MMdaAudioOutputStreamCallback { public: virtual void MaoscOpenComplete(TInt aError) = 0; virtual void MaoscBufferCopied(TInt aError, const TDesC8& aBuffer) = 0; virtual void MaoscPlayComplete(TInt aError) = 0; }

System Provided API – Audio Streaming
sd Class M odel Pl ayer CMdaAudi oOutputStream MMF Open()

Set Up Stream()

MaoscOpenCompl ete()

Wri te()

Wri teData()

MaoscBufferCopi ed()

Wri te() Wri teData()

MaoscBufferCopi ed()

Stop()

Stop Pl ayback()

Done()

MaoscPl aybackCompl ete()

A Streaming Media Application

Content Browsing

UI

Control

Stream Metadata Playback Status

Network API Network Stream

Streaming Engine Audio Data

Audio Streaming API

Audio Engine – Architecture
Main decision – single threaded, or multithreaded Consider implementations of both Highlight strengths and weaknesses

Important Point
The choice between single or multiple threads is not a choice between active objects or threads Active objects are fundamental to the way in which clients and servers communicate Callbacks are just wrapped up active objects If a thread uses any API which uses callbacks, it will need an active scheduler

Single Threaded Architecture
“Simple” Lowest use of system resources Prone to delays caused by system loading Possible for UI to interfere with audio playback

Architectural Description
Conceptually very simple, everything is built into the same application and application logic simply responds to requests and notifications Single thread semaphore used by all servers which app has a session with – Windows, Socket, Location, etc Need to ensure that all event handling code completes quickly Need to consider priority of events

Downfalls of Single-threading
Unbounded time from event completion to RunL Long running event handlers
Some of these are out of your control Displaying softkey menu for example

Very frequent events
UI events when a key is held down

Concrete example
Scrolling through a news story
Either scrolling is not smooth or playback stutters

Multi-Threaded Architecture
Dual-threaded
More complex Heavier loading on system Less susceptible to delays due to system loading UI actions should not impact playback performance Requires mechanism for inter-thread communication

Architectural Description
UI Thread looks after the display and interactions with the user Audio Streaming Thread responsible for reading data from the network, and sending it on to the Audio Streaming APIs Need a way to allow the UI thread to control the Audio Streaming Thread Need a way to allow the Audio Streaming Thread to communicate information back to the UI

Message Queues
Lightweight way to allow small amounts of data to be passed safely between threads Simpler to use than setting up shared memory, mutexes, signalling semaphores etc Fire-and-Forget style messaging, ideal for event notifications Can carry small data types of fixed length
http://developer.symbian.org/wiki/index.php/Threads,_Processes,_and_IPC_(Fundament als_of_Symbian_C++)#Message_Queues

Message Queues
// Creation : RMsgQueue<TMsgType> iStreamMsgQueue; User::LeaveIfError(iStreamMsgQueue->CreateLocal(5)); // Request Notification iStreamMsgQueue->NotifyDataAvailable(iStatus); SetActive(); // retreive data and do something with it. TMsgType msg; iStreamMsgQueue->Receive(msg);

Architecture Continued
Threads communicate by passing messages through the message queue Both threads in same process, so can pass local pointers as well as TClass instances Clearer design
Each thread contains a single state-machine

Dual Thread Example
sd Class Model UI Thread User Playback Thread

Browse Stations()

Download Station Info()

Display Stations() Select Station() Select Station()

Startup() Notify Startup()

Download Packet() Browse News Stories() Play Packet()

Download Story() Read Story()

View Story()

Download Packet()

Advantages
Playback thread is now allocated timeslices independent of the UI App is still susceptible to system loading, but not to the same degree
Threads are handled independently UI and playback priorities explicitly set

Gotchas…
Need to know which thread you’re in
Server handles are thread-local

Callbacks are active-objects too
Still need an AS in your thread

Passing data between threads needs protection
Not an issue here, but needs to be remembered

A final word of warning…
Streaming Audio API is well established and understood
But it doesn’t always behave in the way that you would expect For MP3, MaoscPlaybackComplete does not get called with KErrUnderflow at the end of playback But it does for all other media types Probably a defect in MP3 codec implementation

A Streaming Media Application

Content Browsing

UI

Control

Stream Metadata Playback Status

Network API Network Stream

Streaming Engine Audio Data

Audio Streaming API

Questions?

Sign up to vote on this title
UsefulNot useful

Master Your Semester with Scribd & The New York Times

Special offer for students: Only $4.99/month.

Master Your Semester with a Special Offer from Scribd & The New York Times

Cancel anytime.