Assignment: 2 Date:03/12/2020

Android Multimedia Video

Task: Traversing trough the Android source code listing out the functionalities present in Media player
service, Stage fright engine and OMX in native multimedia framework.

Version: Pie - 9.0.0_r3

Source: https://androidxref.com/

Media Architecture

1) Application Framework

Basically application code utilize android.media APIs to interact with the successive layer through java
native interface.

Provides classes that manage various media interfaces in audio and video.

The Media APIs are used to play and, in some cases, record media files.
e.g., play a video streamed over the web or from local storage

android.media -> Binder IPC

2) Binder IPC

The Binder IPC proxies facilitate communication over process boundaries

Location : frameworks/av/media/libmedia

APIs Functionality
MediaPlayer class can be used to control playback of audio/video files and
streams. e.g., on how to use the methods in this class can be
found in VideoView

1. setListener() Sets media player listener

2. attachNewPlayer() Attach new player called in the current state of media player

3. setDataSource() Set data source with parameters like httpservice , url and headers

4. invoke() Accept the request if current state exists and reply

5. prepareAsync_l() prepareAsync provides the caller with 2 error codes,

one defined in the Android framework and one provided by the
implementation that generated the error
6. start() Checks if mediaplayer started and display current state
7. stop() Checks if media player stopped and display current state
8. pause() Checks if media player paused in the current state
9. isPlaying() Checks if media player started or paused in the current state

MediaCodecInfo.VideoCapabilities class that supports querying the video capabilities of a codec.

 Video performance points are a set of standard performance
points defined by number of pixels, pixel rate and frame rate. 

MediaExtractor facilitates extraction of demuxed, typically encoded, media data

from a data source.

1. getCachedDuration() Returns an estimate of how much data is presently cached in

memory expressed in microseconds
2. getCasInfo(int index)
Retrieves the information about the conditional access system
used to scramble a track.

3. getDrmInitData()
Extract DRM initialization data if it exists

4. getMetrics()
Return Metrics data about the current media container.

5. getSampleCryptoInfo
(MediaCodec.CryptoInfo info) If the sample flags indicate that the current sample is at least
partially encrypted, this call returns relevant information about
the structure of the sample data required for decryption.

6. getSampleTrackIndex()
Returns the track index the current sample originates from (or -1 if
no more samples are available)

7. getTrackCount()
Count the number of tracks found in the data source.

MediaCodec.BufferInfo Per buffer metadata includes an offset and size specifying the
range of valid data in the associated codec (output) buffer.

MediaCodec.Callback MediaCodec callback interface. Used to notify the user

asynchronously of various MediaCodec events.

Media Recorder class is to record audio/video streams form device and setting
various parameters

1. setCamera Checks media recorder state (initialized) and set camera for recording.
Eg: mMediaRecorder->setCamera(camera, proxy)

2. setPreviewSurface try to set preview surface with setting the video source.

3. setVideoSource set video source i.e (mMediaRecorder->setVideoSource(vs).

4. setAudioSource initialize media recorder and sets audio source.

 5. setOutputFormat output format for audio recording is set.

6. setVideoEncoder video encoding format to the source is assigned here.

Location : /frameworks/av/media/libmedia/include/media/mediarecorder

3) Native Multimedia Framework

At the native level, Android provides a multimedia framework that utilizes the Stagefright engine for
audio and video recording and playback. Stage fright comes with a default list of supported software
codecs.

Libmedia utilizes the Media Player Services

I) MediaPlayerService

class is defined in MediaPlayerService.h and definitions are implemented in MediaPlayerService.cpp.

PATH: /frameworks/av/media/libmediaplayerservice/MediaPlayerService

 MediaPlayerService

This class is primary API for providing services to video and sound. It creates player, MediaRecorder ,
remove Media recorder Client, creates Meta data retriever, display codec list.
 AudioOutput provide audio services to Client based on device callback, set audio stream type,
playback rate, audio track, audio attributes, sample rates and volume.
 CallbackData is passed to AudioTrack as user data. tryBeginTrackSwitch/endTrackSwitch are
used when the CallbackData is handed over to next sink.
 ImediaPlayerService is an interface. It creates/remove Media Recorder Client and prepare codec
list.
 BatteryUsageInfo API for battery app to pull data of codecs usage and collect information of
codec usage from media player and media recorder.
 BatteryAudioFlingerUsageInfo count the number of audio streams played, display total time of
audio output device usage and check whether device is currently in use or not.
 Client : This API create player and set video surface texture, buffering settings , sync settings ,
set volume, set audio stream type, set metadata, get current position and duration, reset and
notify, set next player. It also provide services to client like start, stop, pause, isplaying
 AudioDeviceUpdatedNotifier sends notification to client from class Listener.

MediaPlayerService API table:

 API
bool unmarshallFilter(const Parcel& p,
1 Metadata::Filter
*filter,
status_t *status)
bool findMetadata(const
2 Metadata::Filter& filter, const int32_t
val)
virtual int64_t
3 getPlayedOutDurationUs(int64_t nowUs)
const;
virtual ssize_t frameCount()
4 const;
virtual audio_session_t getSessionId()
5 const;
void
6 deleteRecycledTrack_l();
virtual status_t open(uint32_t
7 sampleRate, int
channelCount,audio_channel_mask_t
channelMask,audio_format_t format, int
bufferCount,
AudioCallback cb, void *cookie,
audio_output_flags_t flags =
AUDIO_OUTPUT_FLAG_NONE,
const audio_offload_info_t *offloadInfo
= NULL,bool doNotReconnect =
false,uint32_t suggestedFrameCount =
0);
void switchToNextOutput();
8 This provides to switch
Functionality
It unmarshell a filter
from a parcel. Param p
parcel that should start
with a filter. Returns true
if the parcel starts with
valid filter.
It will find the metadata
type of filter. Returns
true is a match found.
Calculate and returns
the duration of played
samples if played at
normal rate.
Gives the frame count
from the track.
Returns the Sessionid in
AudioOutput class.
It is called to delete
recycled track An
offloaded track isn’t
flushed because the
STREAM_END is
reported slightly to allow
time for the gapless
track. If we decided not
to recycle the track there
could be small amount
of residual data still
playing. We leave
AudioFlinger to drain the
track.
It is called to initialize
AudioTrack implies a
legacy stream type was
generated from audio
attributes.
 next output track for
playback.
static void
9 CallbackWrapper(int event, void *me, Set the callback cookies
void *info);
for the events of audio
trac.
virtual void
10. addBatteryData(uint32_t params); Collects and add the
information of the codec
usage from media player
and media recorder.
virtual status_t
11 pullBatteryData(Parcel* reply); API pulls the data of
codec usage of the
battery app.
virtual sp<IMediaRecorder>
12 createMediaRecorder(const String16 Creates a new media
&opPackageName);
recorder depending on
the calling process.
virtual sp<IMediaPlayer>
13 create(const sp<IMediaPlayerClient>& It is IMediaPlayerService
client,
interface that creates a
audio_session_t audioSessionId); new client from calling
pid.
virtual void disconnect();
14 It disconnects the client
and reset the player.
virtual void serviceDied(uint64_t
15 cookie,const It prompts to client if the
wp<::android::hidl::base::V1_0::IBase>&
service is dead using
who);
listener class
virtual status_t
16 setMetadataFilter(const Parcel& It will set the meta data
filter);
filters for client.
virtual media::VolumeShaper::Status
applyVolumeShaper(const Applies the
sp<media::VolumeShaper::Configuration>&
volumeShaper
configuration,const
sp<media::VolumeShaper::Operation>& configurations and
operation) override; investigate internal
implementations
depends on hardware
output.

 MediaRecorderClient

This API checks camera permissions to set video source and set audio and camera sources.
Provides the check service for camera if we don’t know it exists. Set the size and format rate of
video.
 ServiceDeathNotifier class notifies the client about binderDied and serviceDied.
  AudioDeviceUpdateNotifier sends the notification to client about audio io handle and audio port
handle from class Listener.

 MetadataRetrieverClient

This API implements ImediaMetadataRetriever interface and extract Meta data. It also create Retriever
by setting data source parameters like offset and length from Media player factory.

 getFrameAtTime capture a Video frame at a particular time in color format.

 getImageAtIndex extract image at specified index in color format .

Video Formats: H.263, H.264 AVC(BP,MP), H.265 HEVC ,MPEG-4 SP,VP8, VP9, AV1

File Formats: 3GPP (.3gp), MPEG-4 (.mp4), Matroska (.mkv) ,MPEG-TS (.ts), WebM (.webm)

For video Encoding, Decoding, Playback & Streaming speed and resolution are recommended based on
the certain parameters.

e.g.: (SD- Low/High) quality Format (H.264) -> (176 x 144 px @ 12 fps ) (480 x 360 px @ 30 fps) are
recommended respectively.

>> Similarly, for images also Android framework recommends some formats.

II) Stage fright engine

Stage fright is a native media playback tool used by Android and all these weaknesses reside in it.

Stage fright has built-in software base codec.

 Extractor: extract data into required format and set communication between native media
server to kernel driver.
 Player: captures video and provide sources like playing, starting, etc.
 Codex: prepare playback and setup decoder on request from media service to kernel driver.

what is a codec?

a) A codec is hardware device or a computer program which process input data into output data.
b) A codec encodes and decodes data stream for transmission/storage.

e.g.: H.264, MPEG.

 Android includes Stagefright, a media playback engine at the native level that has built-in
software-based codecs for popular media formats.
 Stagefright uses OpenMax (Open Media Accelerator) codecs.
 Open MAX codes are designed as per DRM standards.
FLOW:

Note: Stagefright also supports integration of custom hardware codec's provided by vendors (for that we need
to register their coded to Android Framework)

Location: /frameworks/av/media/libstagefright/

StagefrightPluginLoader.cpp

class StagefrightPluginLoader

 createCodec() -> creates codec dlsym(mLibHandle, "CreateCodec");

 createBuilder() -> creates builder dlsym(mLibHandle, "CreateBuilder");
 createInputSurface() -> creates input surface.
 GetCCodecInstance() -> gets StagefrightPluginLoader("libstagefright_ccodec.so")
instance.
StagefrightMetadataRetriever.cpp

struct StagefrightMetadataRetriever

 setDataSource() -> gets data source form URI "CreateFromURI()"

 getImageAtIndex() -> gets ImageAtIndex, colorFormat, metaOnly, thumbnail.
 getImageInternal() -> gets TrackMetaData and starts decoding frame by frame.
 extractMetadata() -> gets metada form index & keycode.
 parseMetaData() -> parses metadata (track , artis, gener, album...) & add it to frame.
 clearMetadata() ->clear's metedata.

StagefrightMediaScanner.cpp

struct StagefrightMediaScanner
 FileHasAcceptableExtension --> ".mp3", ".mp4", ".m4a", ".3gp", ".3gpp", ".3g2",
".3gpp2",".mpeg", ".ogg", ".mid", ".smf", ".imy", ".wma", ".aac",".wav", ".amr", ".midi", ".xmf", ".rtttl",
".rtx", ".ota",".mkv", ".mka", ".webm", ".ts", ".fl", ".flac", ".mxmf",".avi", ".mpeg", ".mpg", ".awb",
".mpga", ".mov",".m4v", ".oga" (supported formats).

processFile()

processFileInternal() --> tracknumber,height,width,date,isdrm,compilation,

writer,duration,albumartist,album ...

extractAlbumArt()

PATH: /frameworks/av/media/libstagefright/omx

MPEG4Writer.cpp
MPEG4Writer path: /frameworks/av/media/libstagefright/MPEG4Writer.cpp
MPEG4Writer is an encapsulation class under the Android stagefright media framework. We usually use the
MediaRecorder interface class called video recording to implement the underlying packaging of video recording
through MPEG4Writer.
The last link of video recording of crecording a video is MPEG4Writer.
MP4, 3gp, ismv and other common media packaging formats are derived from this basic file format ( ISO 14496-12
standard).
MPEG4Writer packaged video:
There are three main steps in the Android system video packaging process:

1) At the beginning of recording, write to the head of the file.

2) While recording is in progress, the data block of the audio and video track is written in real time.

3) At the end of the recording, write the index information and update the header parameters.

Exchangeable image file format (officially Exif, according to JEIDA/JEITA/CIPA specifications) is a standard that
specifies the formats for images, sound, and ancillary tags used by digital cameras (including smartphones),
scanners and other systems handling image and sound files recorded by digital cameras.

class MPEG4Writer: public MediaWriter <--- child class of Media writer in public mode inheritance.

Function Internal Description

function calls
virtual status_t getFourCCForMime( Add data source to the track.
addSource(const mime); High Efficiency Image File Format (HEIC)
sp<MediaSource> isHeic() Note: Limitations -No more than one video and/or
&source); one audio source can be added, but multiple
metadata sources can be added

status_t MPEG4Writer: Check mMaxFileSizeLimitBytes at the beginning

start(MetaData *param) since it may be implicitly changed later for 32-bit
file offset even if user does not ask to set it
 explicitly.
mStreamableFile --> to set file as stremeable
mWriteBoxToMemory --> to check reserved space
writeFtypBox(param (cache) (write to file).
); writeFourcc("3gp4");writeFourcc("heic");
startWriterThread(); After checking and setting all these params writing
thread is started

virtual status_t stop (); reset(); sets stopSource = true

virtual status_t pause(); sets mpause=True

virtual bool reachedEOS(); retrurns the status of mReachedEOS

virtual status_t dump(int write(); dumps the buffers into file.

fd, const
Vector<String16>& args);

void beginBox(const char It sets mInMemoryCacheOffset: mOffset and sets

*fourcc) box type based on either integer or string.
void beginBox(uint32_t id);

void writeInt8(int8_t x), htons of val and writes.

void writeInt16(int16_t htons:(6-bit number in host byte order and returns
x)void writeInt32(int32_t a 16-bit number in network byte order used in
x), void writeInt64(int64_t TCP/IP networks)
x);

void endBox(); at each box end (4 bytes ) size of the block is

written.

uint32_t return mInterleaveDurationUs

interleaveDuration() const

status_t sets mInterleaveDurationUs as 1000000

setInterleaveDuration(uint
32_t duration);

int32_t getTimeScale() returns mTimeScale

const mTimeScaleis set different values at different
places .

status_t setGeoData(int mLatitudex10000 = latitudex1000

latitudex10000, int mLongitudex10000 = longitudex10000
longitudex10000); mAreGeoTagsAvailable = true, mMoovExtraSize +=
30

status_t sets capture rate using kMetaKey_CaptureFps

setCaptureRate(float
captureFps);
status_t sets layer count using
setTemporalLayerCount(ui kMetaKey_TemporalLayerCount
nt32_t layerCount);

void  sizenotify notifies while approaching max file size

notifyApproachingLimit();
(MEDIA_RECORD
ER_EVENT_INFO,
MEDIA_RECORDE
R_INFO_MAX_FIL
ESIZE_APPROACH
ING, 0);

virtual void sets mStartTimeOffsetMs with ms

setStartTimeOffsetMs(int
ms)

virtual int32_t returns mStartTimeOffsetMs

getStartTimeOffsetMs()

virtual status_t mReflector = new

setNextFd(int fd); AHandlerReflection<MPEG4Writer>(this);
mLooper = new ALooper();
mLooper->registerHandle(mReflector);
mLooper->start();
 4) OMX

OpenMAX (Open Media Acceleration), often shortened as "OMX", is a non-proprietary and royalty-free
cross-platform set of C-language programming interfaces. It provides abstractions for routines that are
especially useful for processing of audio, video, and still images. It is intended for low power and
embedded system devices that need to efficiently process large amounts of multimedia data in
predictable ways, such as video codecs, graphics libraries, and other functions for video, image, audio,
voice and speech.

OpenMAX provides three layers of interfaces: application layer (AL), integration layer (IL) and
development layer (DL).

Layers

1) OpenMAX AL is the interface between multimedia applications, such as a media player, and the
platform media framework. It allows companies that develop applications to easily migrate
their applications to different platforms (customers) that support the OpenMAX AL application
programming interface (API)

2) OpenMAX IL is the interface between media framework, (such as StageFright or MediaCodec

API on Android, DirectShow on Windows, FFmpeg or Libav on Linux, or GStreamer for
 cross-platform), and a set of multimedia components (such as an audio or video codecs). It
allows companies that build platforms (e.g. allowing an implementation of an MP3 player) to
easily change components like MP3 decoders and Equalizer effects and buy components for their
platform from different vendors.

3) OpenMAX DL is the interface between physical hardware, such as digital signal processor (DSP)
chips, CPUs, GPUs, and software, like video codecs and 3D engines. It allows companies to
easily integrate new hardware that supports OpenMAX DL without reoptimizing their low level
software.

Path-framerwork/av/media/libmedia/IOMX.cpp
Class-BpOMX
Path-framerwork/av/media/libstagefright/omx/omxgraphicbuffersource.cpp
Class-OmxComponentWrapper

Path-framerwork/av/media/libstagefright/omx/OMXmaster.cpp
OMXmaster.cpp Used for:-
1) addPlugin

2)clearPlugins
3)makeComponentInstance
4)destroyComponentInstance
5)enumerateComponents
6)getRolesOfComponent

 OMX_core

The OMX_Core header file contains the definitions used by both the application and the
component to access common items.
OMX_core methods

OMX_init The OMX_Init method is used to initialize the OMX core. It shall be the
first call made into OMX and it should only be executed one time without
an interviening OMX_Deinit call.

OMX_deinit The OMX_Deinit method is used to deinitialize the OMX core. It shall be
the last call made into OMX.

OMX_ComponentNameEnum The OMX_ComponentNameEnum method will enumerate through all the

names of recognised valid components in the system. This function is
provided as a means to detect all the components in the system run-time.
OMX_GetHandle he OMX_GetHandle method will locate the component specified by the
component name given, load that component into memory and then
invoke the component's methods to create an instance of the component.

OMX_FreeHandle The OMX_FreeHandle method will free a handle allocated by the

OMX_GetHandle method. If the component reference count goes to zero,
the component will be unloaded from memory.

OMX_SetupTunnel he OMX_SetupTunnel method will handle the necessary calls to the

components to setup the specified tunnel the two components.
structures/enumerations in OMX_core.h
1) The OMX_COMMANDTYPE enumeration is used to specify the action in
the OMX_SendCommand macro.

define OMX_SendCommand( \
hComponent, \
Cmd, \
nParam, \
pCmdData) \
((OMX_COMPONENTTYPE*)(hComponent))->SendCommand( \
hComponent, \
Cmd, \
nParam, \
pCmdData) /* Macro End */

 Hcomponent -handle of component to execute the command

 Cmd-Command for the component to execute
 nParam-Parameter for the command to be executed. When Cmd has the value
OMX_CommandStateSet, value is a member of OMX_STATETYPE. When Cmd has
the value OMX_CommandFlush, value of nParam indicates which port(s) to flush. -1 is used to
flush all ports a single port index will only flush that port.When Cmd has the value
"OMX_CommandPortDisable"or "OMX_CommandPortEnable", the component's port is given by
the value of nParam. When Cmd has the value "OMX_CommandMarkBuffer" the components
pot is given by the value of nParam.
  pCmdData-Parameter pointing to the OMX_MARKTYPE structure when Cmd has the
value "OMX_CommandMarkBuffer".

2) The OMX_STATETYPE enumeration is used to indicate or change the component state. This
enumeration reflects the current state of the component when used with the
OMX_GetState macro or becomes the parameter in a state change command when used
with the OMX_SendCommand macro.

OMXNodeInstance Used for:-

Path-framerwork/av/media/libstagefright/omx/OMXNodeInstance.cpp
1.setHandle
2. setBufferSource
3.freeNode
4.sendCommand
5.getParameter
6.SetParameter
7.setPortMode
8.enableNativeBuffers
9.getGraphicBufferUsage

 OmxNodeInstance
Path:
/frameworks/av/media/libstagefright/omx/include/media/stagefright/omx/OMXNodeInstance.h

Methods:
 CallbackDispatcherThread :- underlying thread used by the CallbackDispatcher.CallbackDispatcher:
1. Post() :- Posts |msg| to listener’s queue.
2. loop()
3. Dispatch()
API name Functionality
OmxNodeInstance() Constructor:- Initializes various class attributes
setHandle() Sets the mHandle variable.
setBufferSource() Sets mOmxBufferSource variable.
sendCommand() Changes the state of bufferSource to idle or loaded based on cmd and
param.
setParameter() Sets portIndex. Calls enableNativeBuffers_l and
storeMetaDataInBuffers_l
getGraphicBufferUsage Calls the OMX_GetExtensionIndex macro, it will invoke a
component to translate a vendor specific configuration or
parameter string into an OMX structure index and the
OMX_GetParameter macro will get one of the current
parameter settings from the component. This macro
cannot only be invoked when the component is in the
OMX_StateInvalid state.

prepareForAdaptivePlayback():- Uses OMX_SetParameter macro to set up

portIndex, maxFramWidth, MaxFrameHeight, etc.

configureVideoTunnelMode() Uses OMX_SetParameter macro to set up tunneling params.

SetInputSurface()
allocateSecureBuffer()
onMessages()
handleMessage(omx_message &msg)
status_t
updateGraphicBufferInMeta_l()
codecBufferFilled(omx_message
&msg)
Its used to set the buffer Source by calling
setBufferSource(bufferSource);
Add buffer from buffer source.
handles messages and removes them from the list
Handles |msg|, and may modify it. Returns true if completely
handled it and |msg| does not need to be sent to the event
listener.
Updates the graphic buffer handle in the metadata buffer for |buffer|
and |header| to |graphicBuffer|'s handle. If |updateCodecBuffer| is
true, the update will happen in the actual codec buffer.
- Called when omx_message::FILL_BUFFER_DONE is
received.

freeNode() Transition the node from its current state all the way down to
"Loaded".

OMXutils:
Path-framerwork/av/media/libstagefright/omx/OMXutils.cpp
Used for:-
1.DescribeColorFormatParams
2.DescribeDefaultColorFormat
3.GetComponentRole