Assignment: 2 Date:30/11/2020

Android Multimedia Video

Task: Traversing trough the Android source code listing out the functionalities present in Media player
service, Stage fright engine and OMX in native multimedia framework.

Version: Pie - 9.0.0_r3

Source: https://androidxref.com/

Media Architecture

1) Application Framework

Basically application code utilize android.media APIs to interact with the successive layer through java
native interface.

Provides classes that manage various media interfaces in audio and video.
The Media APIs are used to play and, in some cases, record media files.

e.g., play a video streamed over the web or from local storage

android.media -> Binder IPC

2) Binder IPC

The Binder IPC proxies facilitate communication over process boundaries

Location : frameworks/av/media/libmedia

 MediaPlayer class can be used to control playback of audio/video files and streams.

e.g., on how to use the methods in this class can be found in VideoView

 MediaCodecInfo.VideoCapabilities class that supports querying the video capabilities of a

codec. Video performance points are a set of standard performance points defined by
number of pixels, pixel rate and frame rate. 

 MediaExtractor facilitates extraction of demuxed, typically encoded, media data from a

data source.

 MediaCodec.BufferInfo Per buffer metadata includes an offset and size specifying the
range of valid data in the associated codec (output) buffer.

 MediaCodec.Callback
MediaCodec callback interface. Used to notify the user asynchronously of various MediaCodec
events.
Location : /frameworks/av/media/libmedia/include/media/mediarecorder.h

 Media Recorder class is to record audio/video streams form device and setting
various parameters.

Location: /frameworks/av/media/libmedia/mediarecorder.cpp

 setCamera Checks media recorder state (initialized) and set camera for recording.
Eg: mMediaRecorder->setCamera(camera, proxy)
 setPreviewSurface try to set preview surface with setting the video source.
 setVideoSource set video source i.e (mMediaRecorder->setVideoSource(vs).
 setAudioSource initialize media recorder and sets audio source.
 setOutputFormat output format for audio recording is set.
 setVideoEncoder video encoding format to the source is assigned here.

3) Native Multimedia Framework

At the native level, Android provides a multimedia framework that utilizes the Stagefright engine for
audio and video recording and playback. Stage fright comes with a default list of supported software
codecs.

Libmedia utilizes the Media Player Services

I) MediaPlayerService

class is defined in MediaPlayerService.h and definitions are implemented in MediaPlayerService.cpp.

PATH: /frameworks/av/media/libmediaplayerservice/MediaPlayerService

 MediaPlayerService

This class is primary API for providing services to video and sound. It creates player, MediaRecorder ,
remove Media recorder Client, creates Meta data retriever, display codec list.

 AudioOutput provide audio services to Client based on device callback, set audio stream type,
playback rate, audio track, audio attributes, sample rates and volume.
 CallbackData is passed to AudioTrack as user data. tryBeginTrackSwitch/endTrackSwitch are used
when the CallbackData is handed over to next sink.
 ImediaPlayerService is an interface. It creates/remove Media Recorder Client and prepare codec
list.
 BatteryUsageInfo API for battery app to pull data of codecs usage and collect information of
codec usage from media player and media recorder.
 BatteryAudioFlingerUsageInfo count the number of audio streams played, display total time of
 audio output device usage and check whether device is currently in use or not.
 Client : This API create player and set video surface texture, buffering settings , sync settings , set
volume, set audio stream type, set metadata, get current position and duration, reset and notify,
set next player. It also provide services to client like start, stop, pause, isplaying
 AudioDeviceUpdatedNotifier sends notification to client from class Listener.

 MediaRecorderClient

This API checks camera permissions to set video source and set audio and camera sources.
Provides the check service for camera if we don’t know it exists. Set the size and format rate of
video.
 ServiceDeathNotifier class notifies the client about binderDied and serviceDied.
 AudioDeviceUpdateNotifier sends the notification to client about audio io handle and audio port
handle from class Listener.

 MetadataRetrieverClient

This API implements ImediaMetadataRetriever interface and extract Meta data. It also create Retriever
by setting data source parameters like offset and length from Media player factory.

 getFrameAtTime capture a Video frame at a particular time in color format.

 getImageAtIndex extract image at specified index in color format .

Video Formats: H.263, H.264 AVC(BP,MP), H.265 HEVC ,MPEG-4 SP,VP8, VP9, AV1

File Formats: 3GPP (.3gp), MPEG-4 (.mp4), Matroska (.mkv) ,MPEG-TS (.ts), WebM (.webm)

For video Encoding, Decoding, Playback & Streaming speed and resolution are recommended based on
the certain parameters.

e.g.: (SD- Low/High) quality Format (H.264) -> (176 x 144 px @ 12 fps ) (480 x 360 px @ 30 fps) are
recommended respectively.

>> Similarly, for images also Android framework recommends some formats.

II) Stage fright engine

Stage fright is a native media playback tool used by Android and all these weaknesses reside in it.

Stage fright has built-in software base codec.

 Extractor: extract data into required format and set communication between native media
server to kernel driver.
  Player: captures video and provide sources like playing, starting, etc.
 Codex: prepare playback and setup decoder on request from media service to kernel driver.

what is a codec?

a. A codec is hardware device or a computer program which process input data into output
data.
b. A codec encodes and decodes data stream for transmission/storage.

e.g.: H.264, MPEG.

 Android includes Stagefright, a media playback engine at the native level that has built-in
software-based codecs for popular media formats.
 Stagefright uses OpenMax (Open Media Accelerator) codecs.
 Open MAX codes are designed as per DRM standards.

FLOW:

Note: Stagefright also supports integration of custom hardware codec's provided by vendors (for that we
need to register their coded to Android Framework)

Location: /frameworks/av/media/libstagefright/

StagefrightPluginLoader.cpp

class StagefrightPluginLoader
 createCodec() -> creates codec dlsym(mLibHandle, "CreateCodec");

createBuilder() -> creates builder dlsym(mLibHandle, "CreateBuilder");

createInputSurface() -> creates input surface.

GetCCodecInstance() -> gets StagefrightPluginLoader("libstagefright_ccodec.so") instance.

StagefrightMetadataRetriever.cpp

struct StagefrightMetadataRetriever
setDataSource() -> gets data source form URI "CreateFromURI()"

getImageAtIndex() -> gets ImageAtIndex, colorFormat, metaOnly, thumbnail.

getImageInternal() -> gets TrackMetaData and starts decoding frame by frame.

extractMetadata() -> gets metada form index & keycode.

parseMetaData() -> parses metadata (track , artis, gener, album...) & add it to frame.

clearMetadata() ->clear's metedata.

StagefrightMediaScanner.cpp

struct StagefrightMediaScanner
FileHasAcceptableExtension --> ".mp3", ".mp4", ".m4a", ".3gp", ".3gpp", ".3g2",
".3gpp2",".mpeg", ".ogg", ".mid", ".smf", ".imy", ".wma", ".aac",".wav", ".amr", ".midi", ".xmf", ".rtttl",
".rtx", ".ota",".mkv", ".mka", ".webm", ".ts", ".fl", ".flac", ".mxmf",".avi", ".mpeg", ".mpg", ".awb",
".mpga", ".mov",".m4v", ".oga" (supported formats).

processFile()

processFileInternal() --> tracknumber,height,width,date,isdrm,compilation,

writer,duration,albumartist,album ...

extractAlbumArt()

PATH: /frameworks/av/media/libstagefright/omx

1) OMX INTEGRATION

The OpenMAX IL provides a standardized way for Stagefright to recognize and use custom hardware-
based multimedia codecs. Its acts as an interface between the Stagefright Engine and the codec.

Layers
OpenMAX AL is the interface between multimedia applications, such as a media player, and the platform
media framework. It allows companies that develop applications to easily migrate their applications to
different platforms (customers) that support the OpenMAX AL application programming interface (API)

OpenMAX IL is the interface between media framework, (such as StageFright or MediaCodec API on
Android, DirectShow on Windows, FFmpeg or Libav on Linux, or GStreamer for cross-platform), and a set
of multimedia components (such as an audio or video codecs). It allows companies that build platforms
(e.g. allowing an implementation of an MP3 player) to easily change components like MP3 decoders and
Equalizer effects and buy components for their platform from different vendors.

OpenMAX DL is the interface between physical hardware, such as digital signal processor (DSP) chips,
CPUs, GPUs, and software, like video codecs and 3D engines. It allows companies to easily integrate new
hardware that supports OpenMAX DL without reoptimizing their low level software.

Path-framerwork/av/media/libmedia/IOMX.cpp
Class-BpOMX

Path-framerwork/av/media/libstagefright/omx/omxgraphicbuffersource.cpp
Class-OmxComponentWrapper

Path-framerwork/av/media/libstagefright/omx/OMXmaster.cpp
OMXmaster.cpp Used for:-
1)addPlugin
2)clearPlugins
3)makeComponentInstance
4)destroyComponentInstance
5)enumerateComponents
6)getRolesOfComponent

OMXNodeInstance Used for:-

Path-framerwork/av/media/libstagefright/omx/OMXNodeInstance.cpp
1.setHandle
2.setBufferSource
3.freeNode
4.sendCommand
5.getParameter
6.SetParameter
7.setPortMode
8.enableNativeBuffers
9.getGraphicBufferUsage
OMXutils:
Path-framerwork/av/media/libstagefright/omx/OMXutils.cpp
Used for:-
1.DescribeColorFormatParams
2.DescribeDefaultColorFormat
3.GetComponentRole

OMX_core.h
The OMX_Core header file contains the definitions used by both the application and the component to
access common items.

structures/enumerations in OMX_core.h
1.The OMX_COMMANDTYPE enumeration is used to specify the action in the
OMX_SendCommand macro.

define OMX_SendCommand( \
hComponent, \
Cmd, \
nParam, \
pCmdData) \
((OMX_COMPONENTTYPE*)(hComponent))->SendCommand( \
hComponent, \
Cmd, \
nParam, \
pCmdData) /* Macro End */
Hcomponent -handle of component to execute the command
Cmd-Command for the component to execute
nParam-Parameter for the command to be executed.  When Cmd has the value
OMX_CommandStateSet, value is a member of OMX_STATETYPE.  When Cmd has
the value OMX_CommandFlush, value of nParam indicates which port(s) to flush. -1 is used to flush all
ports a single port index will only flush that port.When Cmd has the value
"OMX_CommandPortDisable"or "OMX_CommandPortEnable", the component's port is given by the
value of nParam. When Cmd has the value "OMX_CommandMarkBuffer" the components pot is given by
the value of nParam.
pCmdData-Parameter pointing to the OMX_MARKTYPE structure when Cmd has the value
"OMX_CommandMarkBuffer".

2.The OMX_STATETYPE enumeration is used to indicate or change the component state.This

enumeration reflects the current state of the component when used with the OMX_GetState macro or
becomes the parameter in a state change command when used with the OMX_SendCommand macro.