Assignment: 1 Date:27/11/2020

Android Multimedia Video

Task: Traversing trough the Android source code to get familiar with folders/files and locations.

Version: Pie - 9.0.0_r3

Source: https://androidxref.com/

1) Application Framework

Basically application code utilize android.media APIs to interact with the multimedia hardware.

Provides classes that manage various media interfaces in audio and video.

The Media APIs are used to play and, in some cases, record media files.

e.g., play a video streamed over the web or from local storage

android.media -> Binder IPC

2) Binder IPC

The Binder IPC proxies facilitate communication over process boundaries

Location : frameworks/av/media/libmedia

 MediaPlayer class can be used to control playback of audio/video files and streams.

e.g., on how to use the methods in this class can be found in VideoView
  MediaCodecInfo.VideoCapabilities class that supports querying the video capabilities of a
codec. Video performance points are a set of standard performance points defined by
number of pixels, pixel rate and frame rate. 

 MediaExtractor facilitates extraction of demuxed, typically encoded, media data from a

data source.

 MediaCodec.BufferInfo Per buffer metadata includes an offset and size specifying the
range of valid data in the associated codec (output) buffer.

 MediaCodec.Callback
MediaCodec callback interface. Used to notify the user asynchronously of various MediaCodec
events.

Location : /frameworks/av/media/libmedia/include/media/mediarecorder.h

 Media Recorder class is to record audio/video streams form device and setting
various parameters.

Location: /frameworks/av/media/libmedia/mediarecorder.cpp

 setCamera Checks media recorder state (initialized) and set camera for recording.
Eg: mMediaRecorder->setCamera(camera, proxy)
 setPreviewSurface try to set preview surface with setting the video source.
 setVideoSource set video source i.e (mMediaRecorder->setVideoSource(vs).
 setAudioSource initialize media recorder and sets audio source.
 setOutputFormat output format for audio recording is set.
 setVideoEncoder video encoding format to the source is assigned here.

3) Native Multimedia Framework

At the native level, Android provides a multimedia framework that utilizes the Stagefright engine for
audio and video recording and playback. Stage fright comes with a default list of supported software
codecs.

Libmedia utilizes the Media Player Services

I) MediaPlayerService
class is defined in MediaPlayerService.h and definitions are implemented in MediaPlayerService.cpp.

PATH: /frameworks/av/media/libmediaplayerservice/MediaPlayerService.h

 Class MediaPlayerService : public BnMediaPlayerService

PATH: /frameworks/av/media/libmediaplayerservice/MediaPlayerService.cpp

 MediaPlayerService::MediaPlayerService

 MediaPlayerService::removeMediaRecorderClient

 MediaPlayerService::removeClient

Video Formats: H.263, H.264 AVC(BP,MP), H.265 HEVC ,MPEG-4 SP,VP8, VP9, AV1

File Formats: 3GPP (.3gp), MPEG-4 (.mp4), Matroska (.mkv) ,MPEG-TS (.ts), WebM (.webm)

For video Encoding, Decoding, Playback & Streaming speed and resolution are recommended based on
the certain parameters.

e.g.: (SD- Low/High) quality Format (H.264) -> (176 x 144 px @ 12 fps ) (480 x 360 px @ 30 fps) are
recommended respectively.

>> Similarly, for images also Android framework recommends some formats.

II) Stage fright engine

Stage fright is a native media playback tool used by Android and all these weaknesses reside in it.

Stage fright has built-in software base codec.

 Extractor: extract data into required format and set communication between native media
server to kernel driver.

 Player: captures video and provide sources like playing, starting, etc.

 Codex: prepare playback and setup decoder on request from media service to kernel driver.

PATH: /frameworks/av/media/libstagefright/

4) OMX INTEGRATION

The OpenMAX IL provides a standardized way for Stagefright to recognize and use custom hardware-
based multimedia codecs. Its acts as an interface between the Stagefright Engine and the codec.
 Path-framerwork/av/media/libmedia/IOMX.cpp
Class-BpOMX

Path-framerwork/av/media/libstagefright/omx/omxgraphicbuffersource.cpp
Class-OmxComponentWrapper

Path-framerwork/av/media/libstagefright/omx/OMXmaster.cpp
OMXmaster.cpp Used for:-
1)addPlugin
2)clearPlugins
3)makeComponentInstance
4)destroyComponentInstance
5)enumerateComponents
6)getRolesOfComponent

OMXNodeInstance Used for:-

Path-framerwork/av/media/libstagefright/omx/OMXNodeInstance.cpp
1.setHandle
2.setBufferSource
3.freeNode
4.sendCommand
5.getParameter
6.SetParameter
7.setPortMode
8.enableNativeBuffers
9.getGraphicBufferUsage

OMXutils:
Path-framerwork/av/media/libstagefright/omx/OMXutils.cpp
Used for:-
1.DescribeColorFormatParams
2.DescribeDefaultColorFormat
3.GetComponentRole

Other Media Files:

PATH: /frameworks/support/media/src/main/java/androidx/media/

E.g.:
 MediaInterface2.java
 MediaPlayer2.java
 MediaPlayerInterface.java

For unit test the services scripts are here.

PATH: /cts/tests/tests/media/src/android/media/cts/

E.g.:

 AudioTrackTest.java
 DecoderTest.java
 DecoderTestAacDrc.java
 DecoderTestXheAac.java
 EncodeDecodeTest.java
 VideoDecoderPerfTest.java
 VideoEditorTest.java
 VideoEncoderTest.java

Note: Most of these files are licensed as APACHE LICENSE, VERSION 2.0

https://www.apache.org/licenses/LICENSE-2.0

Author’s:

 Shweta Udigiri
 Akula Rama Krishna
 Aman Kumar Das
 Ravi Upadhay
 Gesham M