Performance Characterization of Video-Shot-Change Detection Methods

U. Gargi, R. Kasturi, S. Strayer

Presented by: Isaac Gerg

What is a Shot?

The process of identifying changes in the scene content of a video sequence so that alternate representations may be derived for the purposes of browsing and retrieval. ~ Quoted directly Shot A sequence of frames shot from the same camera. Shot-Change examples: cuts, transitions, wipes, etc.

Why Do We Care?

Indexing video retrieval

Compression (e.g. MPEG) determining key frames. Removing commercials! (TiVo)

Preview
1.

Create a method for measuring the performance of a shot-change algorithm.

1. 2.

2. 3. 4.

Apply shot-change algorithms to ground truth video sequence. Perform measurements and throughput analysis. Compare the results.

Measure both false detections & missed detections. Measure performance of both cut detection & gradual transition detection.

Ground Truth Video Sequence

640x480 @ 30 frames/s. ~75 minutes in length M-JPEG format Human volunteers used to establish ground truth. Custom software used to notate shotchange.

Defining a Detection

Algorithm detection must occur within so many frames of ground truth detection.
Mapping Range = RM

Cut changes: RM= 3 Gradual Transition: RM=10

Detection Performance Measurements

Detects Recall Detects MD' s
Detects Precision Detects FA' s
Where: MD is Missed Detections; FA is False Alarms.

Desirable Characteristics

90%-95% recall with 70%-75% precision. Robust. Automatic thresholds. High throughput. Perform well on both cuts and gradual transitions.

Algorithms Evaluated

Color Histograms

Frame Difference Measurements

RGB, HSV, YIQ, XYZ, L*a*b, L*u*v, Munsell, Opponent

Dimensionality 1D, 2D, 3D MPEG Algorithms A, B, C, D, E, F Block Matching Methods A, B, C

Bin-to-bin Differences (B2B), Chi-square test, Histogram intersection, Average Color

Best Methods - Cut

Histogram intersection:
Intersecti on(h1, h2 )

min (h [i], h [i])

1 2 i

N fdint (h1 , h2 ) 1 Intersecti on(h1 , h2 )

1D and 3D methods.

Best Methods - Cut

MTM colorspace (many flops). LAB appeared as good compromise when considering throughput. Opponent (OPP) almost as good as LAB, but needs only integer computations.

[image] Hall, E. L. . Computer Image Processing and Recognition. Academic Press, New York

Best Methods - Cut

Best recall: MPEG-A, 97% with 6% precision. Uses only I frames. Best precision: MPEG-D, 88% with 79% recall. Uses I, B, & P frames.

Worst Methods - Cut

Chi-square test histogram difference:

1 (h1[i] h2 [i])2 fdchi 2 N i h1[i] or h2 [i]

Average color of a frame. 2D methods. Indicates luminance is important. YYY colorspace. Indicates color content is important All the block-matching methods.

Best Methods - Transition

Only MPEG algorithms evaluated. MPEG-D: Uses all frames (I, P, B). Uses multiframe differences to detect gradual transitions. Uses 11 parameters. MPEG-F: Uses color information (Y, Cr, Cb). Uses order statistics to detect gradual transitions. Needs 7 parameters.

Worst Methods - Transition

MPEG-A was the worst. Only contains I frames. Most performed poorly as they expected a particular transition curve.

MPEG - Source Effects

Desirable to have a good MPEG method independent of encoder. Authors found dependence on algorithm performance and MPEG encoder used. MPEG does not specify encoding method, only syntax of encoded bitstream. Different error estimates or DCT matrices may be used during encoding. MPEG-F appeared to be the most robust.

Conclusions

Need accurate model of color. Color & luminance information combined yield best results. MPEG shot detection & gradual transition methods have a long way to go. Encoding too variable. Gradual transitions not detected well by an of the MPEG methods.

Questions?