Image Processing

ThinhUV - iOS Team

Contents
1. Basic concepts
2. Advanced Operations
3. Optimization
4. Demo
5. Q&A
Basic concepts
• What’s Image?

Image is a 2-D array of pixels

- Pixels are samples
- Image has limited resolution
Basic concepts
• Bitmap
- A set of pixels with color and transparency
-> Break into pieces when zoom in
• Vector
- Use methematical fomulas to draw lines and curves
Basic concepts
● Color space
- RGB
- Grayscale
- HSV

● Blending
Basic concepts
• What’s Image Processing?
=> Input + mathematical operation -> output
Input: image, filter, video, output of another one
Mathematical operation: fill color, detect egde, detect face
algorithm, color lookup, Gaussian algorithm …
Output: like input :D
Basic concepts
● Graphic Context: a global state object that holds all the
infomation for drawing.
○ Store resource, state
○ Generate result …
Basic concepts
• Filter: allows us to apply various effects on image, able
chaining
• Capture mode: still and real-time
Basic concepts
What are Apple definitions?

● Core Image
- Greate framework built-in on Apple systems.
- Operate data types from Core Graphics, Core Video and
Image I/O frameworks using either a GPU or CPU rendering
path.
Basic concepts
● Context: CIContext, CGContextRef
● Input and output representation: CIImage, CGImage,
UIImage, Video(Data)
● Filter: CIFilter
Basic concepts
• CIContext
An evaluation context for rendering image processing
results and performing image analysis.

import CoreImage

let context = CIContext(options: nil)

ciContext = CIContext(options: nil)

// Advanced options will be present later

Basic concepts
● CIFilter
- A representation of an image to be processed or produced
by Core Image filters
- Apply for video
- Apply for Game Scene
Basic concepts
● CIFilter
import CoreImage

let filter = CIFilter(name: “CIBoxBlur”)

More 170 built-in filters (Referrence)

Basic concepts
● CIFilter : chain

Input -> filter 1 -> filter 2 -> filter 3 … -> ouput

* Quickly and efficiently

Advanced 0perations
● GPU and CPU
OpenGL and OpenGL ES
import CoreImage

let eaglContext = EAGLContext(api: .openGLES3) ?? EAGLContext(api: .openGLES2) // 1

let ciContext = CIContext(eaglContext: eagl, options: nil) // 2
Advanced 0perations
● Metal: Enabling high performance for graphics rendering
and parallel compute workflows (GPU)

import Metal
let device = MTLCreateSystemDefaultDevice()
let ciContext = CIContext(mtlDevice: device)
Advanced operations
● Shading: use GPU to perform image and video
manipulation much faster than could be done in CPU-
bound routine
● Example: gamma filter
-> 40x faster than Core Image
-> 180x faster than CPU-bound
Advanced operations
● Shader: actual complex to build but very powerful
● Suggested open frameworks: GPUImage

void main()
{
gl_Position = transformMatrix * vec4(position.xyz, 1.0) * orthographicMatrix;
textureCoordinate = inputTextureCoordinate.xy;
}
)
Advanced operations
● Kernel: actual complex to build but very powerful
● Suggested open frameworks: Vivid

kernel vec4 filterKernel(__sample image, __sample blurredImage) {

return vec4(vec3(image.rgb - blurredImage.rgb + vec3(0.5,0.5,0.5)), image.a);
}
Optimization
• Create singleton for CIContext (very important)
• Clear cache if possible
• If the filters chain rendering is slow then try to reorder
filters
• Use autoreleasepool if possible
DEMO