UNIT-4 Robotic Vision

Introduction
Robot vision is defined as the process of capturing, extracting, characterizing and interpreting the
information obtained from images through a camera or a charge coupled device. A basic robot vision system
has a single stationary camera mounted over the workspace. Several stationary cameras are used in order to
gain multiple perspectives. This is an active form of sensing in which the robot is used to position the sensor
on the basis of the results of pervious measurements. Robot vision systems supply valuable information that
can be used to automate the manipulation of objects. With the use of robotic vision, the position, orientation,
identity and condition of each part in the scene can be obtained. The information obtained can then be used
to plan robot motion so as to determine how to grasp a part and how to avoid collisions with obstacles.

Architecture of Robotic Vision System

The architecture of a robotic vision system typically involves several interconnected components that work
together to perceive and interpret visual information. Here's a high-level overview of the architecture:

1. Sensor Module:
Cameras: The primary sensors for capturing visual data. Different types of cameras may be used
depending on the application, such as monocular cameras, stereo cameras, or depth cameras.
Other sensors: In addition to cameras, robotic vision systems may incorporate other sensors like lidar,
infrared sensors, or sonar to provide supplementary information about the environment.

2. Image Acquisition and Preprocessing:

Image Capture: This module is responsible for capturing images from the cameras or other sensors.
Preprocessing: Raw images captured by the sensors often undergo preprocessing steps such as noise
reduction, color correction, and image enhancement to improve the quality of the input data.

3. Feature Extraction and Representation:

Feature Extraction: This step involves identifying key features in the images that are relevant for the task
at hand. Common features include edges, corners, blobs, and texture patterns.
Feature Representation: Extracted features are represented in a suitable format for further processing. This
may involve encoding features as descriptors or feature vectors.

4. Object Detection and Recognition:

Object Detection: Involves detecting the presence and location of objects of interest in the images. This
can be achieved using techniques such as template matching, Haar cascades, or deep learning-based
approaches like convolutional neural networks (CNNs).
Object Recognition: Once objects are detected, the system may perform recognition to identify the type or
class of each object. This may involve comparing detected features against a database of known objects or
using machine learning algorithms for classification.
5. Scene Understanding and Interpretation:
Scene Understanding: This step involves interpreting the overall scene context based on the detected
objects and their spatial relationships. It may include tasks such as scene segmentation, semantic labeling,
and object tracking.
Contextual Reasoning: The system may use contextual information to infer higher-level knowledge about
the scene, such as understanding the intentions of human actors or predicting future actions.

6. Decision Making and Control:

Decision Making: Based on the perceived scene information, the robotic system makes decisions
regarding its actions or responses. This may involve planning trajectories, selecting actions, or triggering
specific behaviors.
Control: The system generates control commands to actuate the robotic hardware (e.g., motors, actuators)
based on the decisions made. This enables the robot to interact with the environment or perform tasks
autonomously.

7.Feedback and Adaptation:

Feedback Loop: The system continuously receives feedback from sensors and other components to assess
the outcomes of its actions. This feedback loop enables the system to adapt and refine its behavior over time
based on experience.
Learning and Adaptation: Robotic vision systems may incorporate learning algorithms to improve their
performance through experience, such as reinforcement learning, online training, or self-supervised learning.

Image Processing
Image Processing Robotic vision continues to be treated including different methods for processing,
analysing, and understanding. All these methods produce information that is translated into decisions for
robots. From start to capture images and to the final decision of the robot, a wide range of technologies and
algorithms are used like a committee of filtering and decisions. Another object with other colours
accompanied by different sizes.

A robotic vision system has to make the distinction between objects and in almost all cases has to tracking
these objects. Applied in the real world for robotic applications, these machine vision systems are designed
to duplicate the abilities of the human vision system using programming code and electronic parts. As
human eyes can detect and track many objects in the same time, robotic vision systems seem to pass the
difficulty in detecting and tracking many objects at the same time

Image Acquisition Camera:

Image acquisition cameras play a fundamental role in robotic vision systems by capturing visual data from
the environment. These cameras are specifically designed for use in robotics and computer vision
applications, offering features tailored to the requirements of real-time image processing and analysis. Here
are some key aspects of image acquisition cameras in robotic vision systems:

1. Types of Cameras:
Monocular Cameras: These cameras have a single lens and capture 2D images of the scene. They are
commonly used for tasks like object detection, tracking, and navigation.
Stereo Cameras: Stereo cameras consist of two or more synchronized cameras positioned slightly apart to
mimic human binocular vision. They capture images from slightly different perspectives, enabling depth
perception and 3D reconstruction of the scene.
Depth Cameras: Also known as 3D cameras or depth sensors, these cameras provide depth information
along with color imagery. They use techniques like structured light, time-of-flight, or stereo vision to
measure distances to objects in the scene.
 Pan-Tilt-Zoom (PTZ) Cameras: PTZ cameras can be remotely controlled to pan, tilt, and zoom,
allowing for flexible viewpoint selection and tracking of objects within a larger area.

2. Key Features:
Resolution: The resolution of the camera determines the level of detail in the captured images. Higher
resolutions are beneficial for tasks that require precise feature extraction or object recognition.
Frame Rate: Frame rate refers to the number of frames per second (fps) captured by the camera. Higher
frame rates are crucial for capturing fast-moving objects or maintaining smooth motion in real-time
applications.
Field of View (FOV): The FOV specifies the extent of the scene captured by the camera. Wide-angle
lenses offer a broader FOV, while telephoto lenses provide a narrower FOV with greater magnification.
Sensitivity and Dynamic Range: Sensitivity to light (low-light performance) and dynamic range (ability to
capture details in both bright and dark areas) are essential for operating in diverse lighting conditions.
Interface: Cameras may utilize various interfaces for data transfer, such as USB, GigE Vision, Camera
Link, or HDMI. The choice of interface depends on factors like data bandwidth, latency requirements, and
compatibility with the robotic system.

Triggering and Synchronization: Cameras may support triggering mechanisms to capture images at specific
times or in response to external events. Synchronization capabilities ensure precise timing between multiple
cameras or with other sensors in the system.

3. Integration with Robotic Systems:

Image acquisition cameras are typically integrated with robotic platforms or control systems using
standardized interfaces and protocols.
They may be mounted on robot arms, mobile robots, or stationary platforms, depending on the application
requirements.
Integration may involve calibration to ensure accurate geometric and photometric alignment between the
camera(s) and the robot's coordinate system.
Image acquisition cameras serve as the eyes of robotic vision systems, enabling robots to perceive and
interact with their surroundings through visual information. The selection of an appropriate camera depends
on factors such as the application's requirements, environmental conditions, and budget constraints.

Image Enhancement:
Image enhancement is a crucial step in the processing pipeline of robotic vision systems. It involves
techniques to improve the visual quality of images captured by cameras, making them more suitable for
subsequent analysis and interpretation. Here are some common image enhancement techniques used in
robotic vision:

1. Noise Reduction:
 Median Filtering: A non-linear filtering technique that replaces each pixel's value with the median value of
neighboring pixels. It effectively removes impulse noise (salt-and-pepper noise) while preserving image
edges.
Gaussian Filtering: Applies a convolution operation with a Gaussian kernel to smooth the image and
reduce Gaussian noise. It's commonly used for additive noise reduction.
Bilateral Filtering: A spatial-domain filter that smooths images while preserving edges. It considers both
spatial and intensity differences between pixels, making it effective for noise reduction without blurring
edges.

2. Contrast Enhancement:
Histogram Equalization: A technique that redistributes pixel intensities in the image histogram to achieve a
more uniform distribution. It enhances the overall contrast of the image, making details more visible.
Adaptive Histogram Equalization (AHE): A variant of histogram equalization that operates on local image
regions rather than the entire image. It adapts to local contrast variations, leading to better results, especially
in images with uneven illumination.
Gamma Correction: Adjusts the gamma value to correct non-linearities in the display or capture devices. It
can enhance the visibility of details in both dark and bright regions of the image.

3. Sharpness Enhancement:
Unsharp Masking: A technique that enhances image details by subtracting a blurred version of the image
from the original. It accentuates edges and fine structures, making the image appear sharper.
High-Pass Filtering: Applies a high-pass filter to extract high-frequency components from the image,
which represent details and edges. By enhancing these components, the image's sharpness is improved.

4. Color Correction:
White Balance Adjustment: Corrects color casts caused by varying light sources (e.g., daylight,
fluorescent). It ensures that white objects appear neutral regardless of the lighting conditions.
Color Space Transformations: Converting images between different color spaces (e.g., RGB, HSV, Lab)
can improve color representation and separation, making it easier to detect and discriminate objects based on
their colors.

5. Edge Enhancement:
Laplacian Filtering: Highlights edges in the image by emphasizing regions of high spatial gradient. It's
particularly effective for detecting sharp transitions between image regions.
Morphological Operations: Dilating and eroding image regions can enhance or suppress edges,
respectively. Morphological operations are useful for preprocessing images before edge detection or
segmentation tasks.
6. Resolution Enhancement:
Super-Resolution: A set of techniques that aim to enhance image resolution beyond the sensor's native
resolution. Super-resolution methods generate high-resolution images from one or more low-resolution input
images, improving image quality and detail.

Image enhancement techniques can be applied individually or in combination to achieve the desired visual
quality and prepare images for subsequent processing tasks in robotic vision systems. The choice of
enhancement methods depends on factors such as the characteristics of the captured images, the specific
application requirements, and computational considerations.

Image Segmentation:
Image segmentation plays a crucial role in robotics, particularly in tasks where robots need to understand
and interact with their environment. Here's how image segmentation is utilized in robotics:

1. Object Detection and Recognition:

Image segmentation helps robots identify and differentiate objects in their surroundings. By segmenting
the image into regions corresponding to different objects or parts of objects, robots can accurately detect and
recognize objects of interest.
This capability is essential for tasks such as pick-and-place operations in manufacturing, object
manipulation in warehouse automation, and identifying obstacles for navigation in autonomous vehicles.

2. Scene Understanding:
Robots use image segmentation to gain a deeper understanding of the scene they are operating in. By
segmenting the image into meaningful regions based on attributes like color, texture, or depth, robots can
extract valuable information about the environment.
Scene understanding enables robots to make informed decisions about their actions, such as avoiding
obstacles, navigating through complex environments, or interacting with objects in a coordinated manner.
3. Human-Robot Interaction:
In human-robot interaction scenarios, image segmentation helps robots perceive and interpret human
actions and gestures. By segmenting the image to identify human body parts or hand gestures, robots can
understand and respond to human commands and gestures more effectively.
This capability is crucial for collaborative robots (cobots) working alongside humans in settings like
manufacturing, healthcare, or service industries.

4. Visual Servoing and Manipulation:

Image segmentation is used in visual servoing applications to guide robot manipulators towards specific
targets or objects of interest. By segmenting the image to identify target objects or regions, robots can
accurately plan and execute manipulation tasks.
Visual servoing and manipulation are essential for tasks such as grasping objects in cluttered
environments, manipulating objects with precision, or performing delicate assembly tasks.

5. Navigation and Mapping:

In mobile robotics, image segmentation helps robots navigate through complex environments and create
detailed maps of their surroundings. By segmenting the image to identify obstacles, landmarks, and
navigable pathways, robots can plan optimal paths and avoid collisions.
Image segmentation is often combined with techniques like simultaneous localization and mapping
(SLAM) to enable robots to localize themselves within the environment while simultaneously mapping the
surroundings.

Image segmentation enhances the perception and cognitive capabilities of robots, enabling them to interact
with the world more intelligently and autonomously. By segmenting images into meaningful regions, robots
can extract valuable information about their environment, make informed decisions, and perform complex
tasks with precision and efficiency.

Image Transformation:
In robotics, image transformation refers to the process of manipulating the visual data captured by cameras
or other sensors to extract useful information or prepare it for further analysis. Image transformation
techniques are essential in enabling robots to understand their surroundings, make decisions, and perform
tasks effectively. Here's how image transformation is used in robotics:

1. Preprocessing:
Before using visual data for tasks such as object detection, localization, or navigation, it often requires
preprocessing to improve its quality and suitability for analysis.
Image preprocessing techniques in robotics may include:
  Noise reduction: Removing noise from images using filters such as median filters or Gaussian filters
to improve the accuracy of subsequent processing steps.
 Image denoising: Techniques like wavelet denoising or total variation denoising are used to reduce
noise while preserving image details.
 Image enhancement: Adjusting the brightness, contrast, or color balance of images to improve their
visual quality and make relevant features more distinguishable.
 Image registration: Aligning images from different viewpoints or sensors to facilitate fusion and
comparison.

2. Feature Extraction:
Image transformation is used to extract relevant features from visual data, such as edges, corners, key
points, or descriptors, which are essential for tasks like object recognition, localization, and mapping.
Feature extraction techniques in robotics may include:
 Edge detection: Identifying edges or boundaries in images using operators like the Sobel, Canny, or
Prewitt edge detectors.
 Corner detection: Identifying corner points in images using algorithms like the Harris corner detector
or the Shi-Tomasi corner detector.
 Key point detection and description: Identifying distinctive key points in images and describing
their local appearance using algorithms like Scale-Invariant Feature Transform (SIFT) or Speeded-Up
Robust Features (SURF).

3. Object Detection and Recognition:

Image transformation techniques are used to detect and recognize objects in robot environments, enabling
robots to interact with and manipulate objects effectively.
Object detection and recognition techniques in robotics may include:
 Template matching: Comparing image patches with predefined templates to detect objects with
similar appearance.
 Machine learning-based methods: Training classifiers or deep learning models to detect and
recognize objects from visual data.
 Histogram-based methods: Comparing color histograms or texture histograms to identify objects
based on their visual characteristics.

4. Scene Understanding:
Image transformation helps robots understand the structure and layout of their environment, enabling
them to navigate, plan paths, and make decisions autonomously.
Scene understanding techniques in robotics may include:
 Image segmentation: Dividing images into semantically meaningful regions or objects to facilitate
understanding and interpretation.
 Semantic labeling: Assigning semantic labels or categories to image regions based on their content,
such as road, sidewalk, building, or vegetation.
 Depth estimation: Estimating the depth or distance to objects in the scene using techniques like
stereo vision, structured light, or time-of-flight sensors.
Overall, image transformation techniques are indispensable tools in robotics, enabling robots to perceive,
understand, and interact with their environment using visual information effectively. These techniques play a
crucial role in various robotics applications, including autonomous navigation, object manipulation, human-
robot interaction, and environmental monitoring.

Camera Transformations and Calibrations in Robotics:

In robotics, camera transformations and calibrations are essential processes for accurately interpreting visual
data and enabling robots to perceive and interact with their environment effectively. Here's an overview of
camera transformations and calibrations in robotics:

1. Camera Transformations:
Camera transformations involve transforming the coordinates of points in the 3D world space into the 2D
image space captured by the camera, and vice versa. These transformations are crucial for tasks such as
object localization, navigation, and scene reconstruction. The main types of camera transformations in
robotics include:

 Projection Matrix: The projection matrix maps 3D points in the world space to their corresponding
2D image coordinates. It considers intrinsic camera parameters such as focal length, principal point,
and lens distortion.

 Extrinsic Parameters: Extrinsic parameters describe the position and orientation of the camera
relative to the world coordinate system. They specify the translation and rotation of the camera with
respect to a reference frame, such as the robot's base or a global coordinate system.

 Homogeneous Transformations: Homogeneous transformations represent both translation and

rotation in a unified mathematical framework. They are commonly used to express the relationship
between different coordinate frames, such as the camera frame, robot base frame, and world frame.

 Coordinate Frame Transformations: Coordinate frame transformations involve converting points

between different coordinate frames, such as transforming points from the camera frame to the robot
base frame or vice versa.

2. Camera Calibration:

Camera calibration is the process of estimating the intrinsic and extrinsic parameters of a camera to
accurately model its behavior and correct distortions. Calibration is crucial for ensuring accurate
measurements and precise localization in robotics applications. The main steps in camera calibration
include:
  Intrinsic Parameters Calibration: Intrinsic parameters characterize the internal geometry and
optics of the camera, including focal length, principal point, and lens distortion parameters (radial
and tangential distortion).

 Extrinsic Parameters Calibration: Extrinsic parameters determine the position and orientation of
the camera relative to the robot or world coordinate system. This involves accurately estimating the
translation and rotation of the camera with respect to a known reference frame.

 Calibration Pattern Acquisition: Calibration patterns, such as checkerboards or grids with known
dimensions, are used to capture images from different viewpoints. These patterns provide the
necessary geometric constraints for estimating camera parameters.

 Camera Parameter Estimation: Using images of the calibration pattern captured from different
viewpoints, camera parameters are estimated through optimization techniques such as least squares
minimization or bundle adjustment.

 Validation and Error Analysis: Once the camera parameters are estimated, the calibration results
are validated using additional images or test data. Error analysis helps assess the accuracy of the
calibration and identify any residual distortions or inaccuracies.

Camera transformations and calibrations are fundamental processes in robotics, enabling robots to
accurately interpret visual data, localize themselves in the environment, and interact with objects with
precision and reliability. These techniques are essential for a wide range of robotics applications, including
robot navigation, object detection and recognition, augmented reality, and 3D reconstruction.

Industrial Applications of Robot Vision

Robot vision, also known as machine vision, plays a significant role in various industrial applications,
enhancing automation, quality control, and efficiency. Here are some common industrial applications of
robot vision:

1. Quality Inspection and Assurance:

Robot vision systems are used extensively for inspecting manufactured parts, detecting defects, and
ensuring product quality.
Vision-guided robots can perform visual inspections at high speed and with high precision, detecting
imperfections such as surface scratches, cracks, dimensional deviations, or missing components.
Industries like automotive, electronics, aerospace, and pharmaceuticals rely on robot vision for quality
assurance throughout the production process.

2. Assembly and Manufacturing:

 Vision-guided robots are employed in assembly lines to accurately position components, align parts, and
perform intricate assembly tasks.
Vision systems provide feedback to robots, allowing them to adapt their actions based on variations in part
geometry or orientation.
Collaborative robots (cobots) equipped with vision systems can work alongside humans, handling delicate
assembly tasks or assisting with manual labor.

3. Robotic Pick-and-Place:
Vision-guided robots are used for automated picking and placing of objects in manufacturing,
warehousing, and logistics operations.
Vision systems identify objects on conveyor belts or in bins, determine their position and orientation, and
guide robotic arms to grasp and manipulate them accurately.
Applications include sorting, packaging, palletizing, and order fulfillment in industries such as e-
commerce, food and beverage, and consumer goods.

4. Barcode Reading and Identification:

Robot vision systems are employed for reading and decoding barcodes, QR codes, and other types of
symbologies used for product identification and tracking.
Vision-guided robots scan labels, tags, or markers on products, packaging, or containers, extracting
information such as serial numbers, batch codes, or product IDs.
Barcode reading enables traceability, inventory management, and supply chain optimization in
manufacturing and logistics.

5. Robotic Welding and Fabrication:

Vision-guided robotic welding systems use cameras to monitor weld seams, detect joint locations, and
adjust welding parameters in real-time.
Vision systems ensure precise alignment of welding torches or laser heads with workpieces, improving
weld quality, consistency, and productivity.
Industries like automotive, aerospace, and metal fabrication rely on vision-guided welding for producing
high-quality welds in complex components.

6. Packaging and Labeling:

Vision systems are used for inspecting packaging materials, verifying label placement, and ensuring
compliance with regulatory requirements.
Robots equipped with vision systems handle packaging tasks such as filling, sealing, labeling, and
cartooning, ensuring product integrity and brand consistency.
Vision-guided packaging systems improve efficiency, reduce errors, and enable flexible packaging
solutions for diverse product types and formats.
7. Material Handling and Sorting:
Vision-guided robots are employed for sorting, segregating, and handling objects based on their visual
characteristics, such as color, shape, size, or texture.
Vision systems identify and classify objects on conveyor belts or in bins, directing robots to transport
them to designated locations or processing stations.
Material handling applications include recycling, waste management, mail sorting, and parcel handling in
logistics and distribution centers.

Robot vision systems are integral to modern industrial automation, enabling robots to perceive and interact
with the environment intelligently, accurately, and autonomously. They play a vital role in improving
productivity, quality, and flexibility in manufacturing and logistics operations across various industries.