A

PROJECT REPORT

ON

Vision-based Programming, AI in camera vision application

Submitted by:

DIVYA SETH
(BANASTHALI VIDYAPITH)

PROJECT INTERN
 TVS MOTOR

Thirukkurungudi Vengaram Sundram Motor

CERTIFICATE

Certified that this project report “Vison based Programming, AI in camera vision application
” is the bonafide work of “DIVYA SETH” who carried out the project work under my
supervision.

S.K.Vijay I Venkata Vishnu Vihari

General Manager Senior Manager
Project Mentor Project Guide
TVS Motor TVS Motor

Himansu Shaw
Assistant Manager
Project Guide
TVS Motor
 ACKNOWLEDGEMENT

I would like to express our special thanks of gratitude to my project mentor S.K. VIJAY, as well
as my project guide I VENKATA VISHNU VIHARI & my project guide HIMANSU SHAW
who gave me the golden opportunity to do this wonderful project on the topic “Vision-based
Programming, AI in camera vision application”, which also helped me in doing a lot of research
and I, came to know about so many new things. I’m thankful to them. Secondly, I would also
like to thank our parents who helped us a lot in finalizing this project within the limited time
frame.

Name: DIVYA SETH

(BANASTHALI VIDYAPITH)

PROJECT INTERN
TVS MOTOR
 TABLE OF CONTENTS
TITLE
LIST OF FIGURES
1.0 INTRODUCTION
1.1 AI Machine Vision
1.2 Applications of MV
2.0 IMPLEMENTATION
2.1 Pre-Design Solution
2.2 Camera Selection Criteria
2.3 Problem Statement
3.0 METHODOLOGY
3.1 7 Layers of OSI Model
4.0 CHALLENGES
4.1 System Integration
5.0 RESULT AND DISCUSSION
5.1 Node-Red
5.2 Dashboard Result
6.0 CONCLUSION
7.0 DISCUSSION & FUTURE WORK
REFERENCES
 LIST OF FIGURES

Figure No. Title

1.1 AI Machine Vision
1.2 Image based Inspection & Sorting
2.1 Smart Camera
2.2 Compact Vision System
2.3 White Light Interferometry
2.4 Scanning Lasers
3.1 7-Layers of OSI Model

CHAPTER 1

INTRODUCTION
1.1 AI Machine Vision

Fig. 1.1 AI Machine Vision

Machine vision (MV) is the technology and methods used to provide imaging-based
automatic inspection and analysis for such applications as automatic inspection, process
control, and robot guidance, usually in industry. Machine vision refers to many technologies,
software and hardware products, integrated systems, actions, methods, and expertise.
Machine vision as a systems engineering discipline can be considered distinct from computer
vision, a form of computer science. It attempts to integrate existing technologies in new ways
and apply them to solve real-world problems. The term is the prevalent one for these
functions in industrial automation environments but is also used for these functions in other
environments such as security and vehicle guidance.

The overall machine vision process includes planning the details of the requirements and
project and then creating a solution. During run-time, the process starts with imaging,
followed by automated analysis of the image and extraction of the required information.

Imaging-based automatic inspection and sorting

 Fig. 1.2 Image-Based Inspection & Sorting

The primary uses for machine vision are imaging-based automatic inspection and sorting and
robot guidance. The former is abbreviated as "automatic inspection".

Methods and sequence of operation

The first step in the automatic inspection sequence of operation is the acquisition of an
image, typically using cameras, lenses, and lighting that has been designed to provide the
differentiation required by subsequent processing. MV software packages and programs
developed in them then employ various digital image processing techniques to extract the
required information and often make decisions (such as pass/fail) based on the extracted
information.

Equipment
The components of an automatic inspection system usually include lighting, a camera or
other imager, a processor, software, and output devices.
Imaging
The imaging device (e.g. camera) can either be separate from the main image processing unit
or combined with it in which case the combination is generally called a smart camera or
smart sensor. Inclusion of the full processing function into the same enclosure as the camera
is often referred to as embedded processing. When separated, the connection may be made to
specialized intermediate hardware, a custom processing appliance, or a frame grabber within
a computer using either an analog or standardized digital interface (Camera Link,
CoaXPress). MV implementations also use digital cameras capable of direct connections
(without a frame grabber) to a computer via FireWire, USB, or Gigabit Ethernet interfaces.

While conventional (2D visible light) imaging is most commonly used in MV, alternatives
include multispectral imaging, hyperspectral imaging, imaging various infrared bands, line
scan imaging, 3D imaging of surfaces, and X-ray imaging. Key differentiations within MV
2D visible light imaging are monochromatic vs. color, frame rate, resolution, and whether or
not the imaging process is simultaneous over the entire image, making it suitable for moving
processes.

Though the vast majority of machine vision applications are solved using two-dimensional
imaging, machine vision applications utilizing 3D imaging are a growing niche within the
industry. The most commonly used method for 3D imaging is scanning-based triangulation
which utilizes the motion of the product or image during the imaging process. A laser is
projected onto the surfaces of an object. In machine vision, this is accomplished with a
scanning motion, either by moving the workpiece or by moving the camera & laser imaging
system. The line is viewed by a camera from a different angle; the deviation of the line
represents shape variations. Lines from multiple scans are assembled into a depth map or
point cloud. Stereoscopic vision is used in special cases involving unique features present in
both views of a pair of cameras. Other 3D methods used for machine vision are time of flight
and grid-based. One method is grid array-based systems using the pseudorandom structured
light system as employed by the Microsoft Kinect system circa 2012.
Image processing
After an image is acquired, it is processed. Central processing functions are generally done
by a CPU, a GPU, an FPGA, or a combination of these. Deep learning training and inference
impose higher processing performance requirements. Multiple stages of processing are
generally used in a sequence that ends up as the desired result. A typical sequence might start
with tools such as filters which modify the image, followed by extraction of objects, then
extraction (e.g. measurements, reading of codes) of data from those objects, followed by
communicating that data, or comparing it against target values to create and communicate
"pass/fail" results. Machine vision image processing methods include;

Stitching/Registration: Combining adjacent 2D or 3D images.

Filtering (e.g. morphological filtering)
Thresholding: Thresholding starts with setting or determining a gray value that will be
useful for the following steps. The value is then used to separate portions of the image, and
sometimes to transform each portion of the image to simply black and white based on
whether it is below or above that grayscale value.

Pixel counting: counts the number of light or dark pixels[citation needed]

Segmentation: Partitioning a digital image into multiple segments to simplify and/or change
the representation of an image into something more meaningful and easier to analyze.

Edge detection: finding object edges

Color Analysis: Identify parts, products, and items using color, assess quality from color,
and isolate features using color.
Blob detection and extraction: inspecting an image for discrete blobs of connected pixels
(e.g. a black hole in a grey object) as image landmarks.

Neural net / deep learning/machine learning processing: weighted and self-training multi-
variable decision making Circa 2019 there is a large expansion of this, using deep learning
and machine learning to significantly expand machine vision capabilities.
Pattern recognition including template matching. Finding, matching, and/or counting specific
patterns. This may include the location of an object that may be rotated, partially hidden by
another object, or varying in size.

Barcode, Data Matrix, and "2D barcode" reading

Optical character recognition: automated reading of text such as serial numbers
Gauging/Metrology: measurement of object dimensions (e.g. in pixels, inches, or
millimeters)
Comparison against target values to determine a "pass or fail" or "go/no go" result. For
example, with code or bar code verification, the read value is compared to the stored target
value. For gauging, a measurement is compared against the proper value and tolerances. For
verification of alphanumeric codes, the OCR'd value is compared to the proper or target
value. For inspection for blemishes, the measured size of the blemishes may be compared to
the maximums allowed by quality standards.

Outputs
A common output from automatic inspection systems is pass/fail decisions. These decisions
may in turn trigger mechanisms that reject failed items or sound an alarm. Other common
outputs include object position and orientation information for robot guidance systems.
Additionally, output types include numerical measurement data, data read from codes and
characters, counts and classification of objects, displays of the process or results, stored
images, alarms from automated space monitoring MV systems, and process control signals.
This also includes user interfaces, interfaces for the integration of multi-component systems,
and automated data interchange.

1.2 Application of machine vision

1. Final inspection of sub-assemblies.
2. Engine part inspection.
3. Label inspection on products.
4. Checking medical devices for defects.
5. Final inspection cells.
 6. Robot guidance.
7. Verifying Datamatrix codes.
8. Checking orientation of components.
9. Traceability of manufactured products.
10. Packaging Inspection.
11. Checking laser marks and cuts.
12. Medical vial inspection.
13. Food pack checks.
14. Reading bar codes.
15. Verifying engineered components.
16. Object Detection
17. Measurement
18. Flaw Detection
19. Print Defect Identification
20. Identification
21. Locating
22. Counting
23. Deep Learning/Artificial Intelligence (Ai)
24. Pattern Recognition & Pattern Matching
25. Robot Control
26. Image Archiving

CHAPTER 2

IMPLEMENTATION

2.1 PRE-DESIGNED SOLUTION

1. Smart cameras and sensors
Smart cameras are an elegant solution that combines the sensor, processor, and I/O in a
compact housing, often no bigger than a standard industrial camera. These products offer an
intuitive user interface that can be understood easily with training.

Fig. 2.1 Smart Camera

For configuration of the inspection task a separate computer (i.e. a laptop) is required, that
will be connected to the smart camera via a network interface, however, during runtime no
connection to this computer is necessary.

2. Compact vision systems

Compact vision systems are based on embedded processing technology and combine the
processor, industrial communication interfaces, and support for multiple cameras in a robust,
compact industrial housing. These systems offer optimized heat dissipation and enable the
use of fast processors at maximum performance, allowing the simultaneous operation of
multiple cameras.
 Fig. 2.2 Compact Vision System
Compact vision systems generally include an inbuilt graphical user interface that can be
operated directly with a touch screen monitor or mouse and keyboard. Live images and
results can be displayed on a monitor connected directly to the system.

3. PC based vision systems

A solution based on PC systems offers the highest degree of flexibility, however, care has to
be taken to choose the appropriate operating system and selected components. Modern
cameras produce high data rates and demanding software solutions require immense
computing power. Due to the extreme loads, absolute compatibility must be achieved to
ensure a successful solution. Sourcing all the components separately increases risk and can
take significant time.

4. White light interferometry (WLI)

WLI is a non-contact optical method for surface height measurement on 3-D structures of
various profiles, providing resolutions down to the single-digit nanometer range. This is
accomplished by splitting a beam of white light. Part of the beam will contact a reference
mirror, while the other will contact the workpiece.
 Fig. 2.3 White Light Interferometry

These two beams will interfere—hence the name, “interferometry”—with each other,
creating light and dark bands. When a portion of the specimen is in focus, the bands are
recorded onto a CCD camera, while the software reconstructs the 3-D data set. The
magnification also can be increased to measure the overall surface profile at surface
roughness resolutions.

5. Camera Technologies
Proprietary technology for high-speed 3-D reconstruction of pixel-based images, Points from
Focus (PFF) provides resolution down to 50 nanometers. Relatively speaking, this is a low-
resolution method for measuring the Z-axis, but the speed of PFF makes it an excellent
technology choice for measuring components such as diamond chips, fastener tips, surgical
screws, and a variety of other manufactured micro-components.

6. Structured illumination
A more recently developed technology, structured illumination will continue being refined
for far greater capabilities and super-resolution levels than are available today. With the
increased speeds of image capture devices, image processors (such as frame grabber boards),
and computer systems, processing massive amounts of raw images will not be the bottleneck.
Structured illumination microscopy provides resolution capability similar to WLI
technology. This technology is currently used on biological specimens because of its optical
sectioning capability, which measures focal planes deep into a specimen that would
otherwise require manual sectioning or cutting.

7. Scanning lasers (single point)

Scanning laser technology has improved and evolved since its first applications, and now
includes a variety of methods. One of those methods is known as point scanning. Analogous
to a record player needle, a point scanning laser examines a surface for bumps and vibrations.
A beam scanning at regular intervals in two dimensions can detect fine, structural details,
resulting in high throughput. This single-point scanning method takes more time than
confocal or WLI technologies.

Fig. 2.4 Scanning Lasers

##Important benefits of using programmable software include increased accuracy, the

elimination of operator subjectivity, immediate calculation of summary data and statistics,
creation of custom reports, and the ability to save configurations for repeated, systematic use.
2.2 CAMERA SELECTION CRITERIA
Resolution
The resolution of the camera determines the detail that can be extracted from the image. To
reliably detect or measure a feature in the image, you need at least two pixels on that feature.
Stated another way, the pixel size should be no larger than half the size of the feature to be
measured. “For example, a typical VGA camera will have a resolution of 640 × 480 pixels. The
smallest feature that can be detected is therefore 1/320 or 0.31% horizontally and 1/240 or
0.42% vertically. Higher-resolution sensors can resolve smaller features within a given field of
view (FOV) and/or increase the FOV.”
Care must be taken when using sensors with smaller pixels that the optics in the system have the
resolving power to match the sensors. The advantages of a higher-resolution sensor or sensors
with smaller pixels can be lost in the blur of low-resolution optics. With high-magnification
optics, the wavelength of the light becomes important in determining minimum effective
pixel size.

Sensor size
Sensor size and the optics determine the FOV. For machine vision, the most popular sensor sizes
are 1/2 and 1/3 type (also called 1/2 or 1/3 in.). The 1/2 type refers to the active image area of an
old 1/2-in. imaging tube. When selecting a lens, make sure that it is compatible with a sensor size
equal to or greater than the sensors in the camera. Pixels (or cell) size or pitch is typically
reported in microns. The pixel size is always measured from pixel center to pixel center. The
active imaging size of a sensor is therefore the pixel size times the number of active or effective
pixels.

Sensitivity and wavelengths

The sensitivity specification of a camera is its response to white visible light. For monochrome
cameras, it is usually reported in terms similar to 400 lux,f/5.6, 0 dB. This means for a fixed
visible light intensity of 400 lux and a camera gain set to 0 dB, a lens aperture of f/5.6 is needed
to get a specific target level on the output video signal. The higher the aperture number (smaller
aperture) the more sensitive the camera, since it takes less light to achieve the target output level.
Keep in mind that aperture number and aperture area are not in a linear relationship.
It is useful to compare the sensitivity specification between cameras of similar types from a
single camera manufacturer. Comparing specs among different camera types and different
camera manufactures can be difficult because of varying evaluation methods and camera
characteristics.
For many machine-vision environments, the imaging is done at specific wavelengths instead of
with white light. Most camera manufacturers will provide a spectral-response or quantum-
efficiency chart for their cameras. This chart is usually a relative response (the highest point of
sensitivity is 100%) between the range of 400 to 1000 nm. Most color cameras and some
monochrome cameras have an infrared-cut filter that significantly reduces the sensitivity above
650 nm.

Minimum illumination specification

The minimum illumination specification is a reflection of the low light sensitivity and gain of the
camera. It is usually reported in terms similar to "0.1 lux atf/1.4 with AGC ON without IR filter."
This statement means that with the lens aperture wide open (f/1.4) and automatic gain control on
(gain will be at maximum AGC range), a target output level will be achieved at 0.1 lux. The
lower the lux readings, the more capable the camera is of imaging low light levels.
Care must be taken when using this specification to compare different types of cameras and
different camera manufacturers. This specification is very dependent on the maximum gain of
the camera, exposure time, and target output level. Very high gains are useful to obtain an image
but may be too noisy for the application. Every 6 dB of gain is a doubling of the signal
amplification. This measurement is typically done at video rates (exposure time of 33 ms). Very
low minimum illumination specs are a reflection of very high gains and/or much longer exposure
times.

Signal-to-noise ratio
The signal-to-noise ratio is the ratio of the maximum signal and the minimum noise the camera
can achieve within an output image. This is usually expressed in decibels (dB) in a video camera.
The higher the decibel spec, the better the contrast between signal and noise from the camera. In
machine vision, a digital image is needed, so the image is digitized either in a frame grabber or in
the camera. Since 6 dB represents a doubling of the ratio, 48 dB is equal to 8 bits, and 60 dB is
equal to 10 bits. The effective number of bits will depend on analog signal transmission losses
and the frame grabber.

2.3 PROBLEM STATEMENT

Robots and human labor both are required during the execution of any Automatic or Manual
programming or development, in which the inclusion of humans brings in the human error too
likely child part detection which is missing or it could be any issue big or small. This issue
comes up with a solution of the vision detection system, & hence the development of AI
Machine Vision arrives.
 CHAPTER 3

METHODOLOGY

3.1 7 LAYERS OF OSI MODEL:

The OSI Model (Open Systems Interconnection Model) is a conceptual framework used
to describe the functions of a networking system. The OSI model characterizes
computing functions into a universal set of rules and requirements to support
interoperability between different products and software. In the OSI reference model, the
communications between a computing system are split into seven different abstraction
layers: Physical, Data Link, Network, Transport, Session, Presentation, and Application.

Fig. 3.1 7-Layers of OSI Model

Physical Layer
The lowest layer of the OSI Model is concerned with electrically or optically transmitting
raw unstructured data bits across the network from the physical layer of the sending
device to the physical layer of the receiving device. It can include specifications such as
voltages, pin layout, cabling, and radio frequencies. At the physical layer, one might find
“physical” resources such as network hubs, cabling, repeaters, network adapters, or
modems.
Data Link Layer
At the data link layer, directly connected nodes are used to perform node-to-node data
transfer where data is packaged into frames. The data link layer also corrects errors that
may have occurred at the physical layer. The data link layer encompasses two sub-layers
of its own. The first, media access control (MAC), provides flow control and
multiplexing for device transmissions over a network. The second, the logical link control
(LLC), provides flow and error control over the physical medium as well as identifies
line protocols. Protect Your Network Layers with Forcepoint NGFW
Network Layer
The network layer is responsible for receiving frames from the data link layer and
delivering them to their intended destinations based on the addresses contained inside the
frame. The network layer finds the destination by using logical addresses, such as IP
(internet protocol). At this layer, routers are a crucial component used to quite literally
route information where it needs to go between networks.
Transport Layer
The transport layer manages the delivery and error-checking of data packets. It regulates
the size, sequencing, and ultimately the transfer of data between systems and hosts. One
of the most common examples of the transport layer is TCP or the Transmission Control
Protocol.
Session Layer
The session layer controls the conversations between different computers. A session or
connection between machines is set up, managed, and terminated at layer 5. Session layer
services also include authentication and reconnections.
Presentation Layer
The presentation layer formats or translates data for the application layer based on the
syntax or semantics that the application accepts. Because of this, it is at times also called
the syntax layer. This layer can also handle the encryption and decryption required by the
application layer.
Application Layer
At this layer, both the end-user and the application layer interact directly with the
software application. This layer sees network services provided to end-user applications
such as a web browser or Office 365. The application layer identifies communication
partners, resource availability, and synchronizes communication.

Physical Layer – Repeaters, Hubs, NICs, Cable & Connectors.

Data Link Layer - Logical Link Control, Media Access Control.
Network Layer – Addressing, Routing.
Transport Layer – Gateways.
Application Layer – Hosts.
 CHAPTER 4

CHALLENGES

4.1 SYSTEM INTEGRATION

System integration is the process of connecting different sub-systems (components) into a

single larger system that functions as one. With regards to software solutions, system
integration is typically defined as the process of linking together various IT systems,
services, and/or software to enable all of them to work functionally together.

System Integration for Machine Vision camera has a lot of challenges to face, like
1. The decision of hardware selection that is compactable to all the four components i.e.
Robot, PLC, Camera, and User Interface.
2. The robot doesn’t support any of the communication protocols which can directly
communicate with PLC & Camera……. Hence, We are in requirement of usage of
Digital I/O Communication.
3. Communication between Node-Red & PLC requires MODBUS TCP/IP.
4. Communication between Node-Red & Camera requires PROFINET.

MODBUS TCP/IP, PROFINET, ETHERNET, COAXIAL CABLE, Rs232, etc are types of
data links, and communication between each of the four components requires a different type
of datalink which is the major challenge for SYSTEM INTEGRATION and approach to OSI
MODEL.
 CHAPTER 5

RESULT AND DISCUSSION

5.1 NODE-RED

 Node-RED is a programming tool for wiring together hardware devices, APIs, and online
services in new and interesting ways. It provides a browser-based editor that makes it
easy to wire together flows using the wide range of nodes in the palette that can be
deployed to its runtime in a single click.
 Node-RED provides a browser-based flow editor that makes it easy to wire together
flows using the wide range of nodes in the palette. Flows can be then deployed to the
runtime in a single click. JavaScript functions can be created within the editor using a
rich text editor. A built-in library allows you to save useful functions, templates, or flows
for re-use.
 The lightweight runtime is built on Node.js, taking full advantage of its event-driven,
non-blocking model. This makes it ideal to run at the edge of the network on low-cost
hardware such as the Raspberry Pi as well as in the cloud. With over 225,000 modules in
Node's package repository, it is easy to extend the range of palette nodes to add new
capabilities.
 The flows created in Node-RED are stored using JSON which can be easily imported and
exported for sharing with others. An online flow library allows you to share your best
flows with the world.
5.2 DASHBOARD RESULTS
Result 1:
Result 2:

Result 3:
 CHAPTER 7

CONCLUSION

1. Did research on different Machine Vision camera systems available other than currently
used in live-project which gave me a handful of information about different cameras and
their specification and provides help with some basic day-to-day life activities too.
2. Machine vision is enabling various industries such as retail, insurance, manufacturing,
etc. to achieve greater customer delight and satisfaction. Technological advances in the
machine vision field are continuing to expand the capabilities across industrial sectors
and applications. This development is manifesting itself as greater accuracy,
repeatability, quality, and efficiencies in manufacturing.
3. Also, introduce a new species of camera, sensors, and robots.
4. With the new technology being used, manufacturing processes can also become fast and
reliable with origami robots.
5. Also, ease human intervention.
 CHAPTER 8

DISCUSSION & FUTURE WORK

I have studied a system that provides a web browser-based flow editor, which can be used
to create JavaScript functions. Elements of applications can be saved or shared for re-use.
The runtime is built on Node.js. The flows created in Node-RED are stored using JSON.
Since version 0.14, MQTT nodes can make properly configured TLS connections.

In the future, it would be interesting to expand the system to incorporate dynamic

simulations and verification, as well as to address a wider variety of tasks. I saw that the
system can provide sufficient feedback for standard planning tasks. However, real-world
robots must be able to interact with a physical environment.

Results demonstrate that a design approach that simultaneously tackles geometry and
motion aspects of a robot design hand-in-hand with User-Interface, can lead to better
performance.
 REFERENCES

1. https://nodered.org/docs/
2. https://nodered.org/docs/user-guide/writing-functions
3. https://nodered.org/
4. https://en.wikipedia.org/wiki/Node-RED
5. https://youtu.be/sdGqM6L9tY0
6. https://youtu.be/_R_hQSl1Lq8
7. https://robotiq.com/products/wrist-camera?
_ga=2.22285118.464791987.1623676429-1084210023.1623676429
8. https://blog.robotiq.com/ruc-challenge-update-its-been-a-challenging-
night