MOBILE-BASED AUTOMATED OBJECT COUNTING

FOR INVENTORY MANAGEMENT

MUHAMMAD ADIB HARIRI BIN ABDUL HADI

FACULTY OF COMPUTER SCIENCE AND

INFORMATION TECHNOLOGY
UNIVERSITY OF MALAYA
KUALA LUMPUR

2022
 TITLE OF RESEARCH REPORT
WHICH HAS BEEN APPROVED
BY THE FACULTY

MUHAMMAD ADIB HARIRI BIN ABDUL HADI

THESIS SUBMITTED IN FULFILMENT OF THE REQUIREMENTS

FOR THE DEGREE OF COMPUTER SYSTEM AND NETWORKING

FACULTY OF COMPUTER SCIENCE AND

INFORMATION TECHNOLOGY
UNIVERSITY OF MALAYA
KUALA LUMPUR

2022
 UNIVERSITY OF MALAYA
ORIGINAL LITERARY WORK DECLARATION

Name of Candidate: Muhammad Adib Hariri bin Abdul Hadi

(I.C/Passport No: REDACTED)
Matric No: 17100026/1
Name of Degree: Degree in Computer System and Networking
Title of Project Paper/Research Report/Dissertation/Thesis (“this Work”): Mobile-
Based Automated Object Counting for Inventory Management

Field of Study:

I do solemnly and sincerely declare that:

(1) I am the sole author/writer of this Work;
(2) This Work is original;
(3) Any use of any work in which copyright exists was done by way of fair dealing
and for permitted purposes and any excerpt or extract from, or reference to or
reproduction of any copyright work has been disclosed expressly and
sufficiently and the title of the Work and its authorship have been
acknowledged in this Work;
(4) I do not have any actual knowledge nor do I ought reasonably to know that the
making of this work constitutes an infringement of any copyright work;
(5) I hereby assign all and every rights in the copyright to this Work to the
University of Malaya (“UM”), who henceforth shall be owner of the copyright
in this Work and that any reproduction or use in any form or by any means
whatsoever is prohibited without the written consent of UM having been first
had and obtained;
(6) I am fully aware that if in the course of making this Work I have infringed any
copyright whether intentionally or otherwise, I may be subject to legal action
or any other action as may be determined by UM.

Candidate’s Signature Date: 14 February 2022

Subscribed and solemnly declared before,

Witness’s Signature Date:

Name:
Designation:
 MOBILE-BASED AUTOMATED OBJECT COUNTING FOR INVENTORY
MANAGEMENT

ABSTRACT

Edge detection algorithms can detect objects by detecting the continuity of their edges.
This project successfully developed a mobile app to count objects in an image by
detecting such continuity. The app can count most, if not all, objects in an image selected
by the user. The image is processed in the background, which takes place outside the
app’s environment before returning the quantity to the app. Sellers, in general, have to
manage and track objects by counting them on an interval. This project aims to investigate
the existing edge detection methods for object counting. The app is developed using
Flutter, which consists of three modules; View Products (to retrieve the list of products),
Count Products (to count the objects using edge detection) and Inventory Report (to
record the objects counted twice). OpenCV provides edge detection algorithms. It has
extensive libraries in image processing. The processing takes place in individual pixels;
varying noise levels might significantly affect the accuracy of the processing. The
counting algorithm has many image processing techniques, including Canny edge
detection. The latter reduces noise in an image and detects the continuity along the edges
of every object. Image dilation is then performed to distinguish the objects from the
background. The accuracy of counting those objects is affected by contour detection of
the outer of the objects. The quantity of the objects is recorded and stored into the app. In
conclusion, edge detection successfully counts every object in an image through the
mobile app. However, the project does not include an object recognition component.
Therefore, the algorithm does not distinguish different objects and count them separately.

Keywords: counting, edge detection, Flutter, OpenCV

 ACKNOWLEDGEMENTS

I want to express my gratitude to Allah SWT and countless thank you to Dr.
Roziana Ramli for never giving up for providing me with guidance and improvisation
throughout the project. I want to thank Dr. Ang and Dr. Ismail for my panels during the
viva to review my presentation for my project. I want to say thank you to Econsave Cash
and Carry for their interest in being prospective stakeholders. I also would like to thank
my family because I would not be where I am today without the support from parents and
siblings.
Contents

ABSTRACT ................................................................................................................. ii

ACKNOWLEDGEMENTS ........................................................................................ iii

CHAPTER 1: SYSTEM ANALYSIS AND DESIGN ................................................. 1

1.1: INTERFACE DESIGN ..................................................................................... 3

1.1.1: STORYBOARD ......................................................................................... 3

1.1.2: MOCKUPS................................................................................................. 4

1.2: SYSTEM DESIGN ......................................................................................... 12

1.3: UML DIAGRAM ............................................................................................ 15

CHAPTER 2: SYSTEM DEVELOPMENT .............................................................. 16

2.1: TECHNICAL IMPLEMENTATION ............................................................. 16

2.2: SOURCE CODE ............................................................................................. 17

2.2.1: VIEW PRODUCTS MODULE ............................................................... 21

2.2.2: COUNT PRODUCTS MODULE ............................................................ 30

2.2.3: INVENTORY REPORT MODULE ........................................................ 45

2.3: USER INTERFACE PROBLEMS.................................................................. 48

CHAPTER 3: SYSTEM EVALUATION .................................................................. 49

3.1: OBJECT COUNTING ALGORITHM ........................................................... 49

3.2: SYSTEM ERROR HANDLING ..................................................................... 51

CONCLUSION .......................................................................................................... 55

REFERENCES ........................................................................................................... 56
 LIST OF FIGURES

Figure 1-1: How the user should navigate the app.1 .................................................... 3

Figure 1-2: Home Page................................................................................................. 4

Figure 1-3: View Products Page ................................................................................... 5

Figure 1-4: Add a Product Page ................................................................................... 6

Figure 1-5: Counting Products Page ............................................................................ 7

Figure 1-6: Counting Products - Page 2 ....................................................................... 8

Figure 1-7: Counting Products - Page 2 Page with Date, Shift and Product selected. 9

Figure 1-8: Select a date to view Inventory Report.................................................... 10

Figure 1-9: Inventory Report is displayed according to the date selected by the user.
......................................................................................................................................... 11

Figure 1-10: Flutter interacts with Firebase to store data, and Flask which contains the
OpenCV counting algorithm through ngrok. The quantity returns to Flutter in
JSONified form. ......................................................................................................... 12

Figure 1-11: The UML diagram of the app. ............................................................... 15

Figure 2-1: The hierarchial structure of the app. ........................................................ 16

Figure 3-1: The objects in an image counted by OpenCV. ........................................ 49

Figure 3-2: The app retrieves the quantity from OpenCV. ........................................ 50

Figure 3-3: Error handling in adding a product. ......................................................... 52

Figure 3-4: Error handling when the wrong shift is selected. .................................... 53

Figure 3-5: When one of the products does not have the closing shift's quantity. ..... 54
 CHAPTER 1: SYSTEM ANALYSIS AND DESIGN

Every functional and non-functional requirement has been fulfilled throughout the
development of the project.
The functional requirements are as follows:
1. A user taps on the ‘View Products’ module to enter the product information. The
information is retrieved from the ‘product’ table in Firebase Firestore, sorted
alphabetically. If the said product is not available, a Floating Action Button redirects the
user to the product form. The user adds product details such as image (from camera or
gallery), brand, item name and weight. However, the user must specify the weight in
imperial units. Failure to include one of the units in the ‘Weight’ textbox field results in
an error message.
2. After the user successfully adds the product information, it is stored in the
‘product’ table in the Firebase Firestore in terms of brand, item name, weight and image
URL. The latter is retrieved by uploading the product image to Firebase Storage, which
generates the image URL for the product.
3. In the ‘Count Products’ module, the user first selects a date, opening or closing
shift, and a product to be counted. One counting process is limited to the particular
product on the particular shift for the particular day only. To count a particular product
for a particular day, the user must count the same product for both shifts.
4. The OpenCV algorithm can count every object in the image. First, OpenCV
resizes the image to 800 x 800 pixels. Next, it converts the image to grayscale. Canny
edge detection is performed on the grayscaled image to detect continuity along the edges
of every object. The minimum and maximum threshold values are 30 and 150,
respectively. Then, an image dilation is performed to remove noise. After that, the
algorithm finds contour shapes in the dilated image with cv2.RETR_EXTERNAL and
cv2.CHAIN_APPROX_SIMPLE as parameters. The former extracts extreme outer
contours, while the latter compresses horizontal, vertical, and diagonal segments and
leaves only their endpoints. Finally, the number of counters detected is treated as the
number of objects counted by the algorithm.
5. The user needs to select the shift before starting counting the product carefully.
There are two textboxes for both opening and closing shifts in the' Count Products'
module. However, when the user taps on 'Add to Inventory', it only adds the quantity in
1
the textbox based on the particular shift selected by the user. Attempting to add the
quantity for the closing shift when the corresponding opening shift is not available results
in error.
6. The OpenCV algorithm is far from perfect; the image needs to be free from noise
by placing those objects white background so that only those whole objects are counted.
However, the user is free to edit the quantity detected in the textbox of the ‘Count
Products’ module based on the shift specified by the user.
7. In the ‘Inventory Report’ module, the user can view the quantities for both
opening and closing shifts of a particular object based on the date specified by the user.
The non-functional requirements are as follows:
1. The app requires the user to register before using the app. The user cannot edit the
username upon successful registration.
2. In the ‘Count Products’ module, the user cannot add the same product information
twice. The system detects a duplicate in terms of brand, item name, and weight altogether
and displays an error message.
3. The user must place similar products on a white background to count products
accurately. Including non-related products in the background is discouraged, as the
system cannot distinguish individual objects intelligently.
4. In the ‘Inventory Report’ module, once the quantity of an object for the particular
shift on a particular day has been recorded, the user cannot edit it.
1.1: INTERFACE DESIGN

1.1.1: STORYBOARD

1) Figure 1-1: How the user should navigate the app.1

1.1.2: MOCKUPS

Figure 1-2: Home Page

Figure 1-3: View Products Page
Figure 1-4: Add a Product Page
Figure 1-5: Counting Products Page
Figure 1-6: Counting Products - Page 2
Figure 1-7: Counting Products - Page 2 Page with Date, Shift and Product selected.
Figure 1-8: Select a date to view Inventory Report
Figure 1-9: Inventory Report is displayed according to the date selected by the user.
1.2: SYSTEM DESIGN

Figure 1-10: Flutter interacts with Firebase to store data, and Flask which contains the OpenCV counting algorithm through ngrok. The quantity returns to Flutter in JSONified
form.
 Firebase stores user, product, and inventory information in its Firestore Database.
Firestore is a flexible, scalable mobile development database that syncs data across client
apps through real-time listeners. User information consists of employee ID, full name and
password. Product information consists of a brand, image URL, item name and weight.
Inventory information consists of date, product name, opening quantity and closing
quantity.

The ’Automated Inventory Counting’ app is developed using Flutter. It consists

of three modules; ‘View and Add Products’, ‘Count Products’ and ‘Inventory Report’.
On the ‘View Products’ page, the list of products is sorted alphabetically in terms of the
product name. The user can delete and view each product image. On the ‘Add a Product’
page, the user can add product information such as brand, image, item name, and weight.
The weight text box validates if the imperial weight unit (g, mg, ml, kg, l) is specified.
The Count Products module lets the user select the date, the shift (opening or closing),
and the product to be counted. The user can input the product quantity manually or let the
edge detection algorithm automatically count it on the next page. When adding the
product quantity to the inventory, only the quantity in one of the shifts is recorded. If it is
for the opening shift, the user must repeat the closing shift process. The ‘Inventory
Report’ module consists of inventory information of every product recorded for both
shifts for the particular date.

ngrok is an API that connects the ‘Count Products’ module with the Flask
framework. It converts a local machine host to an HTTP link that the communication can
take place. When Flutter uploads an image the user selects, it goes through the HTTP link
generated by ngrok to the Flask framework.

Flask is a micro web framework written in Python programming language, which

provides a medium for the OpenCV to execute.

OpenCV is an open-source library that processes the image retrieved from Flutter.
OpenCV first reads the image, resizes it to 800 by 800 pixels, then converts it into
grayscale. The Canny edge detection performs on the grayscaled image, with minimum
and maximum threshold values are set to 30 and 150, respectively. After that, OpenCV
dilates the said image to remove black distortion such as labels on the bottles. The
algorithm then finds contour shapes of every item it detects in the dilated image. Finally,
the total number of objects is defined by the number of contours drawn around an object.
When the operation finishes, it launches two windows, ‘Dilate’ and ‘Contours’, which
display the properties of the items detected in the image before returning the total to the
front-end.

The total number of items is generated in JSON format, which is readable by

Flutter and is displayed in the ‘Count Products’ module as a number, presenting the
quantity of the items detected in one of the text fields, depending on the shift specified by
the user.
1.3: UML DIAGRAM

Figure 1-11: The UML diagram of the app.

CHAPTER 2: SYSTEM DEVELOPMENT

2.1: TECHNICAL IMPLEMENTATION

Figure 2-1: The hierarchial structure of the app.

 2.2: SOURCE CODE

main.dart

The “main.dart” file is where the program starts, which is considered the “entry
point” in the program.
home_screen.dart
 The “home_screen.dart” consists of three buttons; which covers the three
modules respectively.
2.2.1: VIEW PRODUCTS MODULE

view_products.dart
Output: List of every product sorted alphabetically, along with its brand and weight. Users
can delete the product details and view the product image.

add_products.dart
Input: Product brand (string), image (image URL), item name (string) and weight (string).

Output: Product details in the “View Products” module.

2.2.2: COUNT PRODUCTS MODULE

count_products.dart
Input: Date, product image (any image formats), product name (string) and shift
(opening/closing) (string).
count_products_buffer.dart
Input: Product image

Output: Opening and closing quantity of the product counted in the image.
Ngrok

Input: ngrok http 4000

Output: Tunneling link to be pasted on “count_products_buffer.dart”.

OpenCV
Input: product image

Output: number of contours; interpreted as number of objects in the image

2.2.3: INVENTORY REPORT MODULE

inventory_report.dart
Input: Date

Output: Product name, closing shift quantity and opening shift quantity
2.3: USER INTERFACE PROBLEMS

In the Home Page of the app, the buttons placement is not centered and small. The
real estate is not utilized to maximize the placement of these three buttons.

In the ‘Count Products’ module, the navigation will not go through if the user has
not selected a date, a shift, and a product to be counted. However, an error dialog box
should be displayed to notify the user. Another one is the lack of circular progress button
while the OpenCV algorithm is processing the image. Some users might not aware of the
counting takes place and just skips the process entirely.

In the ‘Inventory Report’ module, the page do not display the date selected by the
user. In addition, the user might receive an ambiguous error message if one of the products
does not have the quantity for the closing shift.
CHAPTER 3: SYSTEM EVALUATION

3.1: OBJECT COUNTING ALGORITHM

Figure 3-1: The objects in an image counted by OpenCV.

Figure 3-2: The app retrieves the quantity from OpenCV.
3.2: SYSTEM ERROR HANDLING

In the ‘View Products’ page, every product information is displayed

alphabetically in terms of product name, along with its brand and weight. At the right
side, there is a delete icon and the view image icon for every product. The delete icon do
not have a dialog box, prompting the user confirm before proceeding with the deletion.
The image icon might display an error if the user did not include an image for a product.

In the ‘Add a Product’ page, the ‘Weight’ field box checks the entry of a specific
weight unit. The absence of it returns an error.

In the ‘Counting Products – Page 2’ page, if the user accidentally selects closing
shift instead of opening, an error message is displayed because Flutter could not add a
closing quantity when the corresponding opening quantity does not exist.

In the ‘Inventory Report’ page, the date is not displayed on top of the inventory
report. Users have to make a mental note after they select the date. Furthermore, if one of
the products does not have the closing quantity, an error message gets displayed on the
bottom of the screen.
Figure 3-3: Error handling in adding a product.
Figure 3-4: Error handling when the wrong shift is selected.
Figure 3-5: When one of the products does not have the closing shift's quantity.
CONCLUSION

In this project, manual counting is made automatic through edge detection

algorithms. Proper image segmentation techniques are enforced when the image selected
has the noise, and distraction level kept to a minimum. Canny edge detection acts on the
grayscaled image; then, image dilation is performed to draw contours perfectly around
the edges of every object in the image. The mobile app can count things automatically
through edge detection algorithms. The main limitation of the algorithm is it cannot count
objects that are close to each other or objects in a crowded scene accurately; it would
count those separate objects as one instead. The project could be improved by deploying
YOLO, which utilizes deep learning to detect individual objects and count them
separately. It is far more accurate than the edge detection algorithm.
REFERENCES

Choudhury, S. R. (2020, December 14). More people are doing their holiday shopping
online and this trend is here to stay. CNBC.

https://www.cnbc.com/2020/12/15/coronavirus- pandemic-has-pushedshoppers-

to-e-commerce-sites.html.

Digital Commerce 360 Staff. (2021, June 16). Coronavirus adds $105 billion to US

ecommerce in 2020. Digital Commerce 360.

https://www.digitalcommerce360.com/article/coronavirus-impact-online-retail/.

Fleisch, E., & Tellkamp, C. (2005). Inventory inaccuracy and supply chain

performance: a simulation study of a retail supply chain. International Journal

of Production Economics, 95(3), 373–385.

https://doi.org/10.1016/j.ijpe.2004.02.003

Importance of Inventory Management. Fishbowl. (n.d.).

https://www.fishbowlinventory.com/articles/inventory-
management/importanceof-

inventory-management.

Melanie. (2020, January 7). 4 Causes of Inventory Discrepancy in Your Stocktaking

(And What You Can Do About It). Unleashed Software.

https://www.unleashedsoftware.com/blog/causes-inventory-
discrepancystocktaking.

Muthukrishnan, R., & Radha, M. (2011). EDGE DETECTION TECHNIQUES

FOR IMAGE SEGMENTATION. International Journal of Computer Science

& Information Technology (IJCSIT), 3(6), 259–267.

https://doi.org/10.5121/ijcsit.2011.3620

P. Loke, J. Paranjpe, S. Bhabal and K. Kanere, "Indian sign language converter system

using an android app," 2017 International conference of Electronics,

 Communication and Aerospace Technology (ICECA), 2017, pp. 436-439, doi:

10.1109/ICECA.2017.8212852.

Saisha. (2020, September 6). Finding the Edge: Canny and Sobel Edge Detectors (Part

1). Medium. https://medium.com/srm-mic/finding-the-edge-canny-andsobel-

detectors- part-1-65a59b7ef62a

Sugata, T & Yang, C. (2017). Leaf App: Leaf recognition with deep convolutional

neural networks. IOP Conference Series: Materials Science and Engineering.

273. 012004. 10.1088/1757-899X/245/1/012004.