A Survey on Nutrition Monitoring and Dietary
Management Systems
Abstract - A balanced diet with approximate nutrient
intake is vital for infants and children which reduces
the risks of life-threatening diseases such as obesity,
cardiovascular disease and cancer. Unlike adults,
infants require some assistance in their food intake.
The survey provides valuable insights about the
advancements of IoT in healthcare and the need for
nutrition monitoring. Various nutrition monitoring
systems for calorie estimation, prediction have been
developed using machine learning techniques and also
with advanced deep learning based techniques. A
comparative view of the previous works of
researchers in the recent times has been provided.
Keywords – Nutrition Monitoring, Machine Learning,
Neural networks, Deep Learning.
I. INTRODUCTION
Monitoring the daily intake of nutrients and
ensuring a balance is very vital for the growth of
children. Infant child mortality and morbidity, for
children under the age of five is mainly due to
undernutrition. Almost 90% of the children
suffering from undernutrition are from the
developing countries [1]. Fulfilling the balanced
nutrition requirement of the children helps to
reduce malnutrition and reduces the risk of diseases
that are encountered due to lack of nutrition. The
IoT is an enabling technology which consists of
things connected in a network to communicate and
share data. The life of the user becomes
comfortable by adopting the various technologies
based on IoT [2]. The Internet of Things (IoT)
provides a probable solution to suit the demands of
improving the healthcare systems and is also the
main focus of major research works [3-5]. The IoT
has great impact in the healthcare industry and
improves the lives of millions of people all over the
world enabling remote assistance. The IoT has the
capability to safeguard the health of a patient but
also helps the physicians to ensure care [6, 7]. An
analysis was conducted in India on the various
features that affect infant mortality rate in the
Indian states [8].
A. Food Image Databases
A comprehensive collection of quality food images
is key to train a food-classification model and
benchmark the prediction performance on large
food image databases such as Food-101, UEC
Food-100, and UEC Food-256. The current dataset
related to food images have varied aspects namely
the number of food item groups, the total number
of images for a specific food class and also the type
of cuisine [9, 10].
B. Image Based Food Recognition
a. Food image segmentation
Segmentation is the process of decomposition of an
image into a number of non-overlapping
meaningful areas with the similar attributes. When
the complexity and difficulty are considered, the
various segmentation algorithms have been
successful in varied measures. The Clustering
analysis algorithm which has a wide application in
image segmentation, divides the datasets into
different groups in accordance to a particular
standard. The food images are analysed to extract
the food items before the food classification [11-
13].
b. Feature extraction
Feature extraction is a dimensionality reduction
process, where an initial set of raw variables is
reduced to more manageable groups (features) for
processing, while still accurately and completely
describing the original dataset. The image can be
recognized based on characteristics like colour,
shape and the texture [14]. The selection of
relevant features is important when a recognition
model is built for identifying the food images [15].
c. Food classification
Food classification strategies for image recognition
are the machine learning based approach that
consists of features and the deep learning approach.
The Machine learning based approach consists of
feature extraction and use them to construct a
prediction model based on algorithms like Support
Vector Machine (SVM), Bag of Features and K
Nearest Neighbours. In contrast, the emerging deep
learning approach has a large number of connected
layers that learn features, followed by a final layer
responsible for classification. Deep Learning based
approaches have become more popular and
effective.
 d. Food volume estimation and nutrient
analysis
After the food items have been identified, the
nutrient content has to be analysed. At first, the
food candidate regions must be recognized,
segmented, and classified correctly. The volume
from each segmented item will be calculated; and
the nutrient can be estimated based on a nutritional
facts table, such as USDA Food Composition
Database. The volume can either be underestimated
or overestimated with interference from external
factors, such as lighting conditions, blurred images,
and noisy background [16-18]. Only a very few
strategies were reported for estimation of food
volume and calorie intake, as presently the
important focus of this domain still lies in the focus
of food classification [19].
II. REVIEW OF LITERATURE
A. Prabha et al. [20] proposed a Smart Log
system that performs automated nutrition
monitoring and meal prediction. The smart
sensor board consisting of Piezo Electric
sensors is used for nutrition quantification.
The nutrient data acquisition is done using
Optical Character Recognition and by
linking open source Application Program
Interfaces (APIs) through barcodes. The
meal prediction is done by collecting
nutritional value of the leftover food along
with the user’s feedback on the type of
food that is desired. The SR8 database
available through the US Department of
Agriculture website is also analysed using
their API which provides a food report of
associated nutrient values for a particular
food item and a nutrient report which
gives an extensive list of food and their
nutrient values for a selected amount of
nutrients. The results have been analysed
by creating an Attribute-Relation File
Format which inputs the Waikato
Environment for Knowledge Analysis
(WEKA) tool which builds a better
prediction model and is observed that the
Bayesian classifiers provided better
results. The open dataset consisted of
multiple redundant logs and psychological
monitoring mechanisms have not been
incorporated which in turn leads to lack of
accurate prediction.
B. Manal et al. [21] proposed a machine
learning based pipelined approach for
predicting the calories from food images.
The system takes an image of the food
item and passes it through Mathworks
Image Processing which extracts the raw
features and improves the quality. The
image is passed through a compression
phase which helps to reduce the number of
features using the Principal Component
Analysis (PCA) method and scale the
subsequent learning phases. The food type
classification is done by inputting the
compressed image to the classifier. The
food size prediction is done by passing the
compressed image to a regressor. Calories
are predicted by passing the compressed
image and predicted values to another
regressor. This is based on supervised
learning model. The dataset is limited to a
small category and the image cannot be
diversified.
C. Kohila et al. [22] proposed a calorific
value prediction mechanism using image
processing and machine learning. The
image of the food is transmitted through a
mobile device and it initially undergoes
segmentation with Fuzzy C-means
Clustering Segmentation which fixes the
cluster centre based on the group data
unlike the K-means Clustering which can
be erroneous if the cluster centre is not
defined properly by the user. The
mathematical morphology is utilized as a
tool for extracting the image components
and the region shape description such as
erosion, dilation, opening and closing.
Feature extraction is performed to retrieve
interesting parts of the image and then
calorie measurement is done. It has
limited scalability and diversely mixed
food images have not been considered.
D. Sangita et al. [23] proposed a nutritional
status investigation system based on
machine learning. A logical regression
model was considered for the major four
variables namely BMI, HAZ, WHZ and
WAZ individually. The study predicts the
nutrition state of the child in two phases.
Phase I pre-processes the dataset using
SMOTE Resampling method. Feature
extraction is done using machine learning
techniques with Entropy based Gain Ratio
concept. Phase II uses Nominal logistic
regression using iterative reweight least
squares algorithm which predicts the
characteristic features. The dataset is
based on a very specific geographical area
 and over a particular period of time and it phone camera captures the FDA-mandated
also considers only the basic features. Nutritional Facts Label and the other
method is by linking open source
E. Oscar et al. [24] proposed a menu-match: Application Program Interfaces (APIs)
restaurant-specific food logging from through barcodes. The meal prediction is
images. An image recognition framework done by collecting nutritional value of the
based on the bag of visual words approach leftover food along with the user’s
which extracts the base features from the feedback on the purpose of the meal. Food
images and then encoded with locality- Classification is done through an
constrained linear coding (LLC). The algorithm which is built on a Bayesian or
extracted features are pooled using max- Belief Network (BN) with constraint
pooling in a rotation-invariant pooling based method. The algorithms used are
scheme. A regression based method Nutritional Balance Network Algorithm
estimates the calories and along with for building a multi-layer perceptron
feature representation mapped the feature neural network using Stochastic Gradient
space to calories using Support Vector Descent and Food classification algorithm
Regression. The approach is limited for for building Bayesian Network using hill-
discrete serving sizes and custom menu climbing search. The system is designed
and is also dependent on the GPSS of food only for the children and the dataset
consumption. The system lacks user contained multiple redundant logs.
customization and requires cost-sensitive
learning to directly minimize calorie H. Parisa et al. [27] proposed a method for
estimation errors during the training. food calorie measurement using deep
learning neural network. The system uses
F. Kiran et al. [25] proposed a method for food images and the features are extracted
measuring the calories and nutrition from using Graph Cut Segmentation. A pre-
food images using machine learning trained model file is generated with the
techniques. The images got from the help of Convolutional Neural Network
mobile device are pre-processed followed (CNN). It is loaded into the application
by the segmentation step to extract the and tested to perform image recognition
colour and texture features through K process. The stochastic gradient descent
Means clustering. The extracted options algorithm is used to get an improvement
are used for food classification using in the cost. The calorie measurement is
Support Vector Machine (SVM). The food done by two approaches namely finger
portion volume measurement is done by based calorie measurement and calorie
superimposing a grid of squares onto the measurement using distance estimation.
image segment which matches the The system lacked analysing mixed food
irregular shape of the food images easily. portion and only limited samples were
The calorie measurement is done based on considered.
the food mass and nutritional tables. The
system has limited cuisine varieties mixed III. COMPARISON OF RESEARCH WORKS
food images have not been considered. TABLE 1: COMPARISON OF THE RESEARCH
WORKS
G. Prabha et al. [26] proposed a Smart Log
system based on deep learning for Title Approach Advantag Limitation
automated nutrition monitoring system in es
IoT. The system can be considered as a Prabh Machine Automate Psychological
product which includes a smart sensor a et learning d nutrition monitoring
board with a food weighing sensor and a al. based monitorin mechanism is
smart phone application. Automatic [20] prediction g and not
using OCR meal incorporated
Nutrition Quantification is the first step
and APIs prediction
which collects the quantity of nutrient
Mana Machine The The dataset is
values along with the timestamp and l et al. learning pipelined limited to a
transmits it to the cloud. The nutrient data [21] based approach small category
acquisition is done using Optical pipelined is very and lacks
Character Recognition in which the on-
approach effective realistic Kohil Machine The Lacks mobility
using PCA when scenario a et learning approach and there is a
compared al. based Fuzzy has deviation in
to the [22] c-means efficient calorific value
baseline clustering feature between the
for extraction observed and
segmentatio mechanis calculated
n and m values
Morphologi
cal operation
for
extracting
image
components.
Sangi Machine Explored The data is
ta et learning the limited to a
al. with possibiliti particular time,
[23] Entropy es of context and
based Gain using only limited
Ratio artificial features are
concept and intelligenc considered.
Nominal e in
logistic identifyin
regression g probable
using correlates
iterative of
reweight malnutriti
least squares on.
algorithm
Oscar Machine Automate It lacks user
et al. learning d customization
[24] based SVM computer and is GPSS
classifier vision dependent
and LLC system for
logging
food and
robust
calorie
estimation
Kiran Machine IMg2 Has limited
et al. learning calories cuisine
[25] based K app that varieties and
Means determine mixed food
clustering s the images have
and SVM carlorie not been
intake and considered.
estimation
.
Prabh Deep The Psychological
a et Learning prediction monitoring
al. based novel accuracy mechanism is
[26] 5-layer of Smart- not
perceptron Log is incorporated
neural 98.6%.
network and
a Bayesian
Network
based
prediction
algorithm
Parisa Deep The The database [6] Catarina I.Reis,Marisa da silva Maximiano,
et al. Learning accuracy is limited and “Internet of Things and Advanced Application in
[27] based for single mixed food Healthcare”, Oct 2016, ISBN13: 9781522518204.
Convolution food portions have
al neural portions is not been [7] P. Sundaravadivel, E. Kougianos, S. P.
networks 99% considered. Mohanty, and M. Ganapathiraju, “Everything You
with Wanted to Know about Smart Healthcare,” IEEE
virtualizatio Consum. Electron. Mag., vol. 8, no. 1, pp. 18–28,
n January 2018.

[8] Suriyakala Vijayakumar, Deepika M G,

IV. CONCLUSION
The Internet of Things is a booming technology
that spans across various domains and its
incorporation with healthcare is increasing day to
day. This survey presents a comparative view on
the research works in the nutrition monitoring,
calorie estimation and dietary systems. The
machine learning techniques are being replaced by
the deep learning techniques that outperforms the
traditional methodologies. There is also an increase
in the integration of wearable sensors along with
smartphone through mobile technologies which
will be a revolution in food monitoring systems
which helps in disease prediction and prevention.
REFERNCES
[1] Markos Z, Doyore F, Yifiru M, Haidar
J,“Predicting Undernutrition Status of Under-Five
Children Using Data Mining Techniques: The Case
of 2011 Ethiopian Demographic and Health
Survey”, Journal Health Medical Information, 5:
152, 2014.
[2] Sajjad Hussain Shah, Ilyas Yaqoob, “A Survey:
Internet of Things (IOT) Technologies,
Applications and Challenges”, 4th IEEE
International Conference on Smart Energy Grid
Engineering, 2016.
[3] P. Gope and T. Hwang, “BSN-Care: A Secure
IoT Based Modern Healthcare System Using Body
Sensor Network,” IEEE Sensors Journal, vol. 16,
no. 5, pp. 1368–1376, 2016.
[4] N. Zhu, T. Diethe, M. Camplani, L. Tao, A.
Burrows, N. Twomey, D. Kaleshi, M. Mirmehdi, P.
Flach, and I. Craddock, “Bridging e-Health and the
Internet of Things: The SPHERE Project,” IEEE
Intelligent Systems, vol. 30, no. 4, pp. 39–46, 2015.
[5] S. H. Chang, R. D. Chiang, S. J. Wu, and W. T.
Chang, “A Context-Aware, Interactive M-Health
System for Diabetics,” IT Professional, vol. 18, no.
3, pp. 14–22, 2016.
Amalendu Jyotishi and Deepa Gupta, “Factors
Affecting Infant Mortality Rate in India: An
Analysis of Indian States”, International
Symposium on Intelligent Systems Technologies
and Applications, ISTA; Advances in Intelligent
Systems and Computing, Volume 530, Pages 707-
719, Jaipur; India. Publisher Springer, 2016.
[9] Ciocca G, Napoletano P, Schettini R., “Food
recognition: a new dataset, experiments and
results”, IEEE Journal Biomed Health Informatics,
vol.99, pp. 1-1, 2016.
[10] Matsuda Y, Hoashi H, Yanai K., “Recognition
of multiple-food images by detecting candidate
regions”, IEEE International Conference on
Multimedia Expo vol.25, pp. 25-30, 2012.
[11] Xin Zheng, Qinyi Lei, Run Yao, Yifei Gong
and Qian Yin, “Image Segmentation based on
adaptive K-means algorithm”, EURASIP Journal
on Image and Video Processing, Publisher
Springer, 3 August 2018.
[12] Hamirul Binti Hambali, Salwa Khalid
Abdulateef and Massudi Mahmuddin,
“Segmentation of Multi Food Images Using
Integrated Active Contour and K-means”, Journal
of Engineering and Applied Sciences, pp. 3146-
3151, 2018.
[13] Yu Wang, Fengqing Zhu, Carol J. Boushey
and Edward J. Delp, “Weakly Supervised Food
Image Segmentation Using Class Activation
Maps”, Proceedings of IEEE International
Conference on Image Processing, pp. 1277-1281,
2017.
[14] Hong Liang, Yuan Gao, Yunlei Sun and Xiao
Sun, “CEP: Calories Estimation from food photos”,
International Journal of Computers and
Applications, 25 July 2018.
[15] Hamid Hassannejad, Guido Matrella, Paolo
Ciampoini, Ilaria De Munari, Monica Mordoini,
Stefano Cagnoni, “Food Image Recognition Using
very Deep Convolutional Networks”, Proceedings
of the 2nd International Workshop on Multimedia
Assisted Dietary Management, Amsterdam, The
Netherlands, 16 October 2016.
[16] Jia W., Li Y., Qu R., Baranowski, T., Burke,
L., Zhang, H., Sun, M., “Automatic food detection
in egocentric images using artificial intelligence
technology”, Public Health Nutrition, pp. 1-12,
2018 doi:10.1017/S1368980018000538.
[17] Sirichai Turmchokkasam, Kosin
Chamnongthai, “The Design and Implementation
of an Ingredient-Based Food Calorie Estimation
System Using Nutrition Knowledge and Fusion of
Brightness and Heat Information”, IEEE Access,
vol. 6, pp. 46863 – 46876, 15 May 2018.
[18] İbrahim Berkan Aydilek, “Approximate
estimation of the nutrition of consumed food by
deep learning”, Proceedings of IEEE International
Conference on Computer Science and Engineering
(UBMK), 02 November 2017.
[19] Silva BVR, Cui J., “A Survey on Automated
Food Monitoring and Dietary Management
Systems”, J Health Med Informat 8:272, 2017, doi:
10.4172/2157-7420.1000272
[20] Prabha Sundaravadivel, Kavya Kesavan,
Lokeshwar Kesavan, Saraju P. Mohanty, Elias
Kougianos, Madhavi Ganapathiraju, “Smart-log:
An automated, predictive nutrition monitoring
system for infants through the IoT”, IEEE
International Conference on Consumer Electronics
(ICCE), 2018.
[21] Manal Chokr, Shady Elbassuoni, “Calories
Prediction from Food Images”, Innovative
Applications of Artificial Intelligence Twenty -
Ninth IAAI Conference, 2017.
[22] Kohila R., Meenakumari R., “Predicting
calorific value for mixed food using image
processing”, In Proceedings of the 2017
International Conference on Innovations in
Information, Embedded and Communication
Systems (ICIIECS), Coimbatore, India, pp. 1–4,
17–18 March 2017.
[23] Sangita Khare, Kavyashree, S., Gupta, D.,
Jyotishi, A., “Investigation of nutritional status of
children based on machine learning techniques
using Indian demographic and health survey data”,
In: 7th International Conference on Advances in
Computing and Communications, vol. 115, pp.
338–349. Elsevier Procedia Computer Science
2017. https://doi.org/10.1016/j.procs.2017.09.087
[24] Oscar Beijbom , Neel Joshi , Dan Morris ,
Scott Saponas , Siddharth Khullar, “Menu-Match:
Restaurant-Specific Food Logging from Images”,
Proceedings of the 2015 IEEE Winter Conference
on Applications of Computer Vision, pp. 844-851,
05 - 09 January, 2015.
[doi>10.1109/WACV.2015.117]
[25] Kiran Abhore, Dawande N. A., “Measuring
Calories and Nutrition from Food Image”,
International Journal of Advanced Research in
Computer and Communication Engineering, vol. 5,
Issue 6, June 2016.
[26] Prabha Sundaravadivel, Kavya Kesavan,
Lokeshwar Kesavan, Saraju P. Mohanty, and Elias
Kougianos, “Smart-Log: A Deep-Learning based
Automated Nutrition Monitoring System in the
IoT”, IEEE Transactions on Consumer Electronics,
2018. DOI 10.1109/TCE.2018.2867802
[27] Parisa Pouladzadeh, P. Kuhad, S. V. B. Peddi,
A. Yassine, S. Shirmohammadi, "Food calorie
measurement using deep learning neural network",
Proc. IEEE Int. Instrum. Meas. Technol. Conf.
Proc., pp. 1-6, 2016.