Pharmaceutical

Pill Recognition Using Computer Vision Techniques

Charlene Tay and Mridul Birla
School of Informatics and Computing, Indiana University Bloomington

Introduction Marker/Imprint Extraction Results of Shape Determination

What if you could take a photo of an unknown pill and almost immediately Accuracy of Neural Network
find out what it is? 80% 75%
70.66%
70%
• Unidentified and misidentified prescription pills present 60% 58%
55%
challenges for patients and professionals. Taking such pills can
50%
result in adverse drug reactions that affect health or could even Original Image Image after applying erosion technique Output after s ubtracting erosion
output image from original image 40%
cause death. By coming up with ways to easily identify and verify 30%
prescription pills, errors can be greatly reduced. • The most distinguishing factor for pill identification is the markings or 30% 25%

imprints on the tablets or capsules. To extract these features, we used 20% 15%
• Our goal is to produce a framework or set of methods that will a few morphological image processing operations to enhance the 10%
10% 10%

take a consumer-quality image of a pill (taken from mobile imprints. 0%

phones), and help to identify it by returning the most likely • Following that, we used open-sourced Object Character Recognition Capsule Circle Oblong/Oval/Football

matches from our database set of pill images. software to try to extract the numbers and words imprinted.
Pentagon
Square
Fat Oval
Hexagon
Triangle
Special (all other shapes)

• Scale-Invariant Feature Transform (SIFT) descriptors of each pill were

Dataset also extracted to try to capture a representation of the image. • When given the consumer quality images, the neural
The National Library of Medicine has made public a set of pill images that include: network gives an overall of 50.7% accuracy.

• 2000 JPEG high-resolution reference images (one for the front and one for the “ 3228 25mg “ Precision of Different Classifiers

back of each of 1000 pills) from the Computational Photography for Pill 9 0 .0 0 %

8 0 .0 0 %
81% 80%

Identification Project. 74% 75%

7 0 .0 0 %

• 5000 JPEG consumer quality images of the same 1000 pills, taken with digital
6 0 .0 0 % 55%

Shape Determination 5 0 .0 0 %
5 0 .6 8 %

cameras in varying light conditions 4 0 .0 0 %

3 0 .0 0 %
• Each image is converted to 2 0 .0 0 %

greyscale and background is 1 0 .0 0 %

subtracted to get an idea of 0 .0 0 %

Ne u ra l Ne two rk SVM KNN De c i s i o n Tre e Ra n d o m Fo rest Ad a b o o st

overall shape.
Two examples of consumer-quality pill images The corres ponding reference image matches for the
two pills s hown on the left
Challenges and Future Work
Overview of Process • Twelve shape features are
• Limited amount of data - only a few images for
also extracted for each image,
each specific pill. Need to gather more images
including: the 7 Hu invariant
from other resources.
moments, circularity degree,
• Lack of diversity in shape (mostly circle or oblong)
rectangle degree, sphericity • Imprints challenging and requires more research.
degree, concavity degree and
• Color hard to match due to differing light
flat degree.
conditions and camera lenses.
• Create wrapper to classify and suggest best
• We categorized and labeled
matches.
Marker/Imprint Extraction Shape Classification each pill in our training
dataset by shape and trained
a neural network on these References and Acknowledgements
images and features L u , W. (2 0 1 2 ). Me th od o f r Ima ge Sha pe Re co gn ti i o n wi th Ne ura l Ne two rk. In Adv an ce s i n Co mp u ter Sci e nc e a nd In forma ti o n En gi ne e rin g (p p.
5 4 7 -5 5 1 ). Sp rin g er Be lri n He i de b
l e rg.
Cu n h a , A., Ad ã o, T., & Trig ue i ros , P. (20 1 4). He pl me Pil l s: a mo b i le pi l re co gn it i o n too l fo r el de rl y p ers on s. Proc ed i a Te ch no o
l g y, 16 , 15 23-1532.
Us h i z i ma , D., Ca rne i ro, A., So uz a, M., & Me de i ro s , F. (20 15 ). In ve sti g ati ng Pil l Re co g ni tio n Me th od s fo r a Ne w Nati on a l L ib rary of Me d ic in e
Ima g e Da ta s e t. In Adv an ce s i n Vi su al Co mp uti ng (pp . 41 0-4 1 9). Spri n ge r n I tern a it o na l Pu bl i sh n
i g.