210212022 02:11 ‘Acm-Powared Driver Distraction Dataction - CodeProject 009) CODE PROJECT Arm-Powered Driver Distraction Detection % Sergey L. Gladkiy 7 Feb 2022 CPOL How to create a driver distraction detector and how to run it on an Arm-powered device like a Raspberty Pi or Jetson Nano In this article, we demonstrate haw straightforward itis to develop an Al computer vision application for a portable Arm-powered device. We choose this solution for practicality since our driver distraction detection system must function autonomously in a driving car. We show that the application can run on the Arm-pawered device in real-time mode, reaching about two FPS processing speed, ‘This article is a sponsored article, Articles such as these are intended to provide you with information on products and services that we consider useful and of value to developers, Download source - 6.2 KB Download project files - 4.5 MB Introduction When discussing computer vison (CV) based on artificial intelligence (Al) or deep learning (DU, we often imagine a powerful dlesktop machine or server processing images or videos. But sometimes, we need to run a complicated CV algorithm on a portable device. For example, to create a computer system to prevent a driver from becoming distracted, the most practical solution is a stand-alone device with specialized software, A driver, fleet manager, or manufacturer could then place such a device into a vehicle to alert the driver when they are likely distracted. So, can we run a complicated algorithm on portable Arm-powered devices? In this article, we will demonstrate how to create a distracted driver detector and show haw to run it on a Raspberry Pi device. We will use Python to develop our program, OpenCV for the computer vision algorithms, and convolutional neural networks (CNN) to detect possible driver distraction. Inventing the Algorithm ‘We will use a simple detection type that checks if eyes are closed for short intervals. We could characterize many other distraction symptoms, but this one is likely the most reliable, Modern Al algorithms can complete this task with minimal effort. One approach uses a special CNN for detecting so-called facial landmarks. The image below features an extremely common 68-point facial landmarks diagram. hitpsslwwu.codeproject.com/Antcles/5324279/Am-Powered-Driver-Dstraction-Detectiondlsplay=Print ana210212022 02:11 ‘Asm-Powered Driver Distraction Detection - CodeProject 17:18 19 20 21 22.23 24 25 26 na —— 37, 8 43 44 40 27 a7 46 28 29 30 31 32 33 34 35 se 51 52 Using the eye point landmark coordinates, we can calculate the eye's height-to-width ratio. When an eye is closed, this ratio is, significantly lower. Through tracking this data, we can detect the moment of potential distraction, ‘A.common approach to obtaining facial landmarks is detecting a face bounding box (a box around a face) and locating the landmark coordinates within it. Therefore, is algorithm requires two ingredients —a face detector and a landmark evaluator. We will use deep neural networks (DNN) for both subtasks. You can find the face detection TensorFlow model on GitLab. For a facial landmark evaluator, we will use this Caffe model Detecting Facial Landmarks Let us start writing code for our facial landmarks detection algorithm. We begin with the face detector based on the DNN model Python class TF_FD: def init_(self, model, graph, min_size, min_confidence): 3elf.min_size = min_size self.min_confidence = min_confidence self.detector = cv2.dnn.readNetFronTensorflow(model, graph) Lnanes = self.detector. getLayerNames() 3 Len(1_nanes)>0: ) print(‘Face detector loade print(‘Face detector loading FAILED’) else: def detect(self, frame): width = frame. shape[1] height = frame. shape[2] inputslob cv2.dnn.blobFromimage(frane, 1.0, (30, 300), \ (104.0, 177.0, 123.0), True, False) self.detector.setInput(inputBlob, ‘data’); hitpsslwwu.codeproject.com/Antcles/5324279/Ar-Powered-Driver-Dstraction-Detectionaleplay=Prin ata‘210212022 02:11 ‘Asm-Powered Driver Distraction Detection - CodeProject detection = self.detector.forward( ‘detection_out'); n = detection. shape[2] detected = [] for i in range(n): conf = detection[®, 2, i, 2] Af conf >= self.min_confidence: xd = detection[@, @, 4, 3] yl = detection(@, 0, i, 4] detection[a, @, i, 5] detection[a, @, i, 6] # skip faces out of the frame if Utils.point_is out(x1,y1) or Utils.point_is_out(x2, y2): continue fw = (x2-x1)*width fh = (y2-yl)*height if (fap=self.min_size) and (fh>=self.min_size): r= (xt, yy x2, y2) 4 = (cont, *) detected. append(4) return detected ‘This simple clas provides the constructor for loading 2 Tensorflow neural network from the specified model and graph fies. The Opency framework’s Cv2..dnin module delivers methods for loading DNN models of many popular formats, The constructor has ‘wo additional arguments: the minimum face size and minimum detection confidence ‘The class's detect method receives one parameter, Frame, which is an image or a video frame, The function creates a blob object (a special 4D array that we use as input data for the detecton, Note that we used some specific values for the parameters in our model's bLobFromImage function. If you use another face detection model, remember to change the values as needed. Next, we run the detector calling the forward method and extract data (detection confidence and bounding boxes) forall faces satisfying our criteria (the minimum size and confidence}. We next develop the second class, the facial landmarks detector: Python class CAFFE_FLD: def _init_(self, model, proto): Self.detector = cv2.dnn.readNetFromCaffe(proto, model) names = self.detector.getLayerNames() iF len(1_nanes)>@: print(‘Face landnarks detector loaded:") else print(‘Face landnarks detector loading FATLED') def get_face_rect(self, frame, face) width = Frame.shape[1] height = frame. shape[0] (conf, rect) = face Od, yl, x2, y2) = rect fa © (x2-x1)*width fn = (y2-y1)*height if Furth: dx = (fw-#h)/(2*width) xd = xisax x2 = x2-dx else: dy = (fh-fw)/(2*height) yl = yledy y2 = y2-dy hitpsslwwu.codeproject.com/Antcles/5324279/Ar-Powered-Driver-Dstraction-Detectionaleplay=Prin sina‘a102/2022 02:11 ‘Am-Powered Driver Distraction Detection - CodeProject xd = Utils.fit(x1) yl = utils.fit(yi) 32 = Utils FitG2) y2 = utils.fit(y2) rect = (x1, yl, x2, y2) return rect def get_frane_points(self, face_rect, face_points) (x1, yl, x2, y2) = face_rect fw = (x2-x1) fo = (y2-y2) n = len(face_points) frame_points = [] for i in range(n): v = face_points[i] if (1% 2) == @: x1 fu yt df = th v = dveved frane_points.append(v) return frane_points def get_face_image(self, frame, face): width = frame. shape[1] height = frame. shape[ 9] (conf, rect) = face Od, yl, x2, y2) = rect rect = self.get_face_rect(frame, face) (xi, yil, xi2, yi2)"= Utils.rect_to_abs(rect, width, height) roi = frame[yit:yi2, xi1:xi2] gray = cv2.cvtColor(roi, cv2.COLOR_RGB2GRAY) resized = cv2.resize(gray, (60, 62), 9.0, 0.8, interpolation=cv2.INTER_CUBIC) return (rect, gray, resized) def detect(self, fing width = #_ing.shape[1] height = F_ing.shape[0] inputBlob = cv2.dnn.blobFromImage(f_img, 1/127.5, (68, 68), (127.5)) self.detector.setInput(inputolob, "data’); detection = self.detector. forward(); points = detection[a] return points This class also loads a DNN model on initialization, but it uses another function because this model sin Caffe's specific format. The primary method, detect, again creates a blob and runs the neural network to get facial landmarks. In this case, the detect method receives not an entire frame but a specially processed part of the frame containing one face. We can generate this “face image" using the get_face_image method, specially designed for this purpose. It finds the square box containing the face, crops it from the frame, converts the blue, green, red (BGR) data to a gray-scale image (because we have trained our DNN model on gray images), and resizes the image to 60x60 pixels using a high-quality interpolation method hitps slaw. codeproject.com/Antcles/5324279/Arm-Powered-Driver-Dstraction-Detectionleplay=Prin ana‘21022022 02:1 Running Landmark Detection on Raspberry Pi Now that we have designed our face landmark detector, we should tes algorithm with enough frames per second (FPS), We will perform tests on a Raspberry Pi 4 Model B device, We have installed the Python OpenCV framework on the device using pre-compiled binary packages. If you use another device, you should follow the 7 ‘Asm-Powered Driver Distraction Detection - CodeProject suitable guides to install its packages. In this article, we won't use special Al frameworks, and the neural networks are processed without acceleration on GPU or TPU. So, all the ML workloads run anly on the device's CPU. ‘We'll do all tests using video files to ensure the experiments’ repeatability. The video is set in an office but imitates the scene when driving a cat on an Arm-powered device to verify that it can run the v1 ‘The following class runs facial landmark detection on a video file Python class VideoFLD: def def _init_(self, fd, fld): self.fd process(self, video): frame_count = @ detection_nun = 0; dt=@ dtl =e capture = cv2.VideoCapture (video) img = None’ dname = ‘Arn-Powered Driver Distraction Detection’ cv2.namedWindow(dname, cv2.WINDOW_ NORMAL) cv2.resizewindow(dnane, 720, 720) hitpsslwwu.codeproject.com/Antcles/5324279/Am-Powered-Driver-Dstraction-Detectionaleplay=Prn sina‘210212022 02:11 ‘Asm-Powered Driver Distraction Detection - CodeProject # Capture all frames while (True! (ret, frame) = capture. read() if frame is None break frame_count = frame_count+1 # work with square images width = frane.shape[2] height = frame. shape[2] if not (width == height): dx = int((width-height) /2) frame = frane[@:height, dx:dxsheight] tl = tine.tine() faces = self.fd.detect(frame) 2 = time.tine() dt = dt + (t2-t1) #_count = len(faces) detection_nun += f_count draw_points = [] if (Fcount>e) for (i, face) in enunerate(faces) ti = time.time() (fi_rect, fi_gray, i_resized) = self.fld.get_face_image(frane, face) points = sel¥.fld-detect(Fi_resized) frane_points = self.fld.get_frane_points(fi_rect, points) +2 = Time.time() ati = dtl + (t2-t1) draw_points.append( frane_points) Af len(Faces)> Utils.draw_faces(faces, (255, 0, @), 1, frame, Truc) Af len(draw_points)>8: for (4, points) in enumerate(draw_points): Utils.draw_points(points, (@, @, 255), 1, frame) # Display the resulting frame v2. imshow(dname, frame) Lf ev2.waitKkey(1) & @xFF break ord('q"): capture. release() cv2.destroyAl1Windows() fps = 0.0 Lf dese: fps = frame_count/dt fos_l = 0.0 if dt_be: fps_l = detection_num/dt_1 return (detection_num, fps, fps_1) Here, we use our face and landmark detectors to provide the primary function. We use the VideoCapture class from the Opency library to read frames from a video fle and feed them to the detectors Now we can run the algorithm with the following code: Python wpath = '/home/pi/Desktop/PT_DD* n_path = os.path.join(w_path, ‘net') d_nodel = 0s.path.join(n_path, ‘opency_face_detector_uint8.pb") d_graph = 0s.path.join(n_path, ‘opencv_face_detector.pbtxt') fd = TF_FD(fd_model, fd_graph, 30, 0.5) hitpsslwwu.codeproject.com/Antcles/5324279/Ar-Powered-Driver-Distraction-Detectionaleplay=Prn sina‘210212022 02:11 ‘Asm-Powered Driver Distraction Detection - CodeProject ld_model = os.path.join(n_path, ‘face_landnarks.caffemodel') fld_proto = os.path.join(n_path, ‘face_landnarks.prototxt') ld = CAFFE_FLD(fld_model, #1d_proto) v_path = os.path.join(w_path, ‘video') voname = ‘v_1.mpa" v_file = os.path.join(v_path, v_name) vfld = VideoFLD(fd, fd) (detection_num, fps, fps_l) = vfld.process(v_file) print("Face detections: "+str(detection_num)) print("Detection FPS: “+str(fps)) print("Landmarks FPS: "+str(fps_1)) You can see the screened results in the following video: 11 landmarks detection ‘we're only using the P's CPU. distraction, So, it should be good enough for our distraction detection task. mplementing Driver Distraction Detection We are only one step away from the completed algorithm of distracted driver detection: writing the algorithm for evaluating the eye's height-to-width ratio and tracking it to evaluate the moment of possible distraction. First, we add two simple methods to the CAFFE_FLD class: Python (ur facial landmark detection algorithm works well and locates the reference points with reasonable accuracy. It gave us a face detection speed of about 2 FPS and a landmark evaluation speed of about 60 FS, That's definitely usable, and not bad considering This speed should be sufficient for detecting closed eyes within ane to three seconds, applicable in real situations of driver hitps slaw. codeproject.com/Antcles/5324279/Arm-Powered-Driver-Dstraction-Detectionleplay=Prin m8‘210212022 02:11 ‘Asm-Powered Driver Distraction Detection - CodeProject def get_eye_points(self, face_points, eye_id): 10 = 72 Al = i0+12*(eye_id-1) 42 = ine12 eye_points = face_points[i1:i2] return eye_points def get_eye_ratio(self, eye): n= int(1en(eye)/2) pts = np.array(eye, dtype=np.float32) pts = pts.reshape({n, 2]) rect = cv2.mindreakect(pts) (u, h) = rect[1] Af (woh): ratio = h/w else: ratio = w/h return ratio The get_eye_poiints method extracts points of an eye from the array of 68 face landmarks, The get_eye_ratio method evaluates the eye's height-to-wicth ratio Now we can write the cade to track the ratio value and detect moments of possible distraction, Python class DERD: def _init_(self, ratio_thresh, delta_tine, eye: Self.ratio_thresh = ratio_thresh self.delta time = delta tine self.eyes = eyes self.eye_closed_time = 0.0 self.last_tine = 2.0 def start(self, time): self.eye_closed_time = 0.0 self.last_tine = time def detect(self, eyel_ratio, eye2_ratio, time): dt = time - self.last_time distraction = False 1 = (eyei_ratiocself.ratio_thresh) d2 = (eye2_ratio=self.delta_tine: distraction = True hitpsslwwu.codeproject.com/Antcles/5324279/Ar-Powered-Driver-Dstraction-Detectionaleplay=Prin arta‘210212022 02:11 ‘The ratio_thresh argument is the minimum value of the height-to-width ratio to assume the eye is closed. The delta_time parameter denotes how long the eye must be closed to decide whether a distraction occurred, The eyes parameter determines whether one or bot eyes must be closed to consider this a distraction. Finally, we slightly modified our video detector to include this distraction detection algorithm in the code and generate an alarm self.start(tine) self.last_tine = time return distraction when a detection happens. Python ‘Acm-Powared Driver Distraction Dataction - CodeProject class Videooob: def def _init_(self, fd, fld, eye_ratio_thres! self.fd = fd self.fld = fld self.derd = DERD(eye_ratio_thresh, delta_tine, eyes) process(self, video): frame_count = @ detection_nun = 0; dt =e dtl =e capture = cv2.VideoCapture(video) img = None dname = ‘Arn-Powered Driver Distraction Detection’ cv2.namedWindow(dname, cv2.WINDOW NORMAL) cv2.resizeWindow(dnane, 720, 72@) # just suppose FPS=25 delta = 0.040 dd_time = -1000 draw_points = [] faces = [] # Capture all frames while (True! frame_tl = tine.time() (ret, frame) = capture. read() ‘if frame is None break frame_count = frame_count+l frame_tine = (frame_count-1)*delta if frame_counts=1 self.derd.start(frame_tine) # work with square images width = frane.shape(1] height = frane.shape[2] if not (width == height): dx = int((width-height) /2) frame = frane[@:height, dx:dxtheight] Fcount = @ AF (frame_count % 10) faces = [] draw_points = [] ti = time.time() faces = self.fd.detect(frame) 42 = time.tine() hitpsslwwu.codeproject.com/Antcles/5324279/Am-Powered-Driver-Distration-Detectionaleplay=Prin 2, eyes=2, delta_tim orna‘210212022 02:11 ‘Asm-Powered Driver Distraction Detection - CodeProject at = at + (t2-t1) _count = len(faces) detection_num += 1 distraction = False Af (counts): # Supposed one Face at the camera face = faces[@] t= tine.tine() (fi_rect, fi_gray, fi_resized) = self.fld.get_face_image(frame, face) points = self. fid-detect(Fi resized) Frane_points = seif.f1d.get_frane_points(fi_rect, points) t2 = time.time() atl = del + (t2-t1) draw_points.append(frame_points) eyel = self.fld.get_eye_points(frane_points, 1) eye2 = self.fld.get_eye_points(frane_points, 2) #dran_points.append(eyel) #dran_points.append(eye2) 1 = self. fld.get_eye_ratio(eyel) r2 = self. fld.get_eye ratio(eye2) distraction = self.derd.detect (ri, r2, frame_time) Af Len(faces)>@: Utils.draw_faces(faces, (255, ®, @), 1, frame, True) Af Len(draw_points)>@ for (i, points) in enumerate(draw_points): Utils.draw_points(points, (®, @, 255), 1, frame) # Show distraction alarm for 1 second if distraction: dd_time = frame_time Af dd_timer: ‘text = "ALARM! DRIVER DISTRACTION xd1 = 10 yd = 50 cv2.putText(frame, text, (xd1, yd), \ cv2.FONT_HERSHEY SIMPLEX, ©.6, (@, @, 255), 1, cv2.LINE_AA) if (frame_tine-dd_time)>1.¢: dd_time = -1000 # Display the resulting frame v2. imshow(dname, frame) frane_t2 = time.time() frane_dt = frane_t2 - frame_tt Af frame_dtcdelta: frane_dt = delta-frane_dt aprint(‘Sleep="+str(frane_dt)) ‘time. sleep(frame_dt) Af cv2.waitkey(1) & @xFF break ord(*q"): capture. release() cv2.destroyAllWindows() fps = 0.0 if dtve: fps = detection_num/dt fos_l = 0.0 hitpsslwwu.codeproject.com/Antcles/5324279/Ar-Powered-Driver-Dstraction-Detectionaleplay=Prin tora‘210212022 02:11 ‘Asm-Powered Driver Distraction Detection - CodeProject detection_num/dt_1 return (detection_num, fps, fps_1) In addition to using the DERD class, we slightly changed the frame processing algorithm, We added a timestamp comparison of frames to estimate time intervals of possible distraction. Also, we now process only every tenth frame to imitate the near real-time processing, Now we can run the completed driver distraction detection algorithm with the following code: Python wpath = */home/pi/Desktop/PI_pb" n_path = os.path.join(w_path, ‘net') #4_nodel = 65.path.join(n_path, ‘opencv_face_detector_uint8.pb") fd_graph = o5.path. Join(n_path, ‘opency_face_detector.pbtxt ) d= TF_D(fd_model, d_graph, 30, 0.5) fld_nodel = os.path.join(n_path, 'face_landnarks.caffemodel" ) #1d_proto = os.path.join(n_path, ‘face landnarks.prototxt') ld = CAFFE_FLD(fld_model, f1d_proto) v_path = 0s.path.join(w_path, 'video') voname = ‘v_1.mpa" v_file = os.path.join(v_path, v_name) vddd = Videoooo(fd, fld, @.3, 1, 2.0) (detection_num, fps, fps_l) = vddd.process(v_file) print("Face detection: print ("Detection FPS print (“Landmarks FPS str(detection_nun)) ‘sstr(fps)) +str(fps_1)) You can see that the algorithm correctly handles and generates the alarm, situation when the eyes appear to be closed during a long enough time interval This helps us catch one cause of driver distraction — when a driver is looking down at a mobile device in their lap, our drowsiness detector identifies them as distracted because it only see their eyelids. As jurisdictions around the world ba mobile device use iil driving, drivers have tried to adapt by holding their devices out of sight. But our distraction detector will catch them by detecting when their eyes don't appear to be fully open. Conveniently, this algorithm can also work to detect driver drowsiness. Our device should raise the alarm whether a driver's eyes merely appear closed because they are looking down at a mobile device, or they actually are closed because the driver is drowsy or asleep, hitps slaw. codeproject.com/Antcles/5324279/Arm-Powered-Driver-Distraction-Detectionaleplay=Prn wna‘210212022 02:11 ‘Acm-Powered Driver Distraction Detection - CodeProject 12 distraction detection The algorithm also correctly handles the situation when the eyes are closed for a short time interval (for example, a driver blinks) or ‘the head is slightly turned for a short time. 13 no distraction Next Steps hitpsslwwu.codeproject.com/Antcles/5324279/Arm-Powered-Driver-Distraction-Detectionaleplay=Prin ana‘210212022 02:11 ‘Asm-Powered Driver Distraction Detection - CodeProject ‘We've implemented one driver distraction algorithm using facial landmarks - but we could add others! Far example, we might want to detect when a driver's head is turned by measuring the angle of the lines between nose landmarks, We might also check if a driver's mouth appears to be opening and closing by comparing the distance between upper and lower mouth landmarks over time. If they are, it likely means the driver is talking or eating while driving ‘To take things even further, we might consider upgrading to an ML model that can do iris detection, and try to determine when a driver's eyes aren't looking at the road, In this article, we demonstrated how straightforward itis to develop an Al computer vision application for a portable Arm-powered device, We chose this solution for practicality since our driver distraction detection system must function autonomously in a driving cat. We showed that the application could run on the Arm-powered device in real-time made, reaching about two FPS processing speed, Nevertheless, we can still investigate many aspects to improve this driver distraction detection system. For instance, can we increase ‘the FPS? To answer this, we should pay attention to the application's slowest part— face detection with the TensorFlow neural network. Can we improve this model's performance? Yes, We can use Arm's Arm NN library, which they specially developed to accelerate processing DNN models on Arm-powered devices. With the Arm NN library, we also could run the NN model on connected GPU or NPU units to achieve near real-time speed. This, would give us more flexibility to invent advanced algorithms of driver distraction detection or to use ather face detection DN models, lke the BlazeFace neural model Other improvements to our solution might involve generating new distraction criteria, For example, we can infer that drivers are likely distracted if their eyes or head are directed elsewhere for more than a determined time interval We hope these ideas have piqued your interest, We encourage you to expand on this solution or create your ovin portable Al solutions on Arm-poviered devices. History ‘ebruary, 2022: Initial version. License This article, along with any associated source code and file, is licensed under The Code Project Open License (CPOL) About the Author Sergey L. Gladki. Team Leader VIPAKS Russian Federation mal EDUCATION: Master's degree in Mechanics. PhD degree in Mathematics and Physics. hitps slaw. codeproject.com/Antcles/5324279/Arm-Powered-Driver-Distraction-Detectionaleplay=Prn 1314‘210212022 02:11 ‘Acm-Powared Driver Distraction Dataction - CodeProject PROFESSIONAL EXPERIENCE: 15 years’ experience in developing scientific programs (C#, C++, Delphi Java, Fortran). SCIENTIFIC INTERESTS: Mathematical modeling, symbolic computer algebra, numerical methods, 3D geometry modeling, artificial intelligence, differential equations, boundary value problems. Comments and Discussions Ao messages have been posted for this article Visit https://www.codeproject.com/Articles/5324279/Arm-Powered- Driver- Distraction-Detection to post and view comments on this article, or clck here to gat a print view with messages Permalink ‘Asticle Copyright 2022 by Sergey L Glackiy Advertise Everything else Copyright © CodeProject, 1999- Privacy 2022 Cookies Terms of Use webos 28.2022.02.10 hitps:slwwu.codeproject.com/Artcles/5324279/Arm-Powered-Driver-Distraction-Detectiondlsplay=Print tana