MM-TrafficRisk: A Video-based Fleet Management
Application for Traffic Risk Prediction, Prevention,
and Querying
Minh-Son Dao, Muhamad Hilmil Muchtar Aditya Pradana, and Koji Zettsu
National Institute of Information And Communications Technology, Tokyo, Japan
{dao, hilmil, zettsu}@nict.go.jp
Abstract—This paper introduces MM-TrafficRisk, an innova-
tive fleet management application that harnesses dashcam video
data, environmental data, and physiological data to forecast,
mitigate, and investigate traffic-risk events while uncovering
traffic-risk patterns. To provide a comprehensive overview of this
application, we outline its system architecture, encompassing a
database, ETL processes, UI/UX components, and a fine-grained
text-video search engine. Additionally, we present a groundbreak-
ing two-stage near-miss accident prediction model designed to
identify near-miss incidents within dashcam video databases,
and the text-to-video search engine, facilitating rapid searches
for traffic-risk events based on textual queries. These models
and search engines are rigorously evaluated in a controlled
laboratory environment to showcase their advantages. Moreover,
we highlight several essential functions of the MM-TrafficRisk
application through snapshots, emphasizing the collaborative
efforts between a government agency and industrial companies to
develop and deploy this application in practical settings. We also
delve into our future endeavors, focusing on multi-modal deep
learning event prediction and the adaptability of our application
to Edge AI environments.
Index Terms—traffic-risk forecasting, multi-modal deep learn-
ing models, industry-government collaboration, cross-modal
search engine
I. I NTRODUCTION
The topmost priority for the industry, businesses, and gov-
ernment alike is ensuring transportation safety. Each of these
entities focuses on distinct aspects: the industry strives to
create products that enhance traveler safety, businesses are
concerned with logistics cost, fleet management, and insur-
ance, while the government’s primary objective is to formulate
policies that safeguard human life and raise awareness about
traffic risks for travelers.
In industry, we are currently witnessing the evolution of
autonomous vehicles capable of operating without human
drivers. These technologies rely on a suite of sensors both
within and outside the vehicle, such as dashcams, stereo
cameras, and onboard sensors, as well as remote sensors
like satellites, CCTV systems, and loop sensors. Additionally,
artificial intelligence techniques are essential for extracting
real-time insights from the data collected. This, however,
results in a substantial operating cost for autonomous vehicles.
While the vision of making autonomous vehicles accessible
for all purposes is intriguing, it will require more time before
these technologies become widely adopted and commonplace
worldwide. While awaiting the maturity of autonomous vehi-
cles, individuals have been exploring alternative, convenient,
and cost-effective solutions that offer intelligence to assist
with transportation needs, both in direct tasks like driving and
indirect roles such as coaching and management. One of these
solutions involves the development of video-based systems
that utilize what are known as ’smart cameras.’ These cameras
are equipped with various techniques and can connect with in-
cabin and wearable sensors to help drivers respond promptly
and appropriately to traffic hazards. The use of video-based
safety systems in commercial motor vehicles can help improve
road safety and prevent accidents [1], [2]. These systems
use cameras to capture footage of the road and the driver’s
behavior, which can be analyzed to identify risky driving
habits and provide feedback to drivers. The implementation
of these systems comes with challenges such as concerns over
driver privacy and the cost of equipment and data analysis.
However, video-based safety systems have the potential to
significantly improve fleet driver safety and reduce the risk of
accidents on the road. Additionally, video recorders could be
used to capture data that could help in accident investigations
and improve safety.
In the pursuit of enhancing transportation safety, govern-
ments cannot remain on the sidelines. They have implemented
several policies aimed at supporting the advancement of ad-
vanced driver assistance systems (ADAS) in commercial motor
vehicles that also has the potential to improve road safety [3].
The federal government has taken steps to address this issue
by passing the Infrastructure Investment and Jobs Act (IIJA),
which allocates billions of dollars in funding to transporta-
tion initiatives and directs the Secretary of Transportation to
conduct various motor vehicle safety-related studies. The IIJA
includes provisions aimed at addressing driver distraction and
mandates the use of driver monitoring systems to minimize or
eliminate driver distraction, driver disengagement, automation
complacency by drivers, and foreseeable misuse of ADAS.
The directive in the IIJA to incorporate privacy and data
security safeguards into any regulations or rules relating to the
adoption of ADAS highlights the importance of stakeholders
understanding how these technologies work and what privacy
risks may arise.
The use of video telematics in commercial motor vehicles
can help fleets get smarter by providing data to improve safety
and increase efficiency [4]. This data can be used to implement
training and safety programs that can help reduce accidents
and improve driver safety. The telematics market is set to
experience explosive growth due to the rise in the adoption
of connected car technologies, advancements in wireless com-
munication technology, and the growth of the IoT. Ultimately,
the use of these technologies in commercial motor vehicles
is important to improve road safety and prevent accidents.
By balancing safety, privacy, and transparency, stakeholders
can facilitate the safe and responsible deployment of these
technologies in the commercial transportation industry.
As this trend continues, we are motivated to leverage
multimedia data analytics and AI technology to create a fleet
management system that uses dashcam video data and other
data collected from wearable and IoT sensors. Unlike current
commercial products, our system places a strong emphasis on
predicting near-miss accidents in order to help drivers avoid
collisions. A near-miss accident, which is also known as a
near accident or close call, refers to an incident that had the
potential to cause harm, injury, or property damage, but was
ultimately prevented by chance or corrective action.
In our commitment to offer users, particularly fleet man-
agers and driving coaches, a comprehensive understanding of
the attributes of traffic risks for enhancing the effectiveness of
management, training, and self-learning, we have introduced a
sophisticated text-to-video feature. This feature enables users
to search for specific traffic risk events whose content aligns
with their textual queries. It seamlessly integrates with a traffic
risk event browsing and search interface, bridging the gap
between users’ conceptualizations and real-world occurrences.
Moreover, this interface empowers users to visualize correla-
tions and co-occurrences among data associated with the same
event, facilitating the development of new models for detecting
novel traffic risk events.
In this paper, we present a collaborative effort between a
government agency and industrial companies resulting in a
fleet management product designed to enhance transportation
safety for both individuals and fleet companies. The gov-
ernment agency has focused on developing algorithms and
AI models for detecting and identifying traffic-risk events,
while the industrial companies have contributed by providing
essential infrastructure, conducting user surveys, gathering
feedback, supplying necessary data, and offering testing fa-
cilities for system evaluation. This paper’s contributions are
outlined as follows:
1) We highlight the benefits of multi-sector collaboration
in research and development, culminating in the intro-
duction of a new product to the market.
2) We introduce two pivotal functions critical for improv-
ing transportation safety: traffic-risk prediction, which
includes near-miss accident prediction, and retrieval,
specifically text-to-video searching.
3) We present a comprehensive fleet management sys-
tem encompassing system architecture and user in-
terface/user experience (UI/UX) design, meeting the
rigorous criteria set by both government and industry
stakeholders, bridging the gap between academia and
practical applications.
The structure of this paper is as follows: In Section II, we
offer a concise overview of commercial products associated
with traffic-risk prevention utilizing dashcams. Section III
delves into the MM-TrafficRisk System, a collaborative effort
involving multiple sectors. Section IV presents experimental
results for the system’s two primary functions: near-miss
accident prediction and text-to-video searching. Finally, in
Section V, we draw conclusions and outline future avenues
of work.
II. R ELATED W ORKS
Several technology companies have developed solutions to
enhance safety in the trucking industry using various technolo-
gies such as Video-Based Safety (VBS), dash cameras, and
smart video technology. These technologies provide numerous
benefits such as real-time feedback to drivers, reduced accident
rates, and improved driver behavior. They can also identify
high-risk drivers, enhance driver coaching and training, and
reduce liability for fleets.
However, there are also disadvantages to consider. One
major concern is driver privacy, as drivers may feel uncom-
fortable being constantly monitored. This can lead to potential
pushback from drivers who may perceive it as a violation
of their privacy. Additionally, the cost of implementing these
technologies, including hardware and software, can be expen-
sive for some fleets.
Omnitracs [5] has developed a safety solution for the truck-
ing industry using Video-Based Safety (VBS) technology to
improve safety. This product offers various advantages of VBS
technology, including real-time feedback to drivers, decreased
accident rates, and better driver behavior. Furthermore, it
explains how VBS technology can aid in driver coaching and
training, identifying high-risk drivers, and minimizing liability
for fleets.
Wexinc [6] offers a useful fleet management tool that
enhances driver efficiency and safety through the use of dash-
cams. These cameras provide valuable insights into driving
behaviors and accident causes, and can serve as evidence in the
event of an accident. Furthermore, they can reduce accidents
by providing feedback to drivers and serving as a deterrent for
risky behavior.
Verizon Connect [7] offers a smart video technology that
can improve fleet safety. This technology allows for the
monitoring of driver behavior and the detection of risky driving
habits. Furthermore, it can help to clear drivers who were
not at fault in an accident. The use of this technology can
lead to improved driver behavior, reduced insurance costs,
and better legal protection. However, there are also disad-
vantages to using smart video technology, such as concerns
about driver privacy and potential pushback from drivers who
are uncomfortable being monitored. In addition, the cost of
implementing the technology may be significant, including the
need for hardware and software investments.
Netradyne [8] is a tech firm that concentrates on creating ad-
vanced and efficient safety solutions for business fleets. Their
principal product, Driveri, is a video telematics system that
employs artificial intelligence and machine learning to assess
driver behavior and provide immediate feedback to enhance
safety. Additionally, it provides tools for driver training and
coaching, as well as fleet management capabilities like GPS
tracking, vehicle diagnostics, and route optimization. Further-
more, Netradyne provides other safety solutions, such as a
collision avoidance system and a distracted driving detection
system.
Omnitracs, Wexinc, Verizonconnect, and Netradyne are
some of the technology companies providing safety solutions
to the trucking industry. Each company has unique products
that utilize VBS, dash cameras, smart video technology, and
AI to provide fleet management features, analyze driver be-
havior, and offer real-time feedback to drivers. While these
technologies have many advantages, it is important for fleets
to implement them with a focus on driver buy-in and under-
standing, as well as ensuring compliance with privacy laws
and regulations. By doing so, fleets can improve safety and
efficiency, reduce accident rates, and provide legal protection
for their drivers.
Regrettably, the aforementioned products lack the capa-
bilities of predicting near-miss accidents and providing a
finely detailed text-video search engine within their systems.
These functionalities have been exclusively developed and
incorporated into our own system.
III. MM-T RAFFIC R ISK S YSTEM
The transportation industry has been rapidly adopting
technology-driven solutions to improve fleet management,
reduce operational costs, and enhance passenger safety. In
this paper section, we present our work on developing a
comprehensive product architecture for a transportation com-
pany that integrates various technologies, including UI/UX
design, AI models, and data augmentation techniques. Our
product includes a user-friendly UI/UX design that enables
seamless interaction with fleet managers and enhances the
accuracy of the AI models. We developed a dashcam-based
near-miss accident prediction model that uses video data to
predict potential accidents and prevent them from happening.
Additionally, we designed a cross-modal text-video search
engine that allows users to search for specific events in the
dashcam database using textual queries. Finally, we propose a
data augmentation approach to enrich and diversify the training
data and enable self-supervised learning for annotation, which
enhances the effectiveness of our AI models. Our product pro-
vides a robust and comprehensive solution for transportation
companies looking to leverage technology to improve fleet
management and enhance passenger safety.
A. System Architecture
In this subsection, we will provide a detailed explana-
tion of the significant components of the fleet management
system. The system has been designed and deployed for a
transportation company, with servers located at the company’s
headquarters and branches. Clients have been installed in every
vehicle of the company, including dashcams, wearable sensors,
and in-cabin IoT sensors. By providing the transportation
company with reliable and timely information about their
fleet’s operations, including driver behavior analysis, near-
miss accident detection, and abnormal event identification, the
system aims to enhance the company’s overall performance.
Figure 1 illustrates the overview of system architecture.
The system comprises three main parts: the front-end,
back-end, and AI services. The front-end is responsible for
displaying data and information, as well as allowing users to
interact with the system. It is designed to be user-friendly and
intuitive, with a dashboard that provides a real-time overview
of the fleet’s operations. Users can also access more detailed
information about individual vehicles and drivers, as well as
run reports and analytics.
The back-end is responsible for managing services and
carrying out commands. It receives requests from the front-
end and processes them to provide the desired information or
action. It also returns feedback to the front-end, allowing users
to see the result of their actions.
The AI services are in charge of running AI services, which
focus on detecting and searching events and data augmentation
and annotation. This includes the dashcam-based near-miss
accident prediction model and the cross-modal text-video
search engine, which allows users to search for specific events
from the dashcam database. The AI services also augment
data for enriched and diverse training data and self-supervised
learning for annotation, which helps improve the accuracy of
the AI models over time.
Finally, the system’s data storage and components con-
nection are critical components. The system’s data storage
is designed to be scalable and secure, allowing the system
to store large amounts of data from multiple sources. The
components connection ensures that all the different parts of
the system work together seamlessly, allowing for real-time
data processing and analysis.
B. Dashcam-based near-miss accident prediction
A near-miss accident, also known as a near collision, refers
to an incident in which a vehicle or pedestrian almost collided
with another object or person but was able to avoid it at the
last moment. Although these events may not result in property
damage or injury, they can serve as warning signs for future
accidents. Identifying and analyzing near-miss accidents can
provide valuable insights into potential hazards on the road,
helping to prevent more serious accidents from occurring in the
future. Therefore, it is essential to include near-miss accident
detection in transportation safety measures to improve road
safety and reduce the risk of accidents.
 Fig. 1. System Architecture: Overview
In this subsection, we introduce our approach for predicting
near-miss accidents using dashcam video data. Our two-stage
algorithm is designed to be highly effective: In stage one, we
focus on detecting the safety area. In stage two, we use 30
seconds of video data from the current time and go back 30
seconds to apply our classification models, predicting what
type of near-miss accident might occur. With this innovative
approach, we can proactively alert drivers to potential hazards
on the road, ultimately reducing the likelihood of accidents
and promoting safer driving habits.
In the first stage, Equation 1, as introduced in [9], is utilized
to calculate the minimum stopping distance required by a
vehicle, considering the present velocity and coefficient of
friction that depends on various factors such as weather and
road conditions. The primary goal of this stage is to convert
the 3D first-person view videos into 2D top-view videos using
a meter-based coordinate system. This conversion process
accurately identifies moving objects, including cars, bicycles,
and pedestrians, and also identifies obstacles and road signs
with their precise location, velocity, and trajectory information,
which is crucial for the near-miss accident detection.
dstop = v 2 /2µg (1)
where
• dstop : the stopping distance (m)
• v: the initial velocity of the vehicle (m/s)
• µ: the coefficient of friction [10]
2
• g: the acceleration due to gravity (9.8m/s )
To accomplish the task of converting the 3D first-person
view videos into 2D top-view videos, we employ a series
of techniques. First, we use the Deep Hough Transform [11]
and YOLOP [12] to detect semantic lines and determine the
vanishing point. With the help of lane and road sign detection,
we then calculate the extrinsic camera parameters. Using the
known intrinsic camera parameters, the dashcam position, and
the extrinsic camera parameters, we can calibrate the 3D
first-view frame to a 2D top-view image. To estimate the
velocity of all objects that appear in the field-of-view of the
dashcam, we utilize the GPS sensors installed in the vehicle
to estimate the ego-vehicle velocity, along with the 2D map.
This process allows us to accurately measure the location,
velocity, and trajectory of objects, such as cars, bicycles,
and pedestrians, that are necessary for detecting near-miss
accidents. The overview of the first stage of the near-miss
accident prediction algorithm is depicted in Figure 2.
An example of the results from stage 1 is depicted in Figure
3. The left side of the figure shows the current velocity of
the ego-truck, indicating a ”NOT SAFE” situation due to the
sudden brake of the upfront car, which resulted in the violation
of the safety distance. On the right side of the figure, a 2D
map is presented, depicting the travelling objects, with the
ego-truck in the center and a green track.
Once the safety area has been breached, an alarm is
activated, and a 30-second risk video is generated, which
concludes at the point of activation. Subsequently, the second
stage of the algorithm is initiated, wherein the S3D model,
originally introduced in [13], is employed with modifications
to cater to our near-miss accident dataset. The model is trained
to extract pertinent features that are required for constructing
the classifier.
Eleven types of near-miss accidents, focusing on the nature
of their occurrence as shown in Figure 4, have been defined
based on the accident and near-miss accident definitions of
Tokyo University of Agriculture and Technology (TUAT) [14]
and the National Center for Statistics and Analysis (NCSA) 1 .
1 https://www.nhtsa.gov/data

Fig. 2. near-miss accident prediction: An Overview
To ensure accuracy, we also conducted a survey with a trans-
portation company to refine our definitions. Leveraging these
definitions, we annotated data collected by the transportation
company and re-annotated other open datasets such as DADA
[15] and RetroTruck [16].
Since we have extracted the velocity, position, and trajectory
of all the objects that appear within the field-of-view of the
dashcam, these pieces of information can be employed as
fine-grained features, together with the S3D-based features
extracted from the entire video and the road lanes, to develop
the classifier. Ultimately, our system is capable of alerting
drivers to critical moments that may pose a threat to their
safety, while also providing insight into the potential type of
accident that could occur if swift and appropriate action is not
taken.
C. Cross-modal text-video search engine
In addition to the primary goal of predicting near-miss
accidents, the capability to search and query other traffic
risk events that occurred during the travel time from the
company’s storage is also a crucial task. This task serves
multiple purposes, including gaining a better understanding of
driver behavior, unexpected accidents, and providing evidence
for policies and insurance companies. Moreover, it provides
valuable information for the company to train or retrain
their drivers to respond effectively to various types of traffic
risks. By collecting and analyzing this data, we can extend
the number of accidents that our model can predict in the
future through annotations and retraining processes, ultimately
improving the safety of drivers on the road.
To accomplish our objective, we have developed a cross-
modal text-video search engine based on the methodology
outlined in [17]. The architectural framework of this model
is depicted in Figure 5. Within this design, our aim is to
generate cross-features that enhance the overall global and
local correlations between visual and textual attributes of
events and objects, respectively. This process begins with the
extraction of visual, textual, and object relation features from
both video and text inputs, employing a video encoder (e.g.,
ViVIT), an object relation encoder (e.g., Visual Genome),
and a text encoder (e.g., BERT). These extracted features
are then supplied to the (Q, V, K) inputs of a multi-head
attention-transformer backbone, along with additional position
encoding.
To improve similarity between an ”entity” in text and
an ”object” in a video while considering object relations
from both visual and textual perspectives, we define a loss
function. Upon completion of the training stage, our system
gains the capability to retrieve traffic-risk videos that meet
the conditions described in the textual queries. For readers
seeking a deeper understanding of the algorithm, we invite
you to consult the details provided in the paper [17].
D. Data augmentation and self-supervised annotation
Developing near-miss accident prediction models requires
advanced technologies to identify and mitigate potential traffic
risks. Current methods for identifying risks typically focus on
detecting anomalies frame-by-frame, rather than identifying
which participant could cause a collision. This approach is lim-
ited by the availability of annotation datasets, which only allow
for the detection of anomalies. Near-miss accidents, which are
narrowly avoided collisions, are a type of traffic risk that is
not differentiated from actual accidents before the collision
occurs. In response, we decide to redefine the definition of an
accident and re-annotate the DADA-2000 dataset to include
near-miss accidents, by extending the duration of the accident
and precisely covering all ego-motions during an incident.
To achieve this, we relied on the method that integrates con-
ditional style translation (CST) and separable 3-dimensional
convolutional neural network (S3D) [18]. The CST architec-
ture is used to augment the re-annotation of the DADA-2000
dataset [15], increasing the number of traffic risk accident
videos and generalizing the performance of the video clas-
sification model under different types of conditions. The S3D
component is useful for video classification to prove dataset
re-annotation consistency.
The evaluation of the proposed method yielded a significant
improvement in accuracy on cross-validation analysis, with
a 10.25% positive margin from the baseline model. The re-
annotation using the proposed method is valuable for the com-
puter vision community to train their models to produce better
traffic risk classification. The authors stress the importance of
consistent dataset annotation in improving the performance
of the classification model to generalize classification on real
applications. These findings highlight the need for increased
dataset annotation consistency to improve the performance of
 Fig. 3. Stage 1 Results: An Example of near-miss Accident Detection

Fig. 4. near-miss Accident Definitions

self-driving systems.

Fig. 5. Text-2-Video: Overview
classification models in real-world accident videos, which has
implications for the development of more reliable and safe
E. UI/UX design
The front-end components depicted in Figure 1 aim to
provide a comprehensive UI/UX for fleet management tasks.
However, due to space limitations, the authors present only a
few snapshots of the system, focusing on statistical reporting,
event browsing, and event searching.
As shown in Figure 6, the system includes a statistical
reporting feature that generates reports on a daily, weekly,
monthly, and yearly basis.
Another snapshot 7 depicts a near-miss accident report,
which provides rich information on where and how the ac-
cident occurred, as well as physiological and in-cabin en-
vironment data. The screen also displays a map and in-
cabin/out-cabin video shots, with the ”orange” part of the
Fig. 6. Statistical Report Screen exhibits various statistical reports aimed at aiding managers in comprehending the fleet’s status better. For instance, in the
top-left, there’s a combined evaluation; in the top-middle, data on safe driving; and in the top-right, information regarding economical driving.
bar under the in-cabin video indicating the time of the near-
miss accident. Users can interact with several components
to obtain more specific data, such as the driver’s behavior
before, during, and after the near-miss accident, or how in-
cabin environment factors like temperature and luminance can
affect driver alertness.
The UI/UX offers a significant and valuable advantage to
both regular users and researchers. It enables them to visually
perceive the correlations among various factors, including
human activities (such as gestures, heart rate, and stress level),
in-cabin environmental conditions (like CO2 levels, PM2.5
particulate matter, illumination, and temperature), and external
factors affecting the cabin environment (such as traffic conges-
tion, rain, urban crowded areas, and highway conditions).
This wealth of information provides numerous cues and in-
sights for the development of new models or patterns databases
related to traffic risk. Additionally, it facilitates the manual
annotation of rare events, enabling the successful training of
few-shot learning algorithms.
In this scenario illustrated in Figure 8, the user utilizes
both tabular conditions and free-style text queries to perform
a video search. When the user wishes to view further details
about a specific event, the system directs them to the screen
shown in Figure 7.
Overall, these UI/UX features enable fleet managers to
gain better insight into their operations and make informed
decisions to improve safety and efficiency.
IV. E XPERIMENTAL R ESULTS
In this section, we present in-lab experimental findings
related to two fundamental functions: near-miss accident pre-
diction and text-to-video retrieval. It’s worth noting that while
we have published the results of these experiments in academic
journals and conferences, we are unable to disclose real-
world testing field results due to company confidentiality
considerations.
A. Near-Miss Accident Detection
The results presented in this subsection are primarily derived
from our publication in [19]. We have re-conducted the tests
with certain parameter adjustments and data refinements in
comparison to the original experiments.
We utilized the CST-S3D dataset [18] (mentioned in sub-
section III-D) and NHW dataset (provided by our industrial
partner) for the training and evaluation of our model. For pos-
itive samples, which represent traffic-risk events, we allocated
837 videos for training, 324 for validation, and 132 for testing
from the CST-S3D dataset. The NHW dataset was exclusively
used for testing and contained 231 videos. Conversely, for
negative samples, which serve as non-traffic-risk instances,
we designated 612 videos for training, 138 for validation, and
69 for testing from the CST-S3D dataset. The NHW dataset
contributed 204 videos for testing purposes.
Figure 9 presents the results of Near-Miss Accident detec-
tion in the Hitting/Conflicting and Crossing/Perpendicularly
 Fig. 7. near-miss Accident Report: Detailed Screen

Fig. 8. Text-Video Searching

classes. In this context, Hitting/Conflicting refers to a scenario Meanwhile, Crossing/Perpendicularly denotes a conflict where
where the ego-car is on the verge of colliding with an object the ego-car encounters an object crossing its path at a perpen-
situated directly in front of it, as illustrated in Figure 4. dicular angle. It’s worth noting that, in reality, certain classes
 Fig. 9. Near-Miss Accident Detection Results: Hitting/Conflicting and Crossing/Perpendicularly Classes (copied from [19]
have very limited samples, and in some cases, no samples at
all, as evident in Figure 9. For instance, the NWH dataset,
which represents real-world driving scenarios, lacks samples
featuring cyclists, pedestrians, and motorcycles in both the Hit-
ting and Crossing classes. This scarcity poses a considerable
challenge in fine-tuning the model and underscores the need
to augment data from the limited original dataset to enrich
the variety of available samples. For readers seeking more in-
depth insights into the experimental results, we encourage you
to consult the original paper [19].
B. Text-2-Video Retrieval
The results presented in this subsection are primarily de-
rived from our publication in MMM 2023 [17]. We have re-
conducted the tests with certain parameter adjustments and
data refinements in comparison to the original experiments.
In this experiment, we utilized the same dataset as described
in subsection , where each video was segmented into 30-
second clips, consisting of 20 seconds before the occurrence
of a traffic-risk event and 10 seconds thereafter. To enrich the
dataset, we engaged in manual caption creation, employing
crowd-sourced labeling to generate five distinct captions for
each video, spanning from abstract to detailed descriptions.
Subsequently, we applied the algorithm outlined in subsection
III-D to augment the dataset, expanding its volume, and
proceeded to index and store this augmented dataset in our
database. We then activated the text-to-video search engine and
enlisted the assistance of volunteers to query videos using their
own text-based queries. Finally, we evaluated the Precision at
K (P@K) for each volunteer’s search results and calculated the
average to report the final outcomes. The results were notably
positive, with P@10 at 8, P@50 at 46, P@100 at 95, and
P@200 at 194.
V. C ONCLUSIONS
This paper presented a comprehensive product architecture
that integrates various technologies, including UI/UX design,
AI models, and data augmentation techniques. The developed
product features a user-friendly UI/UX design, a dashcam-
based near-miss accident prediction model, and a cross-modal
text-video search engine. Additionally, a data augmentation
approach was proposed to enrich and diversify the training
data and enable self-supervised learning for annotation. The
product offers a robust and comprehensive solution for trans-
portation companies seeking to leverage technology to enhance
fleet management and passenger safety.
This paper also underscores the achievements resulting from
the effective collaboration between a government agency and
industry companies. This collaborative effort has facilitated
the swift transition of knowledge generated in a laboratory
setting to user-centric products for widespread societal benefit.
Such collaboration has not only helped reduce costs but also
enabled the rapid response to genuine market demands and
requirements.
The system is currently undergoing beta testing and will
soon be released to the transportation company. We are con-
tinuously improving the system by incorporating the latest
computer vision and AI technologies to better understand
driver behavior and establish correlations between the driver’s
mental and physical state, the in-cabin environment, and the
surrounding conditions outside the cabin. This will enable
the system to recognize necessary ”signs” that may lead to
accidents and enhance traffic safety.
Furthermore, as an outcome of this collaboration, a new
integrated device consolidating all previously separate IoT
devices utilized for system evaluation is currently in de-
velopment. This device will be a distinctive unit equipped
with CPU/GPU, a video camera, IoT capabilities, and Wi-Fi
connectivity. Each of these devices will also function as an
edge client, actively participating in edge AI initiatives that
hold promise for the future of fleet management and Advanced
Driver Assistance Systems (ADAS).
ACKNOWLEDGMENT
This R&D includes the results of ”Research and develop-
ment of optimized AI technology by secure data coordination
(JPMI00316)” by the Ministry of Internal Affairs and Com-
munications (MIC), Japan.
[11]
R EFERENCES
[1] worktruckonline, “can-video-based-safety-systems-improve-fleet-driver-
safety,” https://www.worktruckonline.com/10186709/can-video-based-
safety-systems-improve-fleet-driver-safety.
[2] truckinginfo, “Ntsb explains how video recorders could have a major
impact on safety,” https://www.truckinginfo.com/313449/ntsb-explains-
how-video-recorders-could-have-a-major-impact-on-safety.
[3] FPF, “FPF-Samsara-White-Paper,” https://fpf.org/wp-
content/uploads/2022/05/FPF-Samsara-White-Paper.pdf.
[4] automotive fleet, “can-video-telematics-get-smarter,”
https://www.automotive-fleet.com/345812/can-video-telematics-get-
smarter.
[5] omnitracs, “Sd20-029,” https://www.omnitracs.com/sites/default/files/files/2021-
08/SD20-029-Video-BasedSafetyeBook-US-GW-v12.pdf.
[6] wexinc, “dash-cameras-a-powerful-fleet-management-
tool-to-improve-driver-efficiency-and-safety,”
https://www.wexinc.com/insights/blog/fleet/dash-cameras-a-powerful-
fleet-management-tool-to-improve-driver-efficiency-and-safety/.
[7] verizonconnect, “smart-video-modern-fleet-safety,”
https://www.verizonconnect.com/resources/article/smart-video-modern-
fleet-safety/.
[8] netradyne, “netradyne,” https://www.netradyne.com/.
[9] M. Sabri and A. Fauza, “Analysis of vehicle braking behaviour and
distance stopping,” in IOP Conference Series: Materials Science and
Engineering, vol. 309, 2019.
[10] J. Mackenzie and R. Anderson, “The potential effects of electronic sta-
bility control interventions on rural road crashes in australia: simulation
of real world crashes,” in Australasian Road Safety Research, Policing
and Education Conference, 2009.
K. Zhao, Q. Han, C.-B. Zhang, J. Xu, and M.-M. Cheng, “Deep hough
transform for semantic line detection,” IEEE Transactions on Pattern
Analysis and Machine Intelligence, vol. 44, no. 9, pp. 4793–4806, 2022.
[12] D. Wu, M.-W. Liao, W.-T. Zhang, X.-G. Wang, X. Bai, W.-Q. Cheng,
and W.-Y. Liu, “YOLOP: You only look once for panoptic driving
perception,” Machine Intelligence Research, vol. 19, no. 6, pp. 550–562,
nov 2022.
[13] S. Xie, C. Sun, J. Huang, Z. Tu, and K. P. Murphy, “Rethinking
spatiotemporal feature learning: Speed-accuracy trade-offs in video
classification,” in European Conference on Computer Vision, 2017.
[14] R. Matsumi, P. Raksincharoensak, and M. Nagai, “Study on
autonomous intelligent drive system based on potential field
with hazard anticipation,” Journal of Robotics and Mechatronics,
vol. 27, no. 1, pp. 5–11, 02 2015. [Online]. Available:
https://cir.nii.ac.jp/crid/1390001288150507520
[15] J. Fang, D. Yan, J. Qiao, J. Xue, and H. Yu, “Dada: Driver attention
prediction in driving accident scenarios,” 2023.
[16] S. Haresh, S. Kumar, M. Z. Zia, and Q.-H. Tran, “Towards anomaly
detection in dashcam videos,” 2020.
[17] D.-D. Pham, M.-S. Dao, and T.-B. Nguyen, “A cross-modal attention
model for fine-grained incident retrieval from dashcam videos,” in MMM
2023. Berlin, Heidelberg: Springer-Verlag, 2023.
[18] H. Pradana, M.-S. Dao, and K. Zettsu, “Augmenting ego-vehicle for
traffic near-miss and accident classification dataset using manipulating
conditional style translation,” 2022 International Conference on Digital
Image Computing: Techniques and Applications (DICTA), pp. 1–8, 2022.
[19] H. Pradana, “An end-to-end online traffic-risk incident prediction in
first-person dash camera videos,” Big Data and Cognitive Computing,
vol. 7, no. 3, 2023. [Online]. Available: https://www.mdpi.com/2504-
2289/7/3/129