Car Dashboard using Unity

By: Nakia Kenon, Chunyu Yang, Xiaotian Liu

Overview

1. Project Background
2. Packages Used
3. Live Weather Forecast + Map
4. Speed Limit Sign Detection
5. Traffic Light Object Detection
6. Challenges
7. Going Forward/Conclusion
Project Background

Goal: Create a simulated car dashboard that

would have some of the features that we found
were important to a driver’s experience. This
semester we implemented three of the items
from our original ideas; live weather forecast,
traffic light detection, and speed limit sign
detection.

Inspiration: We took a lot of advice from Frank

about what we could implement in a semester,
and had to rule out quite a few of the ideas from
our list.
Packages Used
Live Weather Forecast

There were three main steps that I took to get a live weather forecast on our car
window.

1. Identifying which data source was appropriate.

2. Making API requests in Unity
3. Figuring out how elements in Unity could be dynamically updated to reflect
the live weather changes.

(And a lot of trial and error throughout)

OpenWeatherMap API

OpenWeatherMap was an amazing weather resource for this task and for any
future projects in which I might need weather data. Not only do they have current
weather forecasts, but they also have historical weather data, weather maps, air
pollution data, and a slew of other important data sources that pertain to the
weather.

Syntax of API call:

I only needed the hourly predictions, but there were many other categories I could
have worked with for the forecast.
Working with JSON

In C# you cannot access the elements of JSON directly, you have to unmarshal
them into a struct. To do this, a separate struct has to be created that lays out the
structure of the JSON response.
Looking at sections of my struct file:
Applying the data to gameobjects

Once the format of the JSON response is properly reflected in the struct file, we
can use this to unmarshal the data.
Mapbox MiniMap

Using Mapbox's unity package, we can add a small map to our front window.
Mapbox does have a marker gameobject that can walk around the map using
latitude and longitude but the coordinate file must follow a specific syntax to do so.
I think the next step for this map would be creating a coordinate file to have the
player marker move around in NYC.

Marker moving around in Helsinki, Finland:

Result

Combining everything together:

Traffic Light Detection

Two methods that I have tried on:

- Color detection (color + contour, OpenCV)
- Object detection (image of traffic lights showing a red/green signal,
OpenCV, YOLOv3)
Color Detection (OpenCV)

For each frame that is catched:

- Set HSV value range for colors

- Gaussian Blur
- Convert to HSV and erode
- Remove background
- Find Contours (max in range)
- Draw and sign
Object Detection (YOLOv3)

- Real-time object detection

system
- Effective and efficient for traffic
detection autonomous driving
system
- Train Set: 530 red, 430 green,
230 yellow
- Test with traffic video in a static
environment
- Move forward for practical and
more complex real-time traffic
environment
First try on Opencv

Stop sign detect and replace it with a smiley face

Dashboard Workflow
Speed limit sign detection

Python:

Speed limit sign detection

Digit detection

Socket Server/Client communication

Unity/C#:

Socket Server/Client communication

TextMeshPro display
Speed limit sign detection

YOLOv5 🚀 is a family of compound-scaled object detection models trained on

the COCO dataset, and includes simple functionality for Test Time Augmentation
(TTA), model ensembling, hyperparameter evolution, and export to ONNX,
CoreML and TFLite.
Speed limit sign detection

Train my own custom dataset with YOLOv5

Labelimg: LabelImg is a graphical image annotation tool. It is written in Python

and uses Qt for its graphical interface.
Train a YOLOv5s model on COCO128 dataset

COCO stands for Common Objects in Context

COCO is a large-scale object detection, segmentation, and captioning dataset.

COCO has several features:

● Object segmentation
● Recognition in context
● Superpixel stuff segmentation
● 330K images (>200K labeled)
● 1.5 million object instances
● 80 object categories
● 91 stuff categories
● 5 captions per image
● 250,000 people with keypoints
Model result
metrics/mAP_0. metrics/mAP_0.
epoch train/box_loss train/obj_loss train/cls_loss metrics/precision metrics/recall 5 5:0.95

0 0.12496 0.013713 0.030684 0.0048047 0.10633 0.0012062 0.00022433

1 0.12837 0.017796 0.029785 0.0044487 0.10633 0.0010674 0.00023374

498 0.02287 0.013524 0.00076224 0.99504 0.98529 0.99471 0.84961

499 0.021631 0.010347 0.00057336 0.99415 0.98529 0.99471 0.85336

Speed limit sign detection

Python:

Speed limit sign detection

Digit detection

Socket Server/Client communication

Unity/C#:

Socket Server/Client communication

TextMeshPro display
Digit detection

Python-tesseract is an optical character recognition (OCR) tool for python. That is,
it will recognize and “read” the text embedded in images.

Python-tesseract is a wrapper for Google’s Tesseract-OCR Engine.

Speed limit sign detection

Python:

Speed limit sign detection

Digit detection

Socket Server/Client communication

Unity/C#:

Socket Server/Client communication

TextMeshPro display
Socket Server/Client Communication

Socket programming is a way of connecting two nodes on a network to

communicate with each other. One socket(node) listens on a particular port at an
IP, while the other socket reaches out to the other to form a connection. The
server forms the listener socket while the client reaches out to the server.
Demo video
Challenges

Hardware limitation

Difficulty of transmitting data from python to Unity

Learning C#

Figuring out how to apply scripts to gameobjects in order to manipulate them

Difficulty of combining our code

Plan for fall semester

Reconstruct 3D world

Object detection

Improve map