IoT Based Voice Controlled Autonomous Robot

For Mining
By

HANS TIWARI 16BLC1071

AKSHITHA REDDY 16BLC1111
AJAY MARAMPALLY 16BLC1076

A project report submitted to

Dr. Vetrivelan P

SCHOOL OF ELECTRONICS ENGINEERING

in partial fulfilment of the requirements for the course of

ECM4002 – IOT SYSTEM DESIGN

in

B.T ech. ELECTRONICS AND COMPUTER ENGINEERING

VIT UNIVERSITY, CHENNAI

Vandalur – Kelambakkam Road

Chennai – 600127

October 2018
 BONAFIDE CERTIFICATE

Certified that this project report entitled “IOT BASED VOICE CONTROLLED
AUTONOMOUS ROBOT FOR MINING” is a bonafide work of HANS TIWARI
(16BLC1071), AKSHITHA REDDY (16BLC1111) and AJAY MARAMPALLY
(16BLC1076) who carried out
the Project work under my supervision and guidance.

Dr. Vetrivelan P

Associate Professor

School of Electronics Engineering (SENSE),

VIT University, Chennai

Chennai – 600 127.

 ABSTRACT

In this project, a voice-controlled system for autonomous robots is proposed as a project

based on microcontroller. The proposed system consists of a microcontroller and a
voice recognition software that can recognize a limited number of voice patterns.

The commands of autonomous robots are classified and are organized such that one
voice recognition processor can distinguish robot commands under each directory.
Thus, the proposed system can distinguish more voice commands than one voice
recognition processor can.

The main aim of our project is to implement a low cost, reliable and voice controlled
autonomous robot for mining that can be used to remotely control the rover.

The main objective of this project is

 To control a robot from anywhere in the world using Alexa, Raspberry Pi and
ESP-12E (or any ESP board).

 To be able to control the movement of a robot via voice.

 To reduce the workload and human effort.

 To pave the way towards smart technologies.

 ACKNOWLEDGEMENT

We wish to express our sincere thanks and deep sense of gratitude to our project guide,
Dr. Vetrivelan P, Associate Professor, School of Electronics Engineering, for his
consistent encouragement and valuable guidance offered to us in a pleasant manner
throughout the course of the project work.

We are extremely grateful to the Dean of the School of Electronics Engineering,

VIT Chennai, for extending the facilities of the School towards our project and for her
unstinting support.

We express our thanks to our Programme Chair Prof. Susan Elias and Co-Chair
Prof. Reena Monica P for their support throughout the course of this project.

We also take this opportunity to thank all the faculty of the School for their support
and their wisdom imparted to us throughout the course.

We thank our parents, family, and friends for bearing with us throughout the course
of our project and for the opportunity they provided us in undergoing this course in such
a prestigious institution.
 CONTENTS
SERIAL NAME PAGE NO.
NO.

ABSTRACT 3

ACKNOWLEDGEMENT 4

1 INTRODUCTION 6

1.1 OBJECTIVES AND GOALS 6

2 DESIGN 7

2.1 BLOCK DIAGRAM 7

2.2 HARDWARE COMPONENTS 7

2.3 PROCESS FLOW 7

3 PROCEDURE 8

3.1 ADAFRUIT ACCOUNT 8

3.2 LINK ALEXA TO ADAFRUIT USING IFTTT 12

3.3 ESP12 CODE EXPLANATION 16

3.4 WORKING 20

4 SOFTWARE IMPLEMENTATION 21

5 APPLICATION 23

6 REFERENCES 23
 1. INTRODUCTION
In this project, a voice-controlled system for autonomous robots is proposed as a project
based on microcontroller. The proposed system consists of a microcontroller and a
voice recognition software that can recognize a limited number of voice patterns.

The commands of autonomous robots are classified and are organized such that one
voice recognition processor can distinguish robot commands under each directory.
Thus, the proposed system can distinguish more voice commands than one voice
recognition processor can.

In this project, we are going to form a voice-controlled robot car. we use Amazon Alexa
for voice command. we developed a custom skill for the robot. When we say, "Alexa,
Trigger move forward" Alexa trigger the AWS Lambda and Lambda function sends a
command to an MQTT server. For making the car we used Arduino Uno and Raspberry
Pi.

Raspberry Pi receives the MQTT message from the server and sends it to Arduino using
the USB port. Arduino controls the motor according to the received command.
Similarly, the commands “Alexa, Trigger move backward” and the command “Alexa,
Trigger stop” works.

1.1 OBJECTIVES AND GOALS

 This project aims to control a robot from anywhere in the world using Alexa,
Raspberry Pi and ESP-12E (or any ESP board).

 To be able to control the movement of a robotic car through the human voice.

 To reduce the workload and human effort.

 To pave the way towards smart technologies.

 2. DESIGN

2.1 BLOCK DIAGRAM

2.2 HARDWARE COMPONENTS

 Raspberry Pi with AVS installed in it

 USB 2.0 mic /Webcam
 ESP-12E
 Relay module
 RC car

2.3 PROCESS FLOW

So, the process is something like this. First, input is given to Raspberry Pi through
the USB Mic. Now, this recording is sent to Alexa voice services and after voice
recognition, AVS sent the data to IFTTT and it triggers the condition in IFTTT.
According to the recipe, IFTTT will send the command to Adafruit IO which is the
MQTT broker to perform an action. Then ESP12e will receive the data from Adafruit
IO via MQTT protocol and LED will turn ON/OFF according to the command.
 3. PROCEDURE

3.1 SETTING UP ADAFRUIT ACCOUNT FOR COMMUNICATION

First, we will make a feed in Adafruit IO. Feed stores the data sent by IFTTT. To make
feed follow these steps:

Step 1:- Login to Adafruit IO with your credentials or Sign up if you don’t have an
account.

Step 2:- Click on My account -> Dashboard

Step 3:-Click on Actions and Create a New Dashboard.

Step 4:- Give name and description to your feed and click on Create.

Step 5:- Click on Key button and note down the AIO Keys, we will use this key in our
code.
Step 6:- Click on ‘+’ button to create a new block and click on Toggle button.

Step 7:- Now, Enter Name of Feed and click on Create. Then Select the feed and click
on Next step.
Step 8:- In block settings, Write ‘1’ in Button ON text field and ‘0’ in Button OFF
Text field.

Step 9:- Feed is successfully created.

3.2 LINK ALEXA TO ADAFRUIT USING IFTTT

Step 1:- Login to IFTTT with your credentials or Sign Up if you don’t have an account
on it

Step 2:- On My Applets, Click on New Applet

Step 3:- Click on +this
Step 4:- Search Amazon Alexa and click on it, sign in with your amazon developer
account details.

Step 5:- Choose the trigger, Say a specific phrase

Step 6:- Provide “turn on the light” as the phrase, click on Create Trigger.

Step 7:- Click on +that

Step 8:- Search Adafruit and click on it.

Step 9:- Login to Adafruit Account using your credentials. Click on Send data to
Adafruit.
3.3 ESP12 CODE EXPLANATION

First, we need a Adafruit Mqtt library which can be downloaded. Just open Arduino
IDE. Go to examples -> adafruit mqtt library -> mqtt_esp8266
  First, we included all the libraries for ESP8266WIFI and Adafruit MQTT.
#include <ESP8266WiFi.h>
#include "Adafruit_MQTT.h"
#include "Adafruit_MQTT_Client.h"

 We defined the SSID and Password for your Wi-Fi, from which you want to
connect your ESP-12e.

#define WLAN_SSID "xxxxxxxx"

#define WLAN_PASS "xxxxxxxxxxx"

 This section defines the Adafruit server and server port which is fixed as
“io.adafruit.com” and “1883” respectively.

#define AIO_SERVER "io.adafruit.com"

#define AIO_SERVERPORT 1883

 Replace below fields with your username and AIO keys which you have copied
from Adafruit site while making the Feed.
#define AIO_USERNAME "********"
#define AIO_KEY "******************************"

 Then create an ESP8266 WiFiClient class to connect to the MQTT server.

WiFiClient client;

 Setup the MQTT client class by passing in the WiFi client and MQTT server and
login details.

Adafruit_MQTT_Client mqtt(&client, AIO_SERVER, AIO_SERVERPORT,

AIO_USERNAME, AIO_KEY);

 Setup a feed called 'light' for subscribing to changes.

Adafruit_MQTT_Subscribe light = Adafruit_MQTT_Subscribe(&mqtt,

AIO_USERNAME "/feeds/light");

 In setup function, we declare PIN of ESP-12e on which you want to get output. I
am using D0 pin as output. Then, we connect ESP-12e to Wi-fi access point.

void setup() {
Serial.begin(115200);
delay(10);
pinMode(D0, OUTPUT);
Serial.println(F("Adafruit MQTT demo"));
// Connect to WiFi access point.
Serial.println(); Serial.println();
Serial.print("Connecting to ");
Serial.println(WLAN_SSID);
WiFi.begin(WLAN_SSID, WLAN_PASS);
while (WiFi.status() != WL_CONNECTED) {
….
….
…
Setup MQTT subscription for light feed.
mqtt.subscribe(&light);
}

 In loop function, we will ensure that the connection to the MQTT server is alive
using MQTT_connect(); function.

void loop() {
MQTT_connect();

 Now, we subscribe our ‘light’ feed and get the string from adafruit IO and
convert this string to number using atoi();function and write this number to
PIND0 using digitalWrite(); function.

Adafruit_MQTT_Subscribe *subscription;
while ((subscription = mqtt.readSubscription(5000))) {
if (subscription == &light) {
Serial.print(F("Got_light: "));
Serial.println((char *)light.lastread);
uint16_t num = atoi((char *)light.lastread);
digitalWrite(16,num);
}

3.4 WORKING:

 Connect your ESP-12E with the laptop and upload below code (don’t forget to
edit your credentials in the code).

 Connect a LED or relay to pin D0. Now, make sure your Alexa service is running
on your RPi.

 To give any command we need to wake up Alexa service by calling “Alexa” each
time we want to send a command.

 Once we hear the beep, say “Alexa Trigger move forward.” we can see the robot
starts moving forward within a moment. And then if we say “Alexa Trigger move
backward”, the robot moves backward and to stop it we use the command “Alexa
Trigger stop”.
 4. SOFTWARE IMPLEMENTATION

Program code

#include <ESP8266WiFi.h>
#include "Adafruit_MQTT.h"
#include "Adafruit_MQTT_Client.h"
#define WLAN_SSID "**********"
#define WLAN_PASS "**********"
#define AIO_SERVER "io.adafruit.com"
#define AIO_SERVERPORT 1883
#define AIO_USERNAME “**********"
#define AIO_KEY "**********************"
Adafruit_MQTT_Client mqtt(&client, AIO_SERVER, AIO_SERVERPORT,
AIO_USERNAME, AIO_KEY);
Adafruit_MQTT_Subscribe light = Adafruit_MQTT_Subscribe(&mqtt,
AIO_USERNAME "/feeds/light");
void MQTT_connect();
void setup() {
Serial.begin(115200);
delay(10);
pinMode(D0, OUTPUT);
Serial.println(F("Adafruit MQTT demo"));
// Connect to WiFi access point.
Serial.println(); Serial.println();
Serial.print("Connecting to ");
Serial.println(WLAN_SSID);
WiFi.begin(WLAN_SSID, WLAN_PASS);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
Serial.println();
Serial.println("WiFi connected");
Serial.println("IP address: "); Serial.println(WiFi.localIP());
// Setup MQTT subscription for light feed.
mqtt.subscribe(&light);
}
uint32_t x=0;
void loop() {
MQTT_connect();
Adafruit_MQTT_Subscribe *subscription;
while ((subscription = mqtt.readSubscription(5000))) {
if (subscription == &light) {
Serial.print(F("Got_light: "));
Serial.println((char *)light.lastread);
uint16_t num = atoi((char *)light.lastread);
digitalWrite(D0,num);
}
}
void MQTT_connect() {
int8_t ret;
// Stop if already connected.
if (mqtt.connected()) {
return;
}
Serial.print("Connecting to MQTT... ");
uint8_t retries = 3;
while ((ret = mqtt.connect()) != 0) { // connect will return 0 for connected
Serial.println(mqtt.connectErrorString(ret));
Serial.println("Retrying MQTT connection in 5 seconds...");
mqtt.disconnect();
delay(5000); // wait 5 seconds
retries--;
if (retries == 0) {
// basically die and wait for WDT to reset me
while (1);
}
}
Serial.println("MQTT Connected!");
}
Code for Cooja Simulation:
#include "net/rime.h"

#include "random.h"

#include "lib/memb.h"

#include "node

-id.h"

#include "lib/list.h"

#include "net/rime.h"

#include <stdio.h>

# include "contiki.h"

struct service {

struct service *next;

rimeaddr_t service_provider;

char service_name[15];

};

struct broadcast_message {

char service_name[15];

uint8_t seqno;

};

#define MAX_SERVICES 32

MEMB(services_memb, struct service,

MAX_SERVICES);

LIST(services_list);

static struct broadcast_conn broadcast;

static struct unicast_conn unicast;

/*----------------------------------------------
-----------------------------*/

PROCESS(service_advertisement_process, "Service

Advertisment for Internet of Things");

PROCESS(service_request_process, "Service

Request for Internet of Things");

/*----------------------------------------------

------------------------------*/

AUTOSTART_PROCESSES(&service_advertisement_proce

ss, &service_request_process);

/*---Broadcast functions -----------------------

-------------------------------*/

static void

broadcast_recv(struct broadcast_conn *c, const

rimeaddr_t *from)

struct broadcast_message *m;

struct service *e;

m = (struct broadcast_message

*)packetbuf_dataptr();

/* An service advertisement is received, If we

have a record for it we update it, otherwise we

should add it to the list */

for(e = list_head(services_list); e != NULL; e = e->next) {

if(rimeaddr_cmp(from, &e

->service_provider)) {

break;
}

/* The service provider was not found in the

list, so we add a new entry by

allocating memory and fill in the necessary

fields, and add it to the list. */

if(e==NULL)

e = memb_alloc(&services_memb);

16

if(e == NULL)

return;

rimeaddr_copy(&e

->service_provider, from);

strcpy(e

->service_name, m

->service_name);

list_add(services_list, e);

printf("Service advertisement received from

%d.%d: '%s'

\n",

from

->u8[0], from

->u8[1], m

->service_name);

}
static const struct broadcast_callbacks

broadcast_call = {broadcast_recv};

/***** Unicast function

*****************************************/

static void

rec_req_srv(struct unicast_conn *c, const

rimeaddr_t *from)

printf("received service request from %d

\n",

from

->u8[0]);

static const struct unicast_callbacks

unicast_callbacks = {rec_req_srv};

/*----------------------------------------------

-----------------------------*/

PROCESS_THREAD(service_advertisement_process,

ev, data)

static struct etimer et;

struct broadcast_message msg;

static char m[15];

int service_types = 3;

PROCESS_EXITHANDLER(broadcast_close(&broadcast);

)
PROCESS_BEGIN();

list_init(services_list);

broadcast_open(&broadcast, 129,

&broadcast_call);

while(1) {

/* Delay 2

-4 seconds */

etimer_set(&et, CLOCK_SECOND * 4 + random_rand()

% (CLOCK_SECOND * 4));

PROCESS_WAIT_EVENT_UNTIL(etimer_expired(&et));

int nid = node_id;

nid = nid%service_types;

if(nid==0)

nid = service_types;

char m[15] = "Service

-";

char dg[5]; sprintf(dg,

"%d", nid); strcat(m,

dg);

strcpy(msg.service_name, m);

packetbuf_copyfrom(&msg, sizeof(struct

broadcast_message));

broadcast_send(&broadcast);

printf("advertisement sent for %s

\n",

msg.service_name);
}

PROCESS_END();

/*------------------ Service Request------------

---------------------------*/

17

PROCESS_THREAD(service_request_process, ev,

data)

char* req_service_name = "Service

-2";

PROCESS_EXITHANDLER(unicast_close(&unicast);)

PROCESS_BEGIN();

unicast_open(&unicast, 146, &unicast_callbacks);

while(1) {

static struct etimer et;

struct service *n;

etimer_set(&et, CLOCK_SECOND * 8 + random_rand()

% (CLOCK_SECOND * 8));

PROCESS_WAIT_EVENT_UNTIL(etimer_expired(&et));

if(list_length(services_list) > 0)

struct service *selected_service;

for(n = list_head(services_list); n != NULL; n = n->next)

int cmp = strcmp(req_service_name, n

-
>service_name);

if(cmp == 0) {

selected_service = n;

printf("examined node %d (%s)

\n", n

>service_provider.u8[0], n

->service_name);

if(selected_service != NULL)

printf("Sent request for (%s) to %d.%d

\n",

selected_service

->service_name,

selected_service

->service_provider.u8[0],

selected_service

->service_provider.u8[1]);

unicast_send(&unicast, &selected_service

>service_provider);

} // end if

Else
printf("Service list is Empty.

\n");

PROCESS_END();

Cooja Simulation Result:

Adding Motes:

Service discovery protocol in action:

 31

Mote Output Window

5. APPLICATIONS

 The range of this project is a lot. So, the robot can be used for mining and
other applications by improvising it a bit.

 The project can be used for remote actuation and hardware can be modified
according to need.

 It decreases human effort to do a work by just making it do work by human

voice.

 It has a wide variety of applications and leads to the formation of smart

devices which can thereby lead to smart homes and thereby smart cities.
 32

6. REFERENCES
1. https://circuitdigest.com/microcontroller-projects/iot-based-alexa-voice-
controlled-led-using-raspberry-pi-and-esp12
2. https://www.hackster.io/amazon-alexa/projects
3. https://ieeexplore.ieee.org/document/8409265/
4. https://humanizing.tech/amazons-secret-self-driving-car-project-
70d20fa859f
5. https://www.oreilly.com/ideas/how-to-build-an-autonomous-voice-
controlled-face-recognizing-drone-for-200
6. https://www.amazon.com/Make-Raspberry-Pi-Controlled-Robot-
Building/dp/1457186039
7. https://circuitdigest.com/microcontroller-projects/raspberry-pi-amazon-
echo