DOCUMENTATION

July 10, 2019

E-WALL
ELECTRONICS CLUB, IIT KANPUR

July 10, 2019

Contents

0.1 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
0.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
0.3 Our Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
0.4 Why this Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
0.5 Parts and Electronic components . . . . . . . . . . . . . . . . . . . . . . 5
0.5.1 Arduino Mega . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
0.5.2 16-Channel Analog Multiplexer . . . . . . . . . . . . . . . . . . . 6
0.5.3 ESP8266-01 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
0.5.3.1 Working of ESP8266-01: . . . . . . . . . . . . . . . . . . 7
0.5.4 Capacitative Sensing . . . . . . . . . . . . . . . . . . . . . . . . . 8
0.5.4.1 Projective Capacitive Touch . . . . . . . . . . . . . . . . 9
0.6 The Timeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
0.7 Gesture Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
0.7.1 Swipes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
0.7.1.1 Up and Down . . . . . . . . . . . . . . . . . . . . . . . . 17
0.7.1.2 Left And Right . . . . . . . . . . . . . . . . . . . . . . . 18
0.8 Working Applications And Future Possibilities . . . . . . . . . . . . . . 19
0.8.1 Interactive Flexible Touch Board . . . . . . . . . . . . . . . . . . 19
0.8.2 Integration with Desktopography . . . . . . . . . . . . . . . . . . 20
0.8.3 SMART Wall Game . . . . . . . . . . . . . . . . . . . . . . . . . 21
0.8.4 SMART Door Security . . . . . . . . . . . . . . . . . . . . . . . . 21
0.8.5 Wall Swarm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
0.8.6 SMART ROOM . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
0.9 Team . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
0.10 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

1
 2

0.1 Acknowledgements

We would like to thank The Electronics Club and The Science And Technology
Council, IIT Kanpur for providing us the opportunity to work on the summer
project E-Wall. We are grateful to them for providing us the resources, the motiva-
tion as well as their constant guidance for this project. We thank the co-ordinators
of the Electronics Club :
Soumya Ranjan Dash
Jay Mundra
Mudit Agrawal
Nitish Vikas Deshpande
Also we are extremely grateful to Yang Zhang, Chouchang (Jack) Yang, Scott E.
Hudson, Chris Harrison, Alanson Sample for their paper Wall++ that has been a
source of information moving through this project.
 3

0.2 Introduction
• Human environments are typified by walls - homes, offices, schools, museums, hos-
pitals and pretty much every indoor context one can imagine has walls. In many
cases, they make up a majority of readily accessible indoor surface area, and yet
they are static - their primary function is to be a wall, separating spaces and hiding
infrastructure.

• E-Wall lets us convert a normal static wall, into an interactive wall that can detect
touch and multi-touch gestures in a very simple yet effective way. It is based on
the concept of mutual capacitive sensing, which itself is classified further into many
types. Our project is based on the Active Shunt Mode.

Figure 1: Touch Input Recognition

• By gathering data about the different touch inputs made by the user we can im-
plement simple machine learning models that can recognize repetitive patterns and
gestures. Further more we can also undertake live human motion tracking, that is
predict the body posture of the user using the data collected from the E-Wall.

Figure 2: Posture Recognition

 4

0.3 Our Approach

1. The first step of the project includes us wiring down onto a particular method to
go down with our project, this includes deciding the way the input/signal is going
to be sent, as well as deciding the way we are going to receive the output.It also
includes making a very rudimentary prototype just to verify the implementation of
the different ways mentioned earlier This prototype has to be made using copper
tapes.

2. The second step includes us developing a miniature setup of the actual device that
is going to be made at the end of this project. This includes using the paint to
make the electrode pattern onto a surface and then connecting it through copper
tapes. Then we have to validate the method of sending and receiving signal with
this miniature and also develop a software to actually represent the touch gestures
pictorially. This can be done by mapping the capacitance values obtained onto
a map from 0-255 and then making a greyscale image based upon the values of
capacitance obtained from around the area where the user touches.

3. The third step includes further expanding on the scale on which the device is made
and then validating the same method on this device. However this time the no. of
electrodes has to be increased considerably so that the accuracy of the touch detec-
tion can be increased. Once this is done we can implement a simple classification
machine learning model to recognize different gestures like swipes, shapes made like
circles, squares etc. Also multiple-touch recognition has to be worked on.

4. The final step includes making the actual device and further improving the accu-
racy of the device such that other more important applications of the E-Wall can
be worked upon such as posture recognition, basic apps that are based on touch
application something like the Educomp-SmartBoard.
 5

0.4 Why this Approach

• This approach helped us continuously validate as well as improve our method of
implementation of the concept of capacitive sensing on our prototypes. By first
implementing the technique of sending and receiving signals on a smaller prototype
and improving it there we were able to do away with any problems we faced at a
smaller level. Then while implementing it on a larger scale we did not have to face
the same problems that we had dealt with earlier thus reducing the time we had to
spend debugging.

• Furthermore this helped us spend more time on decisions integral in the devel-
opment of the project, one of them being not using conductive paint on a large
prototype even when it gave us good results on a smaller prototype. Also it gave a
good platform to try out different ways to optimize our results like using resistors
of different values, also capacitors which would have been difficult to experiment
with if we had proceeded with a large prototype from the beginning.

0.5 Parts and Electronic components

Our project basically involved two main components, the conductive surface
and the circuit board. The components required for each are listed below :
Conductive Surface
1. (a) Copper Tape
(b) High (approx 10M ohm) and Low (approx 1K ohm) Valued Resistors
(c) Acrylic Board

2. Circuit Board

(a) Arduino Mega

(b) 16 Channel Multiplexer
(c) ESP 8266
(d) Jumper Wires, solder etc.
 6

Figure 3: Arduino-Mega

0.5.1 Arduino Mega

The Arduino Mega is a microcontroller board based on the ATmega2560. It has 54

digital input/output pins (of which 14 can be used as PWM outputs), 16 analog
inputs, 4 UARTs (hardware serial ports), a 16 MHz crystal oscillator, a USB con-
nection, a power jack, an ICSP header, and a reset button. It contains everything
needed to support the microcontroller; simply connect it to a computer with a USB
cable or power it with a AC-to-DC adapter or battery to get started. The Mega
2560 R3 also adds SDA and SCL pins next to the AREF. In addition, there are two
new pins placed near the RESET pin. One is the IOREF that allow the shields to
adapt to the voltage provided from the board. The other is a not connected and is
reserved for future purposes. The Mega 2560 R3 works with all existing shields but
can adapt to new shields which use these additional pins.

0.5.2 16-Channel Analog Multiplexer

We used a 16-Channel Analog Multiplexer of cd74hc4067 variant in our circuit.

The Arduino mega does not have enough inputs to correspond with our number
of electrodes. Therefore we had to use a multiplexer. This multiplexer acts as a
switch that connects the different input pins to a common pin and also sequentially
 7

Figure 4: Multiplexer

alternates between the different input pins available. So by just using 4 pins of the
Arduino we were able to control more than 7 electrodes.

0.5.3 ESP8266-01

Figure 5: ESP8266-01

The main purpose of ESP is to transmit the output, in the form of phase differences
and gain ratios, it receives from the ATMega , to the Laptop at a suitable rate. In-
sulator covering is used to decrease the interference between 80MHz signal and ESP
module while ensuring proper connection between mobile hotspot and ESP.

0.5.3.1 Working of ESP8266-01:

ESP8266-01 is used in ATP mode (connects with a hotspot as it does not behave as
a hotspot) with UDP transport protocol. UDP is a simple high speed low function-
 8

ality wrapper that interface applications to the network layer. UDP sends 8 analog
inputs in the form of strings to an interactive device connected to the hotspot(same
as that of the esp) which is then stored in an excel sheet upon which data manipu-
lation and machine learning can be applied.

0.5.4 Capacitative Sensing

• We used the technique of capacitive touch in our project in place of the com-
1.
monly used resistive touch method as it required a bending screen, while we
aimed at making a hard wall. For capacitive sensing, we implemented the pro-
jective capacitive model. The projective capacitive model uses a grid made out
of good conducting materials, like copper, and registers a touch by noticing
changes in the charges of capacitors, when the grid encounters a dielectric like
our finger.

• Resistive Touch
1,2 : Upper Surface
3,4 : Sensing Surfaces
4,5 : Resistive Sensors
6 : Inner Support
7 : Touch
8 : Detector
 9

• Now, to use this efficiently in our circuit, we used the capacitiveSensor arduino
library. Here, pins of an arduino processor are used as send and receive pins
via a simple RC circuit to detect a touch. The send pin toggles its logical
state, and this change is relayed across the circuit to the receive pin, which
changes its state as well. The time taken for this change is computed. When
a dielectric, like our finger is introduced, this time is affected, and the touch
is detected.

0.5.4.1 Projective Capacitive Touch

• We used a high value resistor to increase the time constant of the circuit (
Resistance * Capacitance) to make the time for the state changes be more
noticeable. Secondly, to register the time, we used the predefined library func-
tions :-

(a) digitalWrite(Pin, Value): This function is used to define a pin as low or

high depending on the Value.
Pin: Arduino pin value
Value: HIGH or LOW
 10

(b) millis(): It returns the time in milliseconds after the arduino board began
running the current program.

(c) pinMode(pin, Mode): This defines a pin to behave as an input or output.

Pin: Arduino pin value
Mode: INPUT, OUTPUT, OR INPUT PULLUP

(d) CapacitiveSensor(SendPin, ReceivePin): This is an instance of the capsense

library in the program and defines the send pin and the receive pin.
 11

0.6 The Timeline

• This section gives a vivid description on how we went with this project, the
problems we faced at each step and how we deliberated and overcame each of
them to satisfactorily finish our project.
• Initially we made a conductive surface of 8 electrodes that were laid out in
a 4X4 matrix and were covered by normal cellophane sheet. Then we use a
circuit to generate a sine wave and tried to send and receive the sine wave
through the electrodes. This sine wave was generated using a 555 timer along
with the right combination of resistors and capacitors.
• However this approach did not prove out to be successful as we were not able
to obtain a simple sine wave further more the signal to noise ratio for the
signal received through the electrodes was also very low due to the external
electronic noise present.

Figure 6: Initial Prototypes

 12

• In the next prototype to be used we made the conductive surface with water-
based nickel paint and then connected these electrodes to each other through
copper tape. However still the horizontal and the vertical electrodes were still
separated through plastic insulation.

Figure 7: 3x3 Prototype(Nickel painted)

• In this prototype instead of using an external circuit, we tried implementing a

sine wave using the arduino, by sending a signal through one of its analog pins.
However on continuous evaluation it was observed that the signal strength of
the signal sent through the arduino depended much upon its own power input
( laptop or Adapter ). This resulted in unforced errors that were difficult to
work upon.
• As you will see below by using the Arduino generated sine wave, we coudn’t
actually differentiate between a touch and a non touch gesture, hence this way
of taking in the data also seemed unproductive.
 13

Figure 8: Sine Wave Non Touch and Touch Data

 14

• Upon further research we tried implementing and arduino based library Capsense
into our prototype. This library’s inbuilt functions helped us quite accurately
obtain a good change in values between the two events of touch and non touch.
Thus we decided to work with this library and improved its accuracy.
• To improve the results obtained we used resistors and capacitors of different
values until we were able get the right values from our electrodes. By using
the values obtained from the horizontal and vertical electrodes we were able
to approximate the area where a touch input had been made. Thus now we
decided to convert our raw data into an image.

Figure 9: Capsense Touch Data

• As you can see in the image above the different colours correspond to the
different electrode. It clearly depicts that at any point if an electrode is touched
then the data corresponding to that particular electrode only changes and not
the other which was the case in the previous prototype.The change in the data
too is very significant as it enables us to measure the pressure with which an
 15

electrode has been touched upon. The higher the change the more the pressure
applied.
• To convert our raw data into an image we used OpenCV. This is a platform
which enables you to take data and convert it into graphics. Its integration
with Python provides considerable ease in its implementation. We mapped
our values onto a scale from 0-255 and then usig function from OpenCV were
able to convert it into a grid. On this grid Black colour signified no touch and
white signified absolute touch.
• On our next larger prototype we decide to let go of the idea of using Nickel
Paint. This was majorly because of two major reasons :
First , the paint was too costly. Implementing the grid on a larger scale using
that paint would have taken out a major plus point of our project, that is, it
had to be cost efficient.
Second, while discussing on ideas we decided on making our touch grid a
flexible grid rather that a rigid one. Paint required solid surface. To implement
our idea of a flexible touch screen we had to let go of paint and just use copper
tapes.
• Then we implemented the same gray scale method to detect touch on this
bigger prototype, as shown in the figure below. Each square represents an
area on the grid. We are working now on a 6 x 7 square grid to demonstrate
single touch as well as working on gesture recognition. This is a flexible surface
and it even then can take in touch inputs.
 16

• Now we are working on an even bigger prototype that will be much more
accurate in pin pointing the location of any touch input made onto the board.
Furthermore this one is being made on a chart paper thus ensures testing and
working on the flexibility of the board, developing more on its efficient use.

Figure 10: Current Prototype which is being worked on.

 17

0.7 Gesture Recognition

0.7.1 Swipes

Since we are able to get the position of the finger input from our touch screen, we
could perform some simple operations on them to enable them to detect gestures.
First we take the initial positions of the touch input, that is, the (xi , yi ) and the
final position (x f , y f ), then we take each coordinates absolute difference.

0.7.1.1 Up and Down

For a gesture that moves vertically, the absolute difference in the y co-ordinate
should be more than the x co-ordinate. Once this is established we take the sign of
the difference.
(y f − yi ) > 0

means it is a up swipe.

(y f − yi ) < 0

means it is a down swipe.

Figure 11: A down swipe

 18

0.7.1.2 Left And Right

For a gesture that moves sideways, the absolute difference in the x co-ordinate
should be more than the y co-ordinate. Once this is established we take the sign of
the difference.
(x f − xi ) > 0

means it is a right swipe.

(x f − xi ) < 0

means it is a left swipe.

We wrote Electronics club (E-Club) from different patterns drawn on our model

Figure 12: Electronics Club

 19

0.8 Working Applications And Future Possibilities

This section describes the varied applications of our flexible touch screen

0.8.1 Interactive Flexible Touch Board

Due to the simple and flexible nature of our touch board it can be integrated into
games and toys. Furthermore it can be calliberated with simple mobile applications
to serve as touch inputs devices for certain educational applications. Due to its high
cost efficiency it stands to be available for the masses. It can be made to recognize
simple alphabet gestures, made to calibrate with simple video games as Tetris, The
Snake etc. Its flexibility also enables us to use it in places where normal touch
screens involving glass cannot be used.

Figure 13: Using the board to draw on a laptop

Right now we are trying to convert this board into an IOT application which can
be used to control simple things like

• controlling the intensity of lights in the room.

• controlling the volume of the sound system in the room
• controlling the drapes of the room.
• setting up a pattern lock and using our board for opening and closing the
doors. This could be integrated with Smart Security as well.

just by using simple gestures made on the board.

Furthermore making a bigger application like the EducomptSmart Board can also
be another course of action, plus additional flexibility could be added to it.
 20

Figure 14: A system like this could be made

0.8.2 Integration with Desktopography

E-Club has already worked to enable hand-enabled controlling of a display screen

in its project Desktopography. There they used Depth Sensing to take in the input.
The way we can integrate our touch board into this project is by replacing depth
sensing by touch inputs. This is supposed to be done by projecting the screen onto
the dimensions of the board and then actually projecting the screen onto the board.
Then the finger input would just be like using a mouse ont the screen. This could
make the input sensing more robust and dependable with minor issues.

Figure 15: Detecting touch input and using it to navigate

 21

0.8.3 SMART Wall Game

We tried to play Iconic Snake game with our Smart Touch pad

Figure 16: Snake Game

0.8.4 SMART Door Security

We can train this model to recognise different pattern and shapes using which we
can make pattern lock like mobiles for Homes

Figure 17: SMART Security

 22

0.8.5 Wall Swarm

We made a virtual Swarm robots to make patterns drawn on our model by inspiring
from another Electronics Club project Swarm Robotics Which represents these
project can be collaborated too like desk topography

Figure 18: Wall Swarm

0.8.6 SMART ROOM

We also made an Model of Room Based on Sole Motive of this project to use Walls
to Controls Home Appliances as Internet Of Things

Figure 19: Room Model

 23

0.9 Team
• Visw Srivastava

• Prashant Raigar

• Lakshay Rastogi

• Manoj Kumar

• Ritesh Birthal

• Gaurav Jiwan

• Manish Choudhary
 24

0.10 References
• https://playground.arduino.cc/Main/CapacitiveSensor/

• http://chrisharrison.net/projects/wallplusplus/wallplusplus.pdf

• https://resenv.media.mit.edu/classarchive/MAS965/readings/SmartSkin.pdf

• https://www.touchsystems.com/resources/projected-capacitive-technology/

• https://github.com/Prashant9800/SUMMER-PROJECT-E-WALL-/blob/master/Ewall7 x7.

• https://www.youtube.com/channel/UC2jj1jWZmDU4vcplCRdpATw