Sign In (https://www.hackster.io/users/auth/arduino?

current_site=arduino&setup=true&redirect_to=%2Fprojecthub%2FVarun_Chandrashekhar%2Foptimized-data-logger-

3000-datapoints-sec-2ff5a2)

We use cookies Ė
Optimized Data Logger (3000
Our websites use cookies (also from third parties) for functional and analytical purposes, and to show you personalised

Datapoints/sec) © MIT
advertisement. You can adjust this in Cookie Settings or learn more by reading our cookie policy
(https://www.arduino.cc/en/cookie-policy).

(http://opensource.org/licenses/MIT)
ONLY REQUIRED ACCEPT ALL
 If you're trying to log data from sensors fast, this device is promising. I have
optimized the code for fast data logging for any sensor.

data collection (/projecthub/projects/tags/data+collection)

monitoring (/projecthub/projects/tags/monitoring)

tracking (/projecthub/projects/tags/tracking)

2,991 VIEWS 8 COMMENTS 10 RESPECTS

COMPONENTS AND SUPPLIES 

LED (generic)
(/projecthub/products/buy/427
× 1 
51?
( (/proj (/projecthub/produc
s=BAhJIhMzNzEwNDgsUHJvamV
/ ecthu ts/buy/42751?
jdAY6BkVG%0A)
p b/pro s=BAhJIhMzNzEwND
(/projecthub/pro
r ducts gsUHJvamVjdAY6BkV
ducts/buy/42751
o /buy/ G%0A)
?
s=BAhJIhMzNzEw j 42751
e ?
NDgsUHJvamVjd
c s=BA
AY6BkVG%0A)
t hJIhM
h zNzE
u wNDg
b sUHJv
/ amVjd
p AY6B
r kVG%
o 0A)
d
u
c
t
s
s
/
b
u
y
/
4
2
7
5
1
?
s
=
B
A
h
JI
h
M
z
N
z
E
w
N
D
g
s
U
H
J
v
a
m
V
j
d
A
Y
 Y
6
B
k
V
G
%
0
A
)

Arduino Mega 2560

(/projecthub/products/buy/145?
× 1 
s=BAhJIhMzNzEwNDgsUHJvamV
( (/proj (/projecthub/produc
jdAY6BkVG%0A)
/ ecthu ts/buy/145?
(/projecthub/pro p b/pro s=BAhJIhMzNzEwND
r ducts gsUHJvamVjdAY6BkV
ducts/buy/145?
o /buy/ G%0A)
s=BAhJIhMzNzEw
NDgsUHJvamVjd j 145?
AY6BkVG%0A) e s=BA
c hJIhM
t zNzE
h wNDg
u sUHJv
b amVjd
/ AY6B
p kVG%
r 0A)
o
d
u
c
t
s
/
b
u
y
/
1
1
4
5
?
s
=
B
A
h
JI
h
M
z
N
z
E
w
N
D
g
s
U
H
J
v
a
m
V
j
d
A
Y
6
B
k
V
G
%
0
A
 A
)

Graphic OLED, 128 x 64

(/projecthub/products/buy/483
× 1 
44?
( (/proj (/projecthub/produc
s=BAhJIhMzNzEwNDgsUHJvamV
/ ecthu ts/buy/48344?
jdAY6BkVG%0A)
(/projecthub/pro p b/pro s=BAhJIhMzNzEwND
r ducts gsUHJvamVjdAY6BkV
ducts/buy/4834
4? o /buy/ G%0A)
s=BAhJIhMzNzEw j 4834
e 4?
NDgsUHJvamVjd
AY6BkVG%0A) c s=BA
t hJIhM
h zNzE
u wNDg
b sUHJv
/ amVjd
p AY6B
r kVG%
o 0A)
d
u
c
t
s
/
b
u
y
/
4
8
3
4
4
?
s
=
B
B
A
h
JI
h
M
z
N
z
E
w
N
D
g
s
U
H
J
v
a
m
V
j
d
A
Y
6
B
k
V
G
%
0
A
)

×
(
/
 /
Flash Memory Card, SD Card p
(/projecthub/products/buy/481 r
o 1 
39?
j (/proj (/projecthub/produc
s=BAhJIhMzNzEwNDgsUHJvamV
ecthu ts/buy/48139?
jdAY6BkVG%0A) e
b/pro s=BAhJIhMzNzEwND
(/projecthub/pro c
ducts/buy/48139 t ducts gsUHJvamVjdAY6BkV
? h /buy/ G%0A)
s=BAhJIhMzNzEw u 48139
NDgsUHJvamVjd b ?
AY6BkVG%0A) / s=BA
hJIhM
p
r zNzE
o wNDg
d sUHJv
u amVjd
c AY6B
kVG%
t
s 0A)
/
b
u
y
/
4
8
1
3
9
?
s
=
B
A
h
JI
h
M
z
 z
N
z
E
w
N
D
g
s
U
H
J
v
a
m
V
j
d
A
Y
6
B
k
V
G
%
0
A
)

Elegoo SD Card Reader/Writer × 1

Single Turn Potentiometer- 100k

ohms
×
(/projecthub/products/buy/564
(
70?
/
s=BAhJIhMzNzEwNDgsUHJvamV
p
jdAY6BkVG%0A)
r
o
 o
j
e
c 1 
t (/proj (/projecthub/produc
h ecthu ts/buy/56470?
u b/pro s=BAhJIhMzNzEwND
(/projecthub/pro
b ducts gsUHJvamVjdAY6BkV
ducts/buy/56470
? / /buy/ G%0A)
s=BAhJIhMzNzEw p 5647
NDgsUHJvamVjd r 0?
AY6BkVG%0A) o s=BA
d hJIhM
u zNzE
c wNDg
t sUHJv
s amVjd
/ AY6B
b kVG%
u 0A)
y
/
5
6
4
7
0
?
s
=
B
A
h
JI
h
M
z
N
z
E
 E
w
N
D
g
s
U
H
J
v
a
m
V
j
d
A
Y
6
B
k
V
G
%
0
A
)

DFRobot Gravity:Digital Push

Button (Yellow)
× 
(/projecthub/products/buy/402
( (/projecthub/produc
12?
/ ts/buy/40212?
s=BAhJIhMzNzEwNDgsUHJvamV
p s=BAhJIhMzNzEwND
(/projecthub/pro jdAY6BkVG%0A)
r gsUHJvamVjdAY6BkV
ducts/buy/40212
? o G%0A)
s=BAhJIhMzNzEw j
NDgsUHJvamVjd e
AY6BkVG%0A) c
t
h
u
u
b
/
p 1
r (/proj
o ecthu
d b/pro
u ducts
c /buy/
t 40212
s ?
/ s=BA
b hJIhM
u zNzE
y wNDg
/ sUHJv
4 amVjd
0 AY6B
2 kVG%
1 0A)
2
?
s
=
B
A
h
JI
h
M
z
N
z
E
w
N
D
g
s
U
 U
H
J
v
a
m
V
j
d
A
Y
6
B
k
V
G
%
0
A
)

SparkFun Breadboard Power

Supply Stick 5V/3.3V
× 1 
(/projecthub/products/buy/646
( (/proj (/projecthub/produc
?
/ ecthu ts/buy/646?
s=BAhJIhMzNzEwNDgsUHJvamV
p b/pro s=BAhJIhMzNzEwND
(/projecthub/pro jdAY6BkVG%0A)
r ducts gsUHJvamVjdAY6BkV
ducts/buy/646?
o /buy/ G%0A)
s=BAhJIhMzNzEw
j 646?
NDgsUHJvamVjd
e s=BA
AY6BkVG%0A)
c hJIhM
t zNzE
h wNDg
u sUHJv
b amVjd
/ AY6B
p kVG%
r 0A)
o
d
d
u
c
t
s
/
b
u
y
/
6
4
6
?
s
=
B
A
h
JI
h
M
z
N
z
E
w
N
D
g
s
U
H
J
v
a
m
V
j
d
 d
A
Y
6
B
k
V
G
%
0
A
)

NECESSARY TOOLS AND MACHINES 


Wire Wrap Tool, Pencil (/projecthub/products/
(/projecthub/pro (/projecthub/products/buy/56390? buy/56390?
ducts/buy/5639 s=BAhJIhMzNzEwNDgsUHJvamVjdAY6BkV s=BAhJIhMzNzEwNDgs
0? G%0A) UHJvamVjdAY6BkVG%0
s=BAhJIhMzNzEw A)
NDgsUHJvamVjd
AY6BkVG%0A)

Soldering iron (generic)

 
Wire Stripper & Cutter, 18-10 AWG / 0.75-
(/projecthub/products/
(/projecthub/pro 4mm² Capacity Wires buy/56321?
ducts/buy/56321 (/projecthub/products/buy/56321? s=BAhJIhMzNzEwNDgs
? s=BAhJIhMzNzEwNDgsUHJvamVjdAY6BkV
UHJvamVjdAY6BkVG%0
s=BAhJIhMzNzEw G%0A) A)
NDgsUHJvamVjd
AY6BkVG%0A)

ABOUT THIS PROJECT 

This year my research project involves running ML models on collected EMG data. For my
project, I need to be able to collect as many datapoints (EMG signals) as possible in a set
time frame. Due to this need, I have created a data logger that can record and store EMG
data values as fast as possible. Currently, it can measure up to 3,000 data points per
second (Calculated using Arduino Serial Monitor Time Output).
The code I have developed can easily be used for any sensor so I thought it was worth
sharing.
This code can:
Store recorded values onto SD card
Display readings on an OLED display
Has a button to start recordings for a set amount of time
Has a potentiometer to change the file name that the data is saved to on the SD
card
This code can be made better. Due to my lack of knowledge in Port Manipulation, I don't
know how to code without using Arduino based functions such as analogRead() or
String(). If you do know how I could optimize my code further please let me know. I
would be happy to update my project.
CODE 

Data Logger
0 (https://github.com/Varun-Chandrashekhar/Fast_Data_Logger/watchers)
Varun-Chandrashekhar (https://github.com/Varun-Chandrashekhar) / Fast_Data_Logger
0 (https://github.com/Varun-Chandrashekhar/Fast_Data_Logger/forks)
(https://github.com/Varun-Chandrashekhar/Fast_Data_Logger)

This code is optimized for fast data logging to SD cards for any sensor. It can record up to 3000 data points every
second. — Read More (https://github.com/Varun-Chandrashekhar/Fast_Data_Logger#readme)

Latest commit to the master branch onas

Download 11-19-2020
zip (https://github.com/Varun-Chandrashekhar/Fast_Data_Logger/zipball/master)

SCHEMATICS 

S
c
h
e
m
a
t
i
c
o
f
D
e
v
i
c
DOWNLOAD (HTTPS://HACKSTERIO.S3.AMAZONAWS.COM/UPLOADS/ATTACHMENTS/1221022/OPTIMIZED_DATA_
e
D
i
a
g
r
a
m
o
f
D
e
v
i
c
DOWNLOAD (HTTPS://HACKSTERIO.S3.AMAZONAWS.COM/UPLOADS/ATTACHMENTS/1221024/OPTIMIZED_DATA_
e
COMMENTS 

Please log in (/projecthub/users/sign_in?

id=371048&m=project&reason=comment&redirect_to=%2Fprojecthub%2FVarun_Chand
rashekhar%2Foptimized-data-logger-3000-datapoints-sec-2ff5a2%23comments) or sign
up (/projecthub/users/sign_up?
id=371048&m=project&reason=comment&redirect_to=%2Fprojecthub%2FVarun_Chand
rashekhar%2Foptimized-data-logger-3000-datapoints-sec-
2ff5a2%23comments&source=popup) to comment.

Varun Chandrashekhar (/projecthub/Varun_Chandrashekhar)

a month ago
(/pr
oject
Hi Guys! My name is Varun Chandrashekhar and I am the creator of this project. If you
have any questions feel free to email me at varunchandra1010@gmail.com
hub/
(mailto:varunchandra1010@gmail.com) or drop a comment below.
Varu
n_C
Visit me on LinkedIn: https://www.linkedin.com/in/varun-chandrashekhar/
hand
(https://www.linkedin.com/in/varun-chandrashekhar/)
rash
ekha
r)

DaD637 (/projecthub/DaD637)
a month ago
(/pr
hello
oject Varun Chandrashekhar, hope all is well. very interesting project, i see you have
listed
hub/ a 2 gigabyte sd card.
roughly,
DaD how many samples does this card hold? and how much time has passed before
the
637)sd card is full?
thanks
allan pickett
1 thank

Varun Chandrashekhar (/projecthub/Varun_Chandrashekhar)

a month ago
(/pr
Although I have listed a 2GB SD card I actually used only a 1GB SD card. This was
oject
partly
hub/ because I wasn't able to format the larger size SD cards for the SD card
reader/writer. I don't exactly know how many sets of samples the card can hold,
Varu
but
n_Cfor my purposes, I was able to store about 60 seconds worth of data and it
seems to work just fine. I personally haven't tried to max out the SD card.
hand
rash
I'm sure it's possible to calculate how many samples a card can hold before maxing
ekhabut it would be highly dependent on what type of values you're putting in the
out,
r)
card. For example, floats take up more space than ints, and unsigned ints take up
even more space etc.
1 thank

Varun Chandrashekhar (/projecthub/Varun_Chandrashekhar)

a month ago
 I(/pr
hope that helps!
oject
1 thank
hub/
Varu
n_C
hand
rash DaD637 (/projecthub/DaD637)
ekha a month ago
(/pr
r)
wow,
oject a 180,000 samples. that is pretty impressive and a lot of data to go through
hub/
DaD
637)
Varun Chandrashekhar (/projecthub/Varun_Chandrashekhar)
a month ago
(/pr
Ioject
am using my device for Machine Learning applications, so I need as much data as
Ihub/
can get. The device could be made faster using port manipulation, but I am not
well
Varu versed in that.
n_C
hand
rash
ekha
DougalPlummer (/projecthub/dougal)
r)
25 days ago
(/pr
Good
oject start, well done!. If you want to go faster then the strategy is to use interrupts.
Split
hub/ the tasks between saving to the card and sampling. Probably good to investigate
circular
dou buffers.
gal)

crlarson (/projecthub/crlarson)
17 days ago
(/pr
Varun,
oject thanks for the post. Recently I built a data recorder and learned that the SD card
management
hub/ takes time to manage "blocks" by erasing/reading/writing/wear-leveling as
it is recording. This has something to do with the 512 byte block size and boundaries.
crlar
This
son) SD management will disrupt the timing of your data flow I'm thinking your EMG
data probably requires smooth time-series data to be meaningful or it will have gaps
that you won't know about. You could store the least significant byte of a millis()
reading to see evidence of gaps. As mentioned by others double cycling buffers can be
used. One buffer can be reading from the sensor while the other buffer can be writing
sensor data to the SD card. Buffers should be sized to be 512, but my RAM isn't big
enough that so I tried to make them 256 so that it would be divisible into the SD block
boundaries. This is supposed to be much faster than writing one sample at a time.
This SD management also causes a power supply voltage sag that inversely spiked my
analog sensor readings. You might find that happened to you too. Andreas Spiess (Video
#10) has a method where he uses takes the VCC readings to make adjustments to his
analog readings on the fly. It worked for me and smoothed out the spikes..
Two size ideas:
1. Arduino analog readings are 10 bit. Your rawEMG is a float (32 bits). I think the
sensor value you are recording is the 10 bit size. That would be a range from 0 to
1023, Then you then convert that into a string and add a comma for up to 5
characters total per sample. Each character needs an 8 bit byte for a total of 40 bits
per sample. It would be possible to compress that down to a 10 bit sample plus a
one byte comma, or line-feed, for 18 bits/sample. It may take some decoding to
read the mixed binary/character data.
2. A 1 G card holds 1,073,741,824 bytes, 8.6 billion bits. So if your sample size is 18 bits,
then that would equate to 470 million samples.. But I've read that there are
reserved spaces, and overadvertised cards. So even if you get 70 or 80% of this you
are able to store for a long time ( (8.6E9 bits / 18 bits /3000 samples per second /
3600 seconds/hr)*75% ~= 33 hours ), if I've got that right.
Hope that helps!

AUTHOR

(/projecthub/Varun_Chandrashekhar)

Varun Chandrashekhar (/projecthub/Varun_Chandrashekhar) 

2 PROJECTS 9 FOLLOWERS

FOLLOW (/PROJECTHUB/USERS/SIGN_UP?ID=366288&M=USER&REASON=FOLLOW&REDIRECT

PUBLISHED ON

November 20, 2020

 RESPECT PROJECT (/PROJECTHUB/USERS/SIGN_UP?ID=371048&M=ARTICLE&REASON=RESPECT&R...

  WRITE A COMMENT

 Share

MEMBERS WHO RESPECT THIS PROJECT

(/projecthub/aula-jazmati) (/projecthub/dextergomes01) (/projecthub/elegoo-

official) (/projecthub/joejoe2) (/projecthub/dougal)

and 5 others

SEE SIMILAR PROJECTS

YOU MIGHT LIKE

SIMILAR PROJECTS YOU MIGHT LIKE

(/projecthub/javidambra/energy-data-logger-3e2dba?
ref=similar&ref_id=371048&offset=0)
 Energy Data Logger (/projecthub/javidambra/energy-data-logger-
3e2dba?ref=similar&ref_id=371048&offset=0)
Project in progress by Javi D'Ambra (/projecthub/javidambra)
16,357 VIEWS 7 COMMENTS 31 RESPECTS

(/projecthub/yvesmorele/rc-loggerstation-gps-data-logger-for-rc-plane-3ea4cd?
ref=similar&ref_id=371048&offset=1)

RC LOGGERSTATION - GPS Data Logger for RC Plane

(/projecthub/yvesmorele/rc-loggerstation-gps-data-logger-for-rc-
plane-3ea4cd?ref=similar&ref_id=371048&offset=1)
Project tutorial by yvesmorele (/projecthub/yvesmorele)
4,358 VIEWS 2 COMMENTS 12 RESPECTS
 4,358 VIEWS 2 COMMENTS 12 RESPECTS

(/projecthub/sridhar-rajagopal/how-to-build-an-arduino-energy-monitor-and-data-
logger-3bf795?ref=similar&ref_id=371048&offset=2)

How to Build an Arduino Energy Monitor and Data Logger

(/projecthub/sridhar-rajagopal/how-to-build-an-arduino-energy-
monitor-and-data-logger-3bf795?
Project tutorial by Sridhar Rajagopal (/projecthub/sridhar-rajagopal)
17,805 VIEWS 4 COMMENTS 26 RESPECTS

(/projecthub/DrGee/yadl-yet-another-data-logger-5ab66f?
ref=similar&ref_id=371048&offset=3)
 YADL: Yet Another Data Logger (/projecthub/DrGee/yadl-yet-
another-data-logger-5ab66f?ref=similar&ref_id=371048&offset=3)
Project showcase by DrGee (/projecthub/DrGee)
9,743 VIEWS 3 COMMENTS 22 RESPECTS

(/projecthub/Arduino_Genuino/mkr-zero-weather-data-logger-574190?
ref=similar&ref_id=371048&offset=4)
 MKR Zero Weather Data Logger
(/projecthub/Arduino_Genuino/mkr-zero-weather-data-logger-
574190?ref=similar&ref_id=371048&offset=4)
Project tutorial by Arduino_Genuino (/projecthub/Arduino_Genuino)
34,003 VIEWS 18 COMMENTS 65 RESPECTS

(/projecthub/jedhodson/temperature-and-humidity-data-logger-breadboard-8c2285?
ref=similar&ref_id=371048&offset=5)

Temperature and Humidity Data logger - Breadboard

(/projecthub/jedhodson/temperature-and-humidity-data-logger-
breadboard-8c2285?ref=similar&ref_id=371048&offset=5)
Project tutorial by Jed Hodson (/projecthub/jedhodson)
11,513 VIEWS 2 COMMENTS 12 RESPECTS

(https://www.arduino.cc)
 Powered by
(https://www.hackster.io)