Research

 Methods  
in  computer  science  
Fall  2014  

Lecture  7  
 
Omprakash  Gnawali  
September  17,  2014  
 Agenda  
Research  Conference  Updates  
Experiment  and  metric  design  exercise  
Experiments  in  uncontrolled  environment  
DescripJon  of  experiment  setup  
HW  
 Metric  Design  Exercise  
Project:  InstrucJon  system  (HCI)  
 
Goal:  Make  furniture  assembly  faster  
 
Possible  metrics  and  experiments?  
Deployment  Experiments  
RealisJc  setup  and  inputs  
 
Uncontrolled  environment  
Spans  the  parameter  space?  
Wireless  Experiments  Today  
Protocol  Comparison  Experiments  
Run  the  new  protocol  
Run  best-­‐known  prior  work  
Compare  
 
SimulaJons  +  Testbed  experiments  

7  
 Serial  Experiments  
Run  one  protocol  at  a  Jme  
Compare  the  results  
  A   B  

  t  

Difficult  to  disJnguish  the  contribuJon  of  

these  these  variables  
Environment  
Protocol  mechanisms  

8  
 RepeaJng  Experiments  Enough?  

1  
Delivery  raJo  

0.8  

0.6  
!

0.4  

0.2  

0  
     5              10          15          20          25          30          35  
Tutornet  
Time  (hrs)  

High  delivery  ra-o  across  -me  

(short  experiments  can  be  misleading!)  
9  
 Concurrent  Experiments  
Run  mulJple  protocols  concurrently  
B  
Compare  the  results   A  

  t  
Advantages  
Consistent  environment  for  both  the  protocols  
 
Concerns  
ContenJon  of  different  types  
 
10  
 EvaluaJon  Strategy  
  Serial   Results  
Experiment  
  ==  
Protocols  
  ?  
Concurrent  
  Experiment   Results  
 
Ideally  same  conclusions  from  both  methods  
EvaluaJng  methodologies  not  protocols  
Experiments  on  Tutornet  testbed  
11  
 Protocols  
CollecJon  
CTP  [Gnawali  2009]  
MulJhopLQI  [TinyOS  2007]  
         (LQI)  
 

12  
 Results  from  Serial  CTP  vs  LQI  
Experiment  on  Tutornet  
8  
7  
6  
5  
4  
CTP  
3   LQI  
2  
1  
0  
Delivery   Cost   Path  Length   Churn/node-­‐hr  
13  
 Results  from  Concurrent  CTP  vs  LQI  
Experiment  on  Tutornet  
8  
7  
6  
5  
4  
CTP  
3   LQI  
2  
1  
0  
Delivery   Cost   Path  Length   Churn/node-­‐hr  
14  
 Pueng  Concurrent  Methodology  to  
Use:  Expts.  with  External  Interference  
Engineered  Scenario  

Both  protocols  struggle  in  the  same  environment.  

15  
Pueng  Concurrent  Methodology  to  Use:  
Experiments  in  a  Dynamic  Network  

Hours  
CTP  and  LQI  react  differently  to  dynamics.  
16  
 HW  
Make  a  list  of  metrics  from  the  ten  
research  papers  you  selected  
 
Most  common  metric,  most  uncommon  
metric  
 
Most  common  and  most  unusual  
experiments