Interim Report II

Sarah Cooper
Aug 3, 2017

Project Title: ROCS Test Automation and Framework Development

Mentors: Jeng Yen (347K) and Trevor Reed (347K)

Progress Report:

The past month has seen myself and my partner, Johan Michalove, make significant
progress towards the completion of our project. Having accomplished the necessary
research on testing frameworks, we decided to move forward using the Robot
Framework, which is described on the home page of their website as an open source
“generic test automation framework for acceptance testing and acceptance test-driven
development”. This framework, first released in 2008, is very well founded and had all of
the key features we were looking for. These included portability, python source, and
straight-forward extensibility.

Once we had chosen our framework, my partner and I began to work on different
aspects of its development. Johan’s focus became hardcoding keywords within Robot,
automating the framework by getting it to run on Jenkins (an open source test
automation server), and moving ROCS onto Amazon Web Services – thus fulfilling the
“cloud-based” portion ROCS’ developmental goals. Meanwhile, I dove into learning
about each of the four ROCS services that are currently supported: SPICE, arm inverse
kinematics, mast inverse kinematics, and Frames. As my familiarity with each of the
services increased, I began to write suites of test cases for each of them, taking extra
care to consider how the user might interface with these services in order to understand
better how the services might break under conditions the developers might not have
anticipated. To date, I have written the test suites for three of the four services and am
working on gaining a deeper level of understanding of this final service so that I can
write an extensive list of tests for it as I have done for the others. That said, there is still
much work that can and should be done to continue to increase the number of unit tests
and make the test suites even more exhaustive.

In the process of immersing myself in these ROCS services, writing test cases for them,
and observing the report logs on Robot that described which tests did or did not pass, I
uncovered various bugs in a few of the services. SPICE in particular, a longstanding
system that ROCS is using to compute time conversions (between ephemeris, Space
Clock, and Local Mars Standard Time) and sun position coordinates was failing the
majority of test cases that I had written. Working with Johan, ROCS developers, and
Boris Semenov – a resident expert on all things SPICE – we pinpointed the source of
the bugs within ROCS. With all of these bug fixes, we were able to decrease the
number of failing test cases from 56 out of 78 to just 4. Despite this improvement, there
is still much to do with the other services. The job of debugging and building up our test
suites will likely continue until the end of my time here at JPL.
Interim Report II
Sarah Cooper
Aug 3, 2017

The drive to automate our testing framework and populate it with tests came not just
from the 10-week deadline inherent in our internships, but also more pressingly from an
upcoming Rover Planning Services (RPS) meeting in which we will be demo-ing our
product to a number of team leads within MSL. Scheduled to take place at the
beginning of the eight-week mark and coinciding with the end of a three-week sprint for
RPS, this is a capstone of sorts for our project and our chance to show our progress
towards developing an end product that may be useful to many people. This ties in with
one of our overarching goals for internship, namely, to start a project and see it through
to completion, creating a product that will merge with the existing workflow of the ROCS
developers and persevere through all later stages of development. Additionally, with this
upcoming demo, our framework has the potential to serve as a model on which other
development teams can base their own frameworks, thus extending its field of influence
to include various Mars2020 projects.

Challenges:

In this past month, as Johan and I have continued to make progress towards the
completion of our task, we have also encountered a number of unexpected challenges.
In reference to the above paragraph, one challenge has been the introduction of the
RPS meeting demo deadline. However, it does not follow that it was an unwelcome
challenge. On the contrary, having this earlier deadline and the knowledge that a larger
audience will be present increased our motivation to create a finished product and gave
us a more tangible deadline to work towards.

As we continued to work on automation and test creation, we faced other challenges

that were more developmental in nature. For me personally, a challenge was the fact
that part way through the process, I had to change the medium through which I wrote
tests. Our original plan was to write tests on Excel to make the writing of tests more
accessible to scientists, working under the assumption that scientist users of ROCS
may want to be able to report any bugs they encounter and that they would feel more
comfortable working in Excel rather than a text editor. However, Excel proved less
compatible with Robot than we originally imagined, so I had to switch to writing in a text
editor (Atom) partway through the test-writing process, adapting the tests I had written
thus far to the new format required. Another complication of sorts that Johan and I have
had to adapt to is that once we saw that a large number of our SPICE tests were failing,
our mentor challenged us to take on the role of ROCS developers ourselves, not just
writing tests for SPICE, but also investigating the bugs and finding solutions to improve
SPICE performance.

A final challenge was that contrary to what we predicted in our first interim reports,
meeting with scientists to learn more about their Mars2020 instruments and how they
might use ROCS has not been a very large aspect of our project this past month. The
other aspects of our task demanded most of our attention especially as we prepared for
the approaching demo of our product, leaving little time to schedule scientist meetings.
In addition, certain individuals who were going to help us on this part of our project
Interim Report II
Sarah Cooper
Aug 3, 2017

proved difficult to get a hold of, somewhat stalling our progress in this area. That said,
we did successfully meet with one PIXL project scientists. Moving forward in the last few
weeks of our internship as the development of the ROCS testing framework reaches a
stable point, we intend to renew our efforts to reach out to scientists and hopefully make
up for lost time on that front.