Make your tests speak TAP

Bruno P. Kinoshita – http://www.kinoshita.eti.br

Gentileza gera gentileza

Bruno P. Kinoshita – http://www.kinoshita.eti.br

Did you understand that?

Bruno P. Kinoshita – http://www.kinoshita.eti.br

What if I translated for you...

Bruno P. Kinoshita – http://www.kinoshita.eti.br

To which language(s)?

Bruno P. Kinoshita – http://www.kinoshita.eti.br

Japanese?

Bruno P. Kinoshita – http://www.kinoshita.eti.br

French?

Bruno P. Kinoshita – http://www.kinoshita.eti.br

Java?

Bruno P. Kinoshita – http://www.kinoshita.eti.br

C++?

Bruno P. Kinoshita – http://www.kinoshita.eti.br

What about this...

Bruno P. Kinoshita – http://www.kinoshita.eti.br

<?xml version="1.0" encoding="UTF-8"?> <testng-results skipped="0" failed="0" total="1" passed="1"> <reporter-output> </reporter-output> <suite name="Default suite" duration-ms="22" started-at="2011-08-29T19:57:54Z" finishedat="2011-08-29T19:57:54Z"> <groups> </groups> <test name="Default test" duration-ms="22" started-at="2011-08-29T19:57:54Z" finishedat="2011-08-29T19:57:54Z"> <class name="br.eti.kinoshita.jcertif.ATestNGTest"> <test-method status="PASS" signature="testCurrentTimeMillis()" name="testCurrentTimeMillis" duration-ms="13" started-at="2011-08-29T19:57:54Z" finishedat="2011-08-29T19:57:54Z"> </test-method> </class> </test> </suite> </testng-results>

Bruno P. Kinoshita – http://www.kinoshita.eti.br

<?xml version="1.0" encoding="UTF-8"?> <testng-results skipped="0" failed="0" total="1" passed="1"> <reporter-output> </reporter-output> <suite name="Default suite" duration-ms="22" started-at="2011-08-29T19:57:54Z" finishedat="2011-08-29T19:57:54Z"> <groups> </groups> <test name="Default test" duration-ms="22" started-at="2011-08-29T19:57:54Z" finishedat="2011-08-29T19:57:54Z"> <class name="br.eti.kinoshita.jcertif.ATestNGTest"> <test-method status="PASS" signature="testCurrentTimeMillis()" name="testCurrentTimeMillis" duration-ms="13" started-at="2011-08-29T19:57:54Z" finishedat="2011-08-29T19:57:54Z"> </test-method> </class> </test> </suite> </testng-results>

Bruno P. Kinoshita – http://www.kinoshita.eti.br

Producer

public class ATestNGTest { @Test public void testCurrentTimeMillis() { Assert.assertTrue( System.currentTimeMillis() > 0 ); } }

<?xml version="1.0" encoding="UTF-8"?> <testng-results skipped="0" failed="0" total="1" passed="1"> <reporter-output> </reporter-output> <suite name="Default suite" duration-ms="22" started-at="2011-08-29T19:57:54Z" finishedat="2011-08-29T19:57:54Z"> <groups> </groups> <test name="Default test" duration-ms="22" started-at="2011-08-29T19:57:54Z" finishedat="2011-08-29T19:57:54Z"> <class name="br.eti.kinoshita.jcertif.ATestNGTest"> <test-method status="PASS" signature="testCurrentTimeMillis()" name="testCurrentTimeMillis" duration-ms="13" started-at="2011-08-29T19:57:54Z" finishedat="2011-08-29T19:57:54Z"> </test-method> </class> </test> </suite> </testng-results>

Bruno P. Kinoshita – http://www.kinoshita.eti.br

Test Result

public class ATestNGTest { @Test public void testCurrentTimeMillis() { Assert.assertTrue( System.currentTimeMillis() > 0 ); } }

<?xml version="1.0" encoding="UTF-8"?> <testng-results skipped="0" failed="0" total="1" passed="1"> <reporter-output> </reporter-output> <suite name="Default suite" duration-ms="22" started-at="2011-08-29T19:57:54Z" finishedat="2011-08-29T19:57:54Z"> <groups> </groups> <test name="Default test" duration-ms="22" started-at="2011-08-29T19:57:54Z" finishedat="2011-08-29T19:57:54Z"> <class name="br.eti.kinoshita.jcertif.ATestNGTest"> <test-method status="PASS" signature="testCurrentTimeMillis()" name="testCurrentTimeMillis" duration-ms="13" started-at="2011-08-29T19:57:54Z" finishedat="2011-08-29T19:57:54Z"> </test-method> </class> </test> </suite> </testng-results>

Bruno P. Kinoshita – http://www.kinoshita.eti.br

Consumer

What about a simpler way to express that idea?

Bruno P. Kinoshita – http://www.kinoshita.eti.br

What about TAP?

Bruno P. Kinoshita – http://www.kinoshita.eti.br

The Test Anything Protocol

Text based Human and machine readable More than 20 years old (released in 1987 with Perl) Language agnostic Used in many projects (see CPAN) Extensible test diagnostics IETF draft

Bruno P. Kinoshita – http://www.kinoshita.eti.br

The Test Anything Protocol
1..1 Plan ok 1 – br.eti.kinoshita.jcertif.ATestNGTest#testCurrentTimeMillis --message: TestNG Test testCurrentTimeMillis severity: '~' Test Result source: br.eti.kinoshita.jcertif.ATestNGTest#testCurrentTimeMillis datetime: '2011-08-29T21:56:20' file: br.eti.kinoshita.jcertif.ATestNGTest line: '~' name: testCurrentTimeMillis extensions: '~' got: '~' expected: '~' display: '~' YAMLish diagnostics dump: '~' error: '~' backtrace: '~' ... YAML is a data serialization format, like JSON

Test Set
Bruno P. Kinoshita – http://www.kinoshita.eti.br

Test Set Plan 1..2 Test Results ok 1 Success # 10:42:15 – 12/12/11 not ok 2 Couldn't open file: input.yaml Diagnostics --file: input.yaml datetime: 12/12/11 – 10:42:15 GMT -3 ...

Bruno P. Kinoshita – http://www.kinoshita.eti.br

Demo

Bruno P. Kinoshita – http://www.kinoshita.eti.br

Integrate different producers and consumers

Bruno P. Kinoshita – http://www.kinoshita.eti.br

Extend your test result

Bruno P. Kinoshita – http://www.kinoshita.eti.br

How could you add extra information here?
<?xml version="1.0" encoding="UTF-8"?> <testng-results skipped="0" failed="0" total="1" passed="1"> <reporter-output> </reporter-output> <suite name="Default suite" duration-ms="22" started-at="2011-08-29T19:57:54Z" finishedat="2011-08-29T19:57:54Z"> <groups> </groups> <test name="Default test" duration-ms="22" started-at="2011-08-29T19:57:54Z" finishedat="2011-08-29T19:57:54Z"> <class name="br.eti.kinoshita.jcertif.ATestNGTest"> <test-method status="PASS" signature="testCurrentTimeMillis()" name="testCurrentTimeMillis" duration-ms="13" started-at="2011-08-29T19:57:54Z" finishedat="2011-08-29T19:57:54Z"> </test-method> </class> </test> </suite> </testng-results>

Bruno P. Kinoshita – http://www.kinoshita.eti.br

Perhaps you imagined an external file?

Bruno P. Kinoshita – http://www.kinoshita.eti.br

"Je continue de penser que les logiciels open source ne sont pas suffisamment évolués pour être aussi performant que les produits commerciaux. C'est vrai pour chacun de ces outils individuellement, mais j'ai découvert récemment que l'intégration de certains de ces outils constituait une plateforme de test outillée très puissante, et peut-être plus puissante qu'un produit commercial". (Olivier Renault)

Bruno P. Kinoshita – http://www.kinoshita.eti.br

There are other test formats

Bruno P. Kinoshita – http://www.kinoshita.eti.br

The most similar to TAP is SubUnit

Bruno P. Kinoshita – http://www.kinoshita.eti.br

There is no silver bullet

Bruno P. Kinoshita – http://www.kinoshita.eti.br

It is difficult to group Test Sets, like in Test Suites

Bruno P. Kinoshita – http://www.kinoshita.eti.br

Gentileza gera gentileza

Bruno P. Kinoshita – http://www.kinoshita.eti.br

Kindness generates kindness

Be kind today to somebody else here in JCertf :)
Bruno P. Kinoshita – http://www.kinoshita.eti.br