Random Testing of WebAssembly

Implementations Using
Semantically Valid Programs
Guy Watson

1
WebAssembly
• Enable high performance
applications on the web

• Deliver code to web browsers

• Compact
• Streamable
• Portable

• Compilation target for other

languages
2
Why is a WebAssembly a good target to test?
• Relatively new language

• Implementations are large and complex

• Guarantees and assumptions about safety

3
Testing WebAssembly
• Existing test suites work up to a point

• The dangerous defects are those that produce incorrect results

• Strict semantics means mutational fuzzing won’t work well

• Random differential testing with semantically valid programs could

expose defects

4
Hypothesis
We propose that random differential testing of WebAssembly
implementations using semantically valid programs allows the
discovery of defects that cause programs to produce incorrect results.
This type of random testing is complementary to existing fuzzing
implementations.

5
WebAssembly, the details
• Stack-based execution • Strict treatment of the stack
• Exactly what was declared,
nothing less and nothing more
• 4 data types:
• i32, i64, f32, and f64
• Structured control flow
• Modules
• Functions • Avoids undefined behavior
• Linear memory
• Variables

6
Example Wasm code fragments
(func (result i32) …
i32.const 6 i64.const 6
i64.const 2
i32.const 2
i64.lt
i32.add
if (result i32)
i32.const 9 i32.const 1
i32.mul) else
i32.const 2
end
…
7
Wasmlike – Generating Random Wasm Programs
• Xsmith specification: Wasmlike
• Semantically correct random programs

• AST generation

• WebAssembly invariants
• Types
• Stack behavior

• Output – Checksum of
• Return value of topmost randomly generated function
• Global variables
• Global memory contents
8
AST Generation
Program Program Program

Function Function Function

+ +

7 +

9
 Example AST

TODO: add the store location

inside the store bubble

10
 Wasmlike example
(add-prop
(add-to-grammar
wasmlike
wasmlike
choice-weight
[Program #f ([globals : VariableDef *] ;;Default
[functions : Func *] [#f (λ () 10)]
[main : Func])]
[Func #f ([params : Param *] [Literal 3]
[Binop 20]
[locals : VariableDef *]
[Comparison 15]
[root : Expr])]
[Unop 20]
...) [IfElse 11]
...)

11
Wasmlike example cont.
(define (binop-rhs) (λ (n t)
(hash 'l t
'r t)))
(define (unop-rhs) (λ (n t)
(hash 'expr t)))

;; The type of the children is unconstrained,

;; they just have to be the
[Comparison [i32 (λ (n t)
(define
(hash
same
child-type (fresh-number))
'l child-type
'r child-type))]]

12
Status of Wasmlike
• Done: • Still needed:
• All 4 Wasm types • Additional memory operations
• Binary and unary arithmetic • Variable interactions
• Comparisons, if/elses, branches, • Tuning
blocks, and loops
• Type conversions
• If possible:
• Functions and calls
• Indirect calls and tables
• Basic memory operations
• Multi-value returns
• Global and local variables

13
Finding Defects
• We expect to find 3 types of defects:

• Semantic

• Crash

• Hang

14
WebAssembly implementations
• Browsers, a JavaScript runtime,
and a WebAssembly runtime
• Firefox
• Chromium
• NodeJS
• WasmTime

• One generated program

• Different outputs means a
semantic bug is present

15
Testing harness
• Generate a program, run it
against each of the
implementations, compare the
results, and record the test run.

• Large amount of tests

• Emulab

• Compare program output

• Collect and view results
16
TODO: afl and binaryen fuzzer slide

17
Status of the testing harness
• Done: • Still needed:
• Configuration for NodeJS • Configurations for
• Firefox
• Runs on Emulab with multiple • Chromium
nodes • WasmTime

• Collects and displays results

18
Schedule

19
Related Work
• Csmith – [Yang et al., PLDI 2011]

• Zest – Rohan Padhye, Caroline Lemieux, Koushik Sen, Mike

Papadakis, and Yves Le Traon, 2019

• AFL - Michał Zalewski, 2014

• Grimoire – Tim Blazytko, Cornelius Aschermann, Moritz Schlögel,

Ali Abbasi, Sergej Schumilo, Simon Wörner, and Thorsten Holz, 2019
20
Conclusion
• WebAssembly is interesting because of its portability and high
performance applications
• WebAssembly implementations are varied, and a good target to
randomly test
• Because Wasm is strict with its properties to provide guarantees
about safety, fuzzing with semantically valid programs could expose
defects
• We plan to differentially test implementations with randomly
generated, semantically valid programs.

21
22
23
Todo: Example output and materials

24