This short primer shows how to use tests as a basis for language development with Spoofax. As an example project we create a small 'calculator' language that shows many of the basics of language definitions.

The source for our example project is collected in [[%ATTACHURL%/before.zip][before.zip]]. The complete project, with all functionality implemented to make the test cases succeed is collected in [[%ATTACHURL%/after.zip][after.zip]].

A full description of the testing language can be found in the paper [[http://researchr.org/publication/KatsVermaasVisser2011][Integrated Language Definition Testing. Enabling Test-Driven Language Development]]. Presentation slides also show Spoofax testing in action: [[http://slidesha.re/tYaHQT]]

%TOC{depth="3"}%

<!-- http://slidesha.re/tYaHQT -->

---++ A Calculator Language

Our example language supports arithmetic expressions, variables, and assignments. An example:

<verbatim>
a = 3
a * 4
</verbatim>

Based on this simple language we can write test cases. Doing test-driven development, these can even drive the development of the language, but in this document we focus on the tests.

Tests can be written using a =.spt= file. Create one in a Spoofax project, and press control-space to get a basic test definition. It will have tests of this form:

<verbatim>
test description [[
  a = 3
  a * 4
]] 0 errors
</verbatim>

As the example shows, each test can quote a program using =[[=, =[[[= or =[[[[= brackets. It can also specify a _description_ for the test, and a condition. This test specifies that =0= semantic =errors= are expected.

---++ Syntax

Example of tests of the syntax:

<verbatim>
test Add [[
  1 + 2
]] parse succeeds

test Abstract syntax (1) [[
  1
]]
  parse to Int("1")

test Abstract syntax (2) [[
  1 * 2
]]
  parse to Mul(Int("1"), _)

test Parentheses [[
  (1 + 2)
]]
</verbatim>

These tests specify parse success, compare the abstract syntax of the test input against a pattern, or simply specify that the test should succeed.

<!--
<verbatim>
"(" Expr ")" -> Expr {bracket}
</verbatim>
-->

Tests can also use concrete syntax to test operator precedence and associativity:

<verbatim>
test Multiply and add (1) [[
  1 + 2 * 3
]] parse to Add(_, Mul(_, _))

test Multiply and add (2) [[
  1 + 2 * 3
]] parse to [[
  1 + (2 * 3)
]]

test Add and multiply [[
  1 * 2 + 3
]] parse to [[
  (1 * 2) + 3
]]

test Add and add [[
  1 + 2 + 3
]] parse to [[
  (1 + 2) + 3
]]
</verbatim>

---++ Evaluation of expressions

The following tests will =run= a transformation named =calc= to test evaluation of expressions:

<verbatim>
test Constant [[
  1
]] run calc to "1"

test Add [[
  1 + 1
]] run calc to "2"

test Subexpression [[
  1 + [[2 + 3]]
]] run calc to "5"
</verbatim>

Note that the last test in this series uses the =[[= ... <tt>]</tt><tt>]</tt> brackets to make a *selection* in the test. The =calc= transformation is evaluated for this selection.

<!--
<verbatim>
  calc:
	 Int(i) -> i
  calc:
	 Plus(x, y) -> <addS> (<calc> x, <calc> y)
</verbatim>
-->

---++ Variables

The following tests exercise the definition of variables:

<verbatim>
test Variable [[
  x
]] parse

test Variable [[
  longname
]] parse
</verbatim>

<!--
=Calculang.sdf=:

<verbatim>
	 ID			-> Exp {cons("Var")}
</verbatim>

assignment statements
-->

<verbatim>
test Assignment [[
  x = 4
]] parse
</verbatim>

<!--
<verbatim>
	 Stm		  -> Start {cons("Statements")}
	 ID "=" Exp -> Stm {cons("Assign")}
	 Exp		  -> Stm
</verbatim>
-->

<verbatim>
test Multiple statements [[
	x = 1
	y = 2
]] parse to Statements([Assign(_, _), Assign(_, _)])

	 Stm*		 -> Start {cons("Statements")}
	 ID "=" Exp -> Stm {cons("Assign")}
	 Exp		  -> Stm
	 ID			-> Exp {cons("Var")}
</verbatim>

<!--
regression in syntax.spt!?

add ={prefer}= attribute to =CalcuLang.sdf=

<verbatim>
	 Exp -> Start {prefer}
</verbatim>

what I want eventually:

<verbatim>
test Eval variable [[
  x = 4
  x
]] run calc to "4"
</verbatim>

but first:
-->

<verbatim>
test Evaluate multiple statements [[
  1
  2
]] run calc to "2"
</verbatim>

<verbatim>
  calc:
	  Statements(s*) -> last
	  where
		  s'*  := <map(calc)> s*;
		  last := <last> s'*
</verbatim>

<verbatim>
test Eval constant [[
  PI
]] run calc to "3.14"

  calc:
	 Var("PI") -> "3.14"
</verbatim>

<!--
revisit:

<verbatim>
test Eval variable [[
  x = 4
  x
]] run calc to "4"
</verbatim>

<verbatim>
  calc:
	 Assign(x, v) -> v'
	 where
		v' := <calc> v;
		rules(
		  GetValue: x -> v'
		)

  calc:
	 Var(v) -> <GetValue> v
</verbatim>
-->

<verbatim>
test Eval multiple variables [[
  x = 2
  y = x * 2 + x
  y
]] run calc to "6"
</verbatim>

---++ Editor features

The following test cases test the editor facilities of the language:

<verbatim>
test Variable unassigned [[
  y
]] 1 error /unassigned/
</verbatim>

This test succeeds if the input has =1 error= and it matches the regular expression =/unassigned/= (as variable =y= is unassigned).

<!--
<verbatim>
  analyze =
	 topdown(try(record-var))

  record-var:
	 Assign(x, e) -> Assign(x, e)
	 with
		rules(
		  GetVar :+ x -> x
		)
  
  constraint-error:
	 Var(v) -> (v, $[Unassigned variable])
	 where
		not(<GetVar> v)
</verbatim>
-->

<verbatim>
test Multiple assignments to same variable [[
  y = 1
  y = 2
  y
]] /multiple/
</verbatim>

This test succeeds if there are one or more (error) messages that match =/multiple/= (as there are _multiple_ definitions of =y=).

<!--
<verbatim>
  constraint-error:
	 Var(v) -> (v, $[Multiple assignments to same variable])
	 where
		all-vs := <bagof-GetVar> v;
		<gt> (<length> all-vs, 1)
</verbatim>
-->

<verbatim>
test Reference resolving (1) [[
  x = 4
  [[x]]
]] resolve

test Reference resolving (2) [[
  [[x]] = 4
  [[x]]
]] resolve #2 to #1

test Content completion [[
  avariable = 1
  [[a]]
]] complete to "avariable"
</verbatim>

These test cases test reference resolving and content completion. They use the =[[= ... <tt>]</tt><tt>]</tt> selection mechanic to select parts of the program. The first test case specifies that reference resolving should work for the selection. The second specifies that the second selection should resolve to the first selection. The last test case specifies that the selection should provide a content completion option =avariable=.

<!--
- - - + + Compilation

<verbatim>
test Constant [[
  1
]] build generate-java "
		public class Output {
		  public static void main(String[] args) {
			 System.out.println(1);
			 System.exit(0);
		  }
		}"

test Constant [[
  1 + 2
]] build generate-java 

  to-java:
	 _ -> $<
		public class Output {
		  public static void main(String[] args) {
			 System.out.println(42);
			 System.exit(0);
		  }
		}
	 >
</verbatim>
-->

---++ Execution

The =Calculang= project generates Java code. The following test case tests the output that is returned when the program is compiled and executed:

<verbatim>
test 42 [[
  42
]] build generate-result to 42
</verbatim>

<!--
test 42, part deux [[
  41 + 1
]] build generate-result to 42
...variables, etc....
...add to evaluation tests??...
</verbatim>
-->

---++ Refactoring

Refactorings are a form of transformations that can be triggered in the editor based on a selection. These test cases test the rename refactoring of Calculang:

<verbatim>
test Basic rename [[
  a = 1
  [[a]]
]] refactor rename-var("b") to [[
  b = 1
  b
]]

test Another rename [[
  a = 1
  b = 2
  [[a]]
]] refactor rename-var("c") to [[
  c = 1
  b = 2
  c
]]

test Rename collision [[
  a = 1
  b = 1
  [[a]]
]] refactor rename-var("b") to _
	1 error
</verbatim>

Note how these tests specify the new name as an argument of the `rename-var` refactoring. Also note the last test which is a negative test case: the refactoring should report an error if a user attempts to rename =a= to =b=.
 
<verbatim>
  rename-var:
	 (newname, selected-name, position, ast, path, project-path) -> ([(ast, new-ast)], fatal-errors, errors, warnings)
	 with
		new-ast  := <topdown(try(rename-type(|selected-name, newname)))> ast; 
		(errors, warnings) := <semantic-constraint-issues> (ast, new-ast);
		fatal-errors := []

  rename-type(|old-name, new-name):
	 Assign(old-name, y) -> Assign(new-name, y)

  rename-type(|old-name, new-name):
	 Var(old-name) -> Var(new-name)
	 
  semantic-constraint-issues:
	 (ast, new-ast) -> (<diff>(new-errors, errors), <diff>(new-warnings, warnings))
	 where
		(_, errors, warnings, _) := <editor-analyze> (ast, "", "");
		(_, new-errors, new-warnings, _) := <editor-analyze> (new-ast, "", "")
</verbatim>

<!--
- - - + + Stratego expressions

As an additional feature, you can test Stratego expressions 

<verbatim>
test Arithmetic API
  <mul> (2, 3) => 6

test String-based arithmetic API
  <mulS> ("2", "3") => "6"

test Foo 
  let
	 foo = !4
  in
	 foo;
	 debug
  end
</verbatim>
-->

<!--
	* Set SKIN = customtitle
	* Set CUSTOMHEADTITLE = Test-Driven Language Development with Spoofax
	* Set CUSTOMTOPICTITLE = Test-Driven Language Development with Spoofax
-->