Testing
1 How to test
2 Types and Examples
3 Testing
3.1 Kinds of tests
3.2 Where do tests go in my project?
8.3

Testing

1 How to test

Writing tests is an essential part of design and implementation. The most important skill in writing tests is to determine what to test, and then determine how to test.

Testing a compiler is particularly nuanced: if you find what looks to be a bug, how can you localize it? The problem could be

Programming under this level of uncertainty is like fighting quicksand: The more you struggle and the more things you change, the less likely it is that you’ll figure out the underlying problem and get unstuck. So what to do?

2 Types and Examples

The one thing you can absolutely rely upon is that if your OCaml code compiles, then it is type-correct: you will never misuse a value of one type as if it were of some other type. This means that if you can encode important invariants about your program in the types, then the OCaml compiler itself will enforce that they are upheld. So before you dive into hacking away, consider the signatures of your functions very carefully.

Imagine you have some type signature Typ1 Typ2 -> Typ3, and you need to provide some implementation of that type. Additionally you have some specification of what that function is supposed to compute: You must check that your concrete implementation of that type works as specified. However thinking about tests after you have completed the implementation is not ideal. Since you have already written your implementation, you will likely come up with tests that you already know will pass, rather than tests that should pass. Here are some recommendations on how to come up with effective test cases:

Writing tests before writing your implementation will give you insight into what your implementation ought to do. Moreover, it will help you work through the types that you have versus the ones you may want, and often just understanding that structure is a big help in understanding the problem.

3 Testing

Obviously, unfortunately, you often can’t write a complete set of tests for your code before you’ve started writing your code, as the process of implementing a design can easily bring issues to the forefront that you didn’t notice or anticipate. Proper testing is an iterative process: starting from initial examples you create an initial implementation, which might suggest additional tests, which might cause test failures, which you need to fix, which might suggest additional tests, etc. A successful set of test cases is one that tests whether your implementation adheres to your design, whether your design leaves loopholes and ambiguities that allow its incorrect usage, and whether the behavior of your implementation can be predicted in all situations. This set of test cases should compile, and upon running, should pass.

NOTE: It is far better to include tests that you know to fail, rather than comment them out or delete them. Leave a (* FIXME *) comment next to the failing tests, explaining what you intended the test to check for, and why you think it’s currently failing. At some point you clearly had a reason for writing the test case, and it would be a shame to lose that insight by deleting the test! Equally bad is commenting the test out, since it gives the misleading impression that everything is fine and all tests pass, when there are known problems remaining...

3.1 Kinds of tests

There are many kinds of tests you may wish to write:

3.2 Where do tests go in my project?

At the moment, we aren’t exploiting OCaml’s support for packaging too heavily: every file is a package, and there are no subdirectories or "Java packages" to worry about. You can even place all your tests within a single test.ml file. Still, it’s worth breaking the tests out into separately-named test suites, so that their purpose and organization is more readily apparent.