Lab 1: Introduction to JUnit testing

Objectives

The objectives of this lab are:

• Write JUnit tests to verify that implementation matches the specification.

• Writing examples as JUnit tests when only given an interface.

• Submit examples to the handin server and experience the Examplar feedback.

• Implement a given simple interface according to specification.

• Submit written code to the server and experience the testing feedback.

You are allowed to work with another person on this lab. However both of you must submit individually to the server for it to count towards your grade.

To maximize your learning we recommend that you start by working through the lab by yourself, and work with another person only to brainstorm or debug problems.

If you have not completed Lab 0 yet, start there.

1 Writing examples to explore an interface

One of the core, initial steps in designing any program is understanding what it is supposed to do. The logical interface of a program is composed of some implementation details (such as Java classes and interfaces) and some written description (such as free-form prose or Javadoc documentation). It is tempting to skim over the written documentation and then ask a person — a colleague, internet help forums, etc — for more detail. But doing so too quickly prevents you from developing your own insights into what should be happening.

In CS2500 and CS2510, we taught you to write examples early in the design process, before you’ve implemented your functionality, to build up intuition about what your code is supposed to do. Examples could take the form of comments, or of executable check-expect assertions. They were separate in purpose from writing test cases, which were written after your implementation was complete: test cases ensure the implementation-specific details and fiddly edge-cases of your code are correct.

In the second half of this course, you will be designing the Java interfaces and classes that your project needs, based only on written descriptions. But for the first half of this course, your assignments will specify the Java interfaces and/or classes that you will need to implement. And this gives you a new and different oppportunity to build your understanding of an assignment, by working with Examplar.

1.1 Examplar 1: sorting strings

Consider a utilty class that provides a method to sort a list of strings by a comparator:

package cs3500.lab1;

public class SortUtils {
  /**
   * Produces a new list containing the elements of values, in sorted order.
   */
  public static List<String> sortStrings(List<String> values, Comparator<String> compare) {
    throw new RuntimeException("Not yet implemented");
  }
}

You know several algorithms for implementing this method; there are even built-in implementations of it in Java already. The purpose statement of the method is given above, and likely looks similar to purpose statements you may have written for your own methods. This purpose statement is badly flawed, in that there is a wide gap between what this statement permits and what our intuition for a sorting method might be.

• Brainstorm at least three failures of the purpose statement above. Consider where it is ambiguous, consider error behaviors, consider mutability, etc. In a single sentence, describe the failure, along the lines of “The purpose statement does not specify whether...” or “The purpose statement incorrectly permits...”, etc.

• Introduce yourself to your neighbor in lab. Compare your collection of problems with your neighbor, and see which problems you found or missed. If you did not consider a situation that you now think is interesting, why do you think you overlooked it?

Once you have a few scenarios discussed, it’s time to try to see which of them were intended by the authors of SortUtils, and which were unintended mistakes.

• Start a new project in IntelliJ. Copy the SortUtils class above into it — you will need to create the appropriate subdirectory src/cs3500/lab1/, in order to match the package cs3500.lab1 with the directory structure. Create a class ExamplarSorting (in the default package, so it belongs directly in the test/ directory), and implement several test methods that probe the failings you noticed. We give you one base-case example to start from:

  import cs3500.lab1.SortUtils;
  
  import org.junit.Assert; // All the various testing methods are defined here
  import org.junit.Test;   // Used to mark which of your methods are your tests
  
  import java.util.List;
  import java.util.ArrayList;
  import java.util.Comparator;
  
  // NOTE: The class must be public, and must have the expected name, or else
  // Examplar cannot create your examples class and run it.
  public class ExamplarSorting {
    // NOTE: Every test method in JUnit is prefixed with this @Test annotation,
    // but can have any name you choose.
    @Test
    public void exampleEmptyBaseCase() {
      List<String> empty = new ArrayList<>();
      List<String> sorted = SortUtils.sortStrings(empty, Comparator.naturalOrder());
      List<String> expected = new ArrayList<>();
      Assert.assertEquals("Empty lists should be sorted", expected, sorted);
    }
    // ...etc
  }

  Each example that you write should construct a list, sort it, and test whether the sorted result makes sense. Note! Your examples will obviously not run yet, since your implementation of SortUtils is incomplete – this is ok.

  You can write your own Comparator implementations, or you can use the static methods on the Comparator interface for convenience. For example, how might you use Comparator.comparingInt to create a comparator that compares two strings by their lengths?

• Submit just your ExamplarSorting.java file to the handin server. (Note: you may discuss examples with your neighbor, but you must each submit individually to the handin server.)

• Look carefully at the output on the handin server, and read through the Examplar reference again. If you submit just the exampleEmptyBaseCase example above, you will find that your examples pass on all the wheats, manage to catch one of the chaffs, and miss several others.

  Develop more examples until you catch chaffs 0–6. (You cannot catch chaffs 7–10 until you complete the next part of the lab.)

  If you get stuck making these examples, consider two key ideas:

  • The shape of the example: The data definition of a list hints to at least two difference shapes for a list: empty and non-empty. However, shape can also include the data within! Draw out your list examples. When you get to your strings, write them out vertically one character at a time. If your examples start to have similar shapes, you may want to investigate how you can create a list that has another shape.

  • The property of the example: Make sure your examples have different properties to them. For instance, what is the difference between the list "a, b, c", "b, a, c" and "a"? Even with three letters, we can craft different lists of various properties related to sortedness.

1.2 Examplar 2: sorting, generically

The authors of the SortUtils class recognized that sorting can be useful beyond just for strings, and so developed a second method:

package cs3500.lab1;

public class SortUtils {
  /**
   * Produces a new list containing the elements of values, in sorted order.
   * @param <T> the type of elements to sort
   */
  public static <T> List<T> sort(List<T> values, Comparator<? super T> comp) {
    throw new RuntimeException("Not yet implemented");
  }
}

• Again brainstorm weaknesses of this purpose statement. Are there any new scenarios you can think of that are unique to the generic function, and couldn’t happen with the strings-only function? Or are there mistakes that could happen with the strings-only version that can no longer happen with the generic one?

• Try writing new examples to probe the behavior of sort. If you try writing a non-empty example, you’ll find that you now fail one of the wheats. Why do you think that is? Discuss with your neighbors.

• Refine your examples until they no longer fail on the wheats, and then try to catch the remaining chaffs. Remember the ideas of shape and property from above.

• Refine your examples further until they completely pass the Precision and Usefulness measures, as well as Correctness and Thoroughness. Discuss with your neighbors or the course staff to try to figure out what other examples you might need to build, or examples you can remove (because they’re covered by other examples).

Hint: for the particular chaffs given to you here, the bugs in sort are a strict subset of the bugs in sortStrings.

2 The Fraction interface

In this section of the lab, you have to implement simple fractions and implement some functionality for them.

A fraction is represented by the Fraction interface. This interface should contain the following methods:

• A method to add two fraction objects: Fraction add(Fraction other).

• A method to add a fraction with another given as a numerator and denominator: Fraction add(int numerator,int denominator). This method should throw an IllegalArgumentException exception if the fraction provided to it is negative.

• A method that returns the decimal value of a fraction, rounded to the given number of places: double getDecimalValue(int places).

2.1 What to do

• Create the above interface, and document its specifications as detailed above.

• Design JUnit tests that verify these specifications for an implementation called SimpleFraction.

• Implement the Fraction interface in a SimpleFraction class. Leave all the methods blank for now, but document them properly. The specifications for this implementation (beyond what the interface specifies) are:

  • This class can only represent a non-negative fraction. Any attempt to create a negative fraction through the constructor should throw an IllegalArgumentException.

  • This class should have a single public constructor that takes the numerator and denominator as integers as its only arguments. Note that these arguments can be individually negative, even as the constructor imposes the above constraint.

  • This class should also override the toString method, which returns a string of the form "n/d". For example, a fraction created with numerator 2 and 4 should return "2/4" through its toString method, whereas a fraction created with numerator -4 and denominator -9 should return "4/9" through its toString method.

  • Note that wherever applicable, you may not assume that this is the only implementation of the Fraction interface.

• For each method to be implemented in the SimpleFraction class: design and write all JUnit tests to verify its specification, then complete the implementation and run the tests. Follow the directions in Lab 0 to place the test files correctly in your project.

• When you and a neighbor (who you have not been working with so far) have completed both your test cases and your implementation, email each other your test cases. Add them to your project, and run them. Do they all pass? If not, is the mistake in (a) your implementation, (b) their test case, (c) the specification, or (d) more than one of the above?

Note: How will you test the constructor? The purpose of the constructor is to create the object as specified, or die trying (i.e. throw an exception). The latter can be readily tested (see Assert.assertThrows, but how to test the former? Since we cannot (should not) directly access fields of the SimpleFunction object from within the test, the test has to rely on other (simple) methods to test this. But how do we know those methods are themselves correct? If these methods are short and simple (e.g. they do not compute anything, but rather directly report something about the object) then it is improbable that they work incorrectly. It is a “leap of faith” in some ways, but it fulfills our objectives of testing everything we implement, while also not resorting to publicizing fields accessible or writing methods just to be able to test. In general, our tests will only ever be able to show that some set of methods is mutually consistent, not that every method is independently correct.}

3 Questions to ponder and discuss

When you are done with the above, here are some additional tasks that you should do. You are encouraged to discuss them with the person next to you, and talk to the course staff as needed.

3.1 Improving the design

The second add method does not really belong in the interface, although hopefully you found it useful. A consequence of putting it in the interface is that it is a public method in all its implementations.

Can you redesign your classes and interfaces such that this method is no longer public, but other methods still work as described? Whatever design you come up with, what are its limitations?

Implement this design and show to the course staff.

3.2 Better testing

How did you test whether your add method works correctly? Did you have a few sample inputs and expected outputs? How many samples did you try? Are they enough?

An alternative way is fuzzy testing, or random-sample testing. This type of testing is useful when a method can have a large number (seemingly infinite) of inputs that produce expected outputs. It works as follows:

1. Determine if it is possible to categorize the possible inputs (e.g. all positive numbers, all negative numbers, etc.). This will depend on the specific problem at hand.

2. For each category, generate a large number of random inputs using a random number generator. Be sure to give a constant seed to the generator, so that one can run the test repeatedly and be assured of the same set of random inputs.

3. Write a test that repeatedly (e.g. using loops) uses a random input and verifies expected output(s).

How large should the sample size be (i.e. how many sets of random inputs should we try)? Note that this testing probabilistic in nature: we are hoping that we test enough inputs so that the probability of any input producing the correct expected output is "high enough". A few thousand samples are usually a good default size to start with.

Test your add method using fuzzy testing. Now test the other methods similarly. Show your work to the course staff.