If you fail to plan you plan to fail, or so the saying goes. And planning is all about thinking ahead, working now on what will be required later. But, when it comes to specifying software, we often do not plan on how we will be testing it until it is delivered. This can be a costly mistake. When it comes to writing a contract for deliverables, it will pay to include a number of Test Cases (or Tests for short) to specify how the delivered software will be judged.
Such Tests would go beyond the typical Requirements Specifications and Use Cases, thereby providing greater clarity to the developer on how ‘success’ will be judged and greater certainty to the customer about what will be delivered.
A Test would specify some real input and indicate the expected output. This is what differentiates a ‘Use Case’ from a ‘Test Case’ – the Use Case simply indicates the expected behaviour while the Test Case actually includes some specific input data as well as documenting the specific outputs expected. For example, in the case of a hypothetical ‘wayfinding app’, rather than specifying that the software should allow for “quick routing from carpark to venue along shortest path“, let us specify some examples of how this would be tested, including the expected outcome. That is, we would indicate that we will be testing the software on an iPhone 6, standing in the Carpark A and that we expect the app to rout us to Lecture Theatre C along route B, calculating the result within less than 1 second while connected to a standard 4G network. Translating each Use Case into a number of Tests, and specifying that these Tests must be demonstrated as part of the deliverables and at hand-over, will significantly reduce the risk for arguments. Furthermore, by giving the developer access to specific test data during the development, mistakes could have been spotted much earlier – one of the problems with complex, data-driven applications is that even so results may look correct, they could well be wrong, and such mistakes will only be obvouis to a subject matter expert, not the developer.
For example, in a recent project, Use Cases and GUI mock-ups indicated that searches within an app should return results and display them in a hierarchical structure. The delivered app did this, and the result, at first glance, appeared correct. However, the execution was very slow and the results, on inspection by someone familiar with the data, were incorrect. This lead to arguments about data integrity, and load procedures, etc.. These arguments ultimately proved incorrect – the fault was in the software. However, the arguments and uncertainty could have been avoided by documenting a number of test cases, including known specified inputs and outputs, in the contractually documented software specifications.
Writing such test cases will introduce any ‘extra work’ – any software will be tested anyway, and test tests we will likely need documenting and formalising. Including Tests in the contracts simply means thinking about and documenting test tests much earlier, well before even signing any contracts.
To avoid a ‘Dieselgate’ type scenario, it may be wise to write a number of alternative Tests. These alternatives would test the same functionality as those specified in the official contractually documented Tests, but using input and outputs that are not known to the developers. This will ensure that the software developers are not simply hard-coding test case results, though this is unlikely, given that, by car industry standards, we are a relatively low risk/value environment were the effort involved in attempting to ‘beat’ the test-case would be greater than complying with the intent and spirit of the specifications.
Software Driven Development: There is a branch of computer science and software development known as ‘test-driven-development’. This involves writing automated software tests for specific bits of code (functions). These tests can then be run to test the code throughout the development and to re-test code as changes are made. The above does not argue for ‘test-driven-development’ but talks about incorporating some of these ideas into software specifications.