Classification of types of test

When should a test be classified as an expected value test rather than a smoke test ? Or a maths test? A brief discussion of how the DataSHIELD team makes these classifications.

• Argument test (arg) : The very first test to write. It sends off DataSHIELD commands to the server with firstly no arguments included (hoping for the relevant error message “please submit a value for $argument_1”), then with the first argument correct and the second argument incorrect (hoping for error message “please submit a correct value for $argument_2”), and so on until the error message has been checked for all possible arguments. Hence, “argument test”.

• Smoke test (smk) : intended to be a general test to check that the structure returned is broadly correct. Goes one step further than the above argument tests, as it tests the output from the server, not just a clientside message. Is important not to get caught up in minutiae, as it is meant to be a stepping stone to tell the user that the function is acting as intended and is creating an output of the correct structure.

• Expected value test (expt) : intended to be a more specific test than the above smoke type, one which checks in detail the output of all numbers returned from a function call. Will check these numbers to a higher degree of accuracy than the above smoke test. For example, it will check all the cells of a table have precisely the same value whether the table is generated in DataSHIELD or in native R.

• Maths test (math) : checks conceptual properties of the function output are correct. For instance, checks that the sum of all probabilities does not exceed 1.

• Disclosure test (disc) : checks that various potentially disclosive parts of DataSHIELD function outputs do not fall below a level where they would be disclosive.