Test Case Design: Absence of evidence vs evidence of absence

Testing for the "absence of something" could be as simple as looking to see if that something is there. However, just because you do not see/find something doesn't mean that it is not there. Furthermore, just because it is not there does not mean that the proceedure to remove something, or prevent something from being there, is working.

Test Case Design

I just got burned by a test case that gave a false positive when it was executed. The verification step was to look for the existance of a secret string in the post-processed log files. I looked in the post-processed logs and and didn't find any evidence of the string thus the test passed.

But, there was a defect and not all log files were instrumented correctly so in a few cases the secret strings were not actually get stripped out of the post-processed logs.

A better test design would be to locate examples of each type of secret string in the raw log files. Then verify that each of these were stripped out by the post-processing. In other words, prove that there is something to be removed before believing that it was removed (and not just missing in the first place.)