|Issue||High risk||Medium risk||Low risk|
|Protection testing provides verification that protection does what it is supposed to do.||Yes||Yes||No|
|Fault models are used to generate and evaluate tests.||Yes||Yes||No|
|Coverage of tests are measured against the fault model.||Yes||Yes||No|
|Testing periods are based on system risk levels.||Yes||Yes||Yes|
|Systems are NOT tested during operational periods.||Yes||No||No|
Fault models are used to generate and evaluate tests.
Fault models are developed to create the basis for identifying the difference between a desired and undesired test outcome and to identify the class of faults that tests might be able to uncover. Without a fault model, testing is shooting in the dark without a clear target. With a fault model, it is possible to determine whether or not the tests are meaningful, redundant, and to what extent they provide "coverage".
Coverage of tests are measured against the fault model.
Coverage is a measurement against the fault model used to express the percentage of faults that the tests would detect if present or determine not to be present if they were not present. As such, it allows the tester to gain and provide clarity around the diagnostic utility of the tests for determining that the controls are in fact working as desired.
Testing periods are based on system risk levels.
The time taken to perform a test depends on the coverage of the test, the size of the test set, and the time per test. Since complete coverage of most fault models in most cases takes a very long time, periodicity of testing is traded off with coverage and test complexity. The tradeoff is inherently limited by the risk of the control failing without that failure being noticed. Hence, the periodicity of the test process is driven by the exposure from undetected control failure which then limits the coverage for the fault model and test times.
Systems containing authoritative high-valued
content are NOT tested during operational periods.
Because systems with high consequences of failure can fail because of a test, testing is often limited to test systems that are as close as possible to operational systems (for validity) or limited to testing during non usage periods such as maintenance windows (when the consequences cannot be induced). It is also important that after testing the unit under test be put back into its proper operating (i.e., original) condition and that such condition be properly verified before going operational. Otherwise, residual effects of the test may produce the potentially serious negative consequences.