Design for Testability
http://www.io.com/~wazmo/papers/design_for_testability_PNSQC.pdf
1. Introduction
This paper defines testability as visibility and control. Visibility is our ability to
observe the states, outputs, resource usage and other side effects of the software under
test. Control is our ability to apply inputs to the software under test or place it in
specified states. Ultimately, testability means having reliable and convenient interfaces
to drive the execution and verfication of tests.
There's been some controversy regarding the merit of using test interfaces as opposed to
testing via the user interface.
2. Visibility Basics
Visibility means the ability to observe the outputs, side effects and internal states of
the software under test.
3. Verbose Output
The verbose modes found in many programs are good examples of testability features that
allow a view into the software operation.
4. Logging events
Good logs contain the information needed to replicate.
ex)Other events to consider logging
- Significant processing in a subsystem.
- Major system milestones. These include the initialization or shutdown of a subsystem or
the establishment of a persistent connection
- Internal interfac crossings.
- Internal errors
- Unusual events. These may be logged as warnings and a sign of trouble if they happen too
often.
- Completion of transactions.
What else can you do with logs?
- Detect internal errors before they appear at the system level.
- Find out how to reproduce bugs
- Learn about your test coverage. What messages haven't your tests yet triggered?
- Learn about customer usage. What features do they use the most? What kinds of mistakes
do they make? Focus your testing on what customers actually do.
- Determine which customer configurations are most common and focus testing on them.
- Uncover usability problems. Are there certain pages that lead to a high level of user
abandonment?
- Understand load patterns and use these as a basis of setting realistic load test
objectives.
5. Diagnostics, Monitoring, and Fault Injection
...
7. Test Interfaces
Software testing occurs through an interface. It might be a graphical user interface(GUI).
It might be an application programming interface(API), a component interface(e.g. COM,
ActiveX, or J2EE), a protocol interface(e.g. XMI, HTTP, SOAP), or a private debugging or
control interface.
Programming interfaces often support automated testing earlier and with less effort than
required when testing via a GUI.
I've often recommanded the use and development of programming interfaces for testing.
8. User Interface Testability(GUI cotrol recognition)
This paper highlishgts serveral testability issues faced by GUI test automation:
- a lack of standards makes GUI testability often a matter of trial and error.
- GUI control recognition technology requires instrumentation.
- GUI test automation requries programmers to adhere to naming standards.
- Custom controls represent a major challenge for test automation.
9. User Interface Standards and Compatibility(test tool, development tool)
You have to actually try the tools out to verify whether a particular test tool will be
able to execute and verify the interfaces created by the development tool.
It seems that the test and development tool vendors lack the motivation to work together.
The development of standard component technologies(J2EE, .NET, COM, VB) allows tool
vendors to provide the glue needed to make their tools easier to customize.
10. Recognizing User Interface Controls
The first step in GUI testability is simply getting the tool to see the individual
controls.
11. Naming Windows and Interface Controls
12. Mapping Custom Interface Controls
13. Supporting Custom Interface Control
14. Accessing Internal Interface to Custorm Controls.