|
|
|
|
|
|
| |
WebDevelopers Ltd.
http://www.webdevelopers.co.nz
Feb, 2003
Abstract
Robust testing of large, complex web applications is an increasing concern, but such software systems are difficult to analyze because they are dynamic in nature, and inherent variability in its output. To support web testing research, this report will explore some of the problems associated with testing for the web, and make an attempt to provide some of the solutions arising from experience and research into this field. The concepts behind the proposed theory arise from experience and the nature of problems arising while building dynamic web applications. Generalized Software testing concepts are explained, which leads to a testing model for web software.
Contents
1 Introduction
1.1 Topics Covered
1.2 Aim of Testing
2 Testing in the Web Development Lifecycle
2.1 Cost of Testing
3 Problems with web applications
3.1 Problems within the Application
3.2 Problems Trying to Test
3.3 Example 1
3.4 Example 2
4 Automated Testing Techniques
4.1 Testing Tools
4.2 Comparators
4.3 Oracle
4.4 Regression Testing
5 Techniques for Web Applications Testing
5.1 Unit Testing using FuseBox Architecture
5.2 Acceptance Testing
6 Recommendations
1. Introduction
Software testing is a vital part of quality assurance. Unfortunately, no widely accepted testing frameworks exist which are suited in their entirety for web applications.
Attempts have been made to provide testing mechanisms for web applications. In the recent past these mostly relied on a fragile manner of testing link validity and HTML syntax. These methods are only testing the interface, not the software.
Better methods have been proposed which emphasize testing each component. Unit testing to test the output of each component is widely used along with common programming languages, ASP, JSP, Perl, PHP and Cold Fusion. Although a vital step, this on its own does not provide the robustness needed. For example, testing an include file <cfinclude qrySelectCustomerById.cfm> will most likely lead to a success for an experienced developer. However, logic of the entire working of a complex system may contain flaws which cannot be noticed by testing the functionality of each individual component.
More recently, the research community at Software Institutions and Universities have been exploring towards more robust testing techniques which can be built into the web development lifecycle. These include acceptance testing, integration and stress testing, and more importantly a way to automate these in order to regression test.
1.1 Topics Covered
The rest of this paper explores this developing area of robust software testing for web applications.
· Sections 2 and 3 state the problems specific to web applications and explain why a testing mechanism is necessary.
· The concept of software testing and techniques used in Computer Science are briefly overviewed in section 4.
· The next section describes testing techniques specific to web applications.
Reading only the final two sections section should allow successful implementation of a testing mechanism, however the previous sections are recommended for understanding the concept of software testing in general.
1.2 Aim of Testing
The aim of software testing is not to reveal the absence of errors, but their presence. This concept does not fit well with the view of a developer, however it must be understood and accepted in order to build a good test suite.
A successful defect test is a test which causes a program to behave in an anomalous way.
Importance of this concept can not be over emphasized. Test cases must be built with the aim of finding defects, not hiding them.
2. Testing in the Web Development Lifecycle
Whether the software development lifecycle model is waterfall, evolutionary, incremental, spiral, or almost any other well known software process model, they all share common activities.
1) Software specification
2) Design and Implementation
3) Software Validation
4) Software Evolution (maintenance)
The testing process can be broken up into 5 stages. Its relationship with each other is shown below.
|
|
 |
These can be integrated into the Process Model.
Requirements Specification:
Acceptance test plans created
System Specification:
System integration test plan created
System Design:
Sub-system integration test plan created
Detailed Design/ Implementation:
Stress test, and module/unit tests created
Software Validation:
Integration testing, Stress testing
Explanation of each of the stages of testing is beyond the scope of this paper. The two most important tests seen by a web developer should be acceptance tests and module or unit tests. These two remain the focus of last two sections. Stress testing is also vital in some high traffic websites.
2.1 Cost of Testing
Depending on the software process model and specific needs of the software, approximately 30 – 50 % of system resources and cost for a project should be dedicated to testing.
Many development methodologies such as eXtreme Programming, and FLIP involve testing as part of the development lifecycle. XP however, encourages test-first programming approach.
Test-first programming puts emphasis on testing right before development even commences. The beauty of this approach is that even though a lot more resources have been allocated to the testing phase initially, it saves even more time and money on debugging and fault correction.
The better the methods we use to design and integrate testing into our process, the more effective we are at understanding what to test and when to stop testing. The more we understand about measuring our effectiveness, the more likely we are to build good, useable software.
3 Problems with web applications
3.1 Problems within the Application
There is no set list, the problems which can occur vary can depend on the application you are building, technologies used, settings and environment, user software configuration and so forth. A developer will often find that with every new project comes a new set of problems. Only experience can better one at trying to predict the problems before they arise. Here is a list of a few common areas of fault that can arise with web applications.
Interface
- Broken HTML, Invalid syntax
- Dead Links
- Javascript errors
- Layout does not conform to required template
- Embedded object missing
- Different layout on different browser
- Different layout on different screen resolutions
Cookie
- Cookie expiration
- Cookie holds wrong data
- Session expiration (often enforced by the server)
- Inexistence of a required variable due to session timeout throws error
- Data entered by users in previous steps of a guided process lost due to session expiration.
- Users with cookies disabled not handled Different scripts using the same name for a cookie, overwriting each other
Form
- Wrong protocol
- Fields missing
- Form field’s name is incorrect
- Javascript action on submission fails
- Required fields left blank
- Input data longer than max length handled by software component
- Input data of a field longer than maxlength of database field being populated
Database
- Query syntax error
- Wrong logic in the query
- Slow response time
- No indication of rollback from a failed transaction
General
- Error handling suppresses the error, which results in wrong representation of information to the user
- Server crashes due to high resource usage
- Jumping onto a page (e.g. book-marking) that requires prior action. For example authentication or previous steps of a guided process like visa application.
3.2 Problems Trying to Test
- Interface errors easier to detect, but JavaScript functionality is hard to test.
- Detecting session and cookie errors requires simulating a browser, and large time is required to cause timeouts.
- Hard to attempt all possibilities of entries in a given form submission.
- Database errors can be detected by unit testing.
- Unexpected output due to suppression of a database error need to be detected.
- Stress testing hard to perform on a shared hosting environment, and while the site is live.
- Automated testing usually populates the database heavily with generated test cases. These not only need to be cleared out, but also put stress on a live site. It may also cause serious implications to consistency of data. For example an auto number ID for a customer maybe incremented by 100-500 integers during one single test.
- We require a way to reverse these changes.
- The site may have to go down while being tested. Testing on a mirrored test site may not provide the same environment.
3.3 Example 1
A form with 2 input text boxes, 1 checkbox and 2 radio button. 1 text box is required, rest are optional.
Test Cases: |
|
Fill Minimal
Fill Maximum
Foreach form object in the maximum fill {
Try boundary values in that object
}
// boundary values for text box
maximum length
minimum length
all spaces
zero length
more than maximum length
//boundary values for checkbox
empty
checked
//boundary values for radio button
both unchecked
First one checked
Second one checked
|
|
|
|
Thus there are a total of 12 test cases. Combinatorial testing is not required with web applications as the impact of one input on the other is unlikely, and should be detected by unit testing.
3.4 Example 2
Application system with three steps |
|
 |
Ideally we test all 6 combinations of form input required from step 1. Assuming all pass, we can conclude that step 1 -> 2 is faultless.
Then we test all 9 ways of going from step 2 -> step 3. Assuming all pass, we can conclude that step 2 -> 3 is faultless.
Then we test all 3 ways of going from step 3 -> completion. Assuming all pass, we can conclude that step 3 -> completion is faultless.
Thus the number of paths equals 18.
But..
Now lets assume more realistic scenario. Step 2 and 3 carry the input as hidden fields. The last completion step inputs all details into a database. Assume one of the 6 combinations of input from step 1 results in SQL error. Clearly this may not be detected by the previous approach.
The correct combination of tests would be as follows.
Test 6 ways of going from step 1 to 2. For each combination, test 9 ways of going from step 2 to 3. For each that combination, test 3 ways of going from step 3 to completion.
Thus there will be 162 (6*9*3) different paths tested!
Problem: A simple 3 step application with 10 input boxes each and 10 checkboxes would result in 2+(10*2) + (10*2).. at least 42 combination of form input at each step. Combining this would result in 73,088 (42 * 42 * 42) paths to be tested!
Number of queries made are: 42 for the first step. 42 * 42 for the second and 42 *42 *42 for the third, resulting in 75895 URL queries.
Unless your aim is to crash the web-server, this method is unacceptable.
Possible Solution 1:
Test all 42 combinations of going from step 1 to 2
Test all 42 combinations of going from step 2 to 3
Test all 42 combinations of going from step 3 to finish
Since the last step is inputting 60 fields of data into the database, test all boundary values of each, NOT combination of each with the other.
Thus there will be 120 (60*2) combinations, plus 2 (All empty all filled). These boundary values have to be in accordance with unit’s input criteria.
Possible Solution 2:
Unit test each component.. step 2, 3, completion.
Work out and see if input can accept values beyond the boundary values specified. Test only those.
4 Automated Testing Techniques
Testing is an expensive process phase. Testing workbenches provide a range of tools to reduce the time required and total testing costs. |
|
 |
4.1 Testing Tools
- Test Design tools : Help generate test inputs
- Physical Design tools – derive by choosing existing data or generating test data (e. g. may randomly choose records from a database)
- Logical design tools – derive from specifications logic
- Test Management tools: assist in test planning, keeping track of which tests have been run ,etc.
- Test Analysis tools
- Static analysis tools: analyze code without executing (It may calculate metrics such as cyclomatic complexity)
- Coverage Tools: determined how much of the software has been covered by a set of tests carried out
- Dynamic analysis tools: assess the system while the software is running. (e. g. tools that can detect memory leaks)
4.2 Comparators
When we automate test cases we need to determine ahead of time what the output will be.
A comparator is a computer program that detects differences between two sets of data. It will highlight differences among two output result sets and/ or changes to databases/ files but it cannot tell you if a test has passed or failed. It only shows us where tests differ. The tester must determine whether or not something has failed when a difference is discovered. This approach suffers the consequence of human interaction; if the predicted outcome was incorrect then the test also is incorrect.
Simple Comparisons:
Looks for exact matches between expected and actual outputs. This however, will not work when there are differences in the outputs which are allowable, time for example, or name of a person.
Intelligent Comparisons:
Allows us to compare expected and actual outcomes that have known allowable differences. These will ignore certain types of differences, and need to be coded as a comparator specification.
The Acceptance testing method described in Section 5.3 is based on Complex Comparisons of output code. |
|
 |
4.3 Oracle
The term Oracle describes expected output in a test case. With Acceptance Testing for web applications we describe a simple method of specifying oracle to the test result.
We have defined an XML syntax for defining oracles to be compared in the html/javascript output. The syntax comprise of two main tag identifiers <oracle> and </oracle> , which can be nested multiple levels. Scope of any parameter not defined in the child is passed on from parent to child. The child oracle overwrites parameters defined by the parent.
Example:
|
|
// html language avoids formatting consideration
// whereas text language would need to compare the formatting
<oracle scope=html avoidSpaces=yes>
<br>
<p>This is the content of successful page.</p>
<table><tr><td>Yes</td></tr></table>
<img src=””>
<oracle scope=htmltext>
Text with any <b>formatting</b>
<font face=arial>I don’t only match text</font>
</oracle>
<p>Hello <oracle scope=dynamic query=firstName>[string]</oracle>! You have logged in <oracle scope=dynamic query=loggedTimes>[int]</oracle> times.</p>
Your last login was on: <oracle scope=dynamic query=dateRegExp>[any]</oracle>
Your last log out was on: <oracle scope=dynamic ignore=true>[any]</oracle>
// This validates integrity of form tags regardless of its placement in the actual
// document (As long as it is within content marked by its parent oracle tag)
<oracle scope=form>
<form method=”get” target=””>
<input type=hidden name=”” value=””>
<input type=submit>
</oracle>
</oracle>
<oracle scope=javascript>
var text = “I look for javascript syntax errors and compare each line”;
</oracle>
<oracle scope=query name=firstName>
SELECT name FROM tablename
WHERE id =”#comparaterInput.id#”
</oracle>
<oracle scope=query name=loggedTimes>
SELECT loggedTimes FROM tablename
WHERE id =”#comparaterInput.id#”
</oracle>
// standard unix regular expression
<oracle scope=query name= dateRegExp>
^[0-3][0-9] day, [01][0-9] month, [0-9]{4} year$
</oracle>
|
|
|
4.4 Regression Testing
As the number of detected faults increases, the probability of the existence of more undetected faults also increases. Therefore is we find a lot of faults in our software there are also likely to be a lot undetected. - Meyers( 1979)
This makes it hard for us to know when to stop looking for faults during testing. We need to be able to estimate the number of remaining faults to know when to stop and so that we can have confidence in our testing success .
Regression test describes the automated task of re-testing software to validate previously working functionality.
We must regression test a web application when:
- Server configuration has changed
- Server software has been updated
- A new commit has been made to the CVS or other versioning system
- A bug fix has been completed
- A new system build has been generated
- Integration testing need to be carried out
- A system has stabilized and the final release build has been generated.
5 Techniques for Web Applications Testing
Testing web applications is not trivial even for an experienced tester with all the tools available today. Web applications testers should rely on at least the following techniques to ensure robustness:
- Unit testing
- Tools such as JUnit are very helpful (http://www.junit.org)
- However, it does not scope outside of the Java world.
- It is ideal to develop web applications as components and be able to dynamically test each component individually by considering the input/outputs, and attempting them through an automated tester or a similar script which has to be written.
- Acceptance testing
- Tests the final output with regards to its specification. We recommend HTTPUnit (http://www.httpunit.org), and also custom-written comparators as mention in the previous section.
- This technique compares the output with the predicted one, derived from initial specifications (or often designed as a specification).
- Site traversal algorithm may be written, or explicit URLs defined.
- Stress testing
- Take any unit tests / acceptance tests and overload the server by making numerous requests. Test for server response as well as problems due to concurrency, synchronization and race-conditions.
- Regression testing
- Test everything you’ve tested before! This is the only way to validate whether the new amendments to the software have introduced any flaws to the functionality working previously.
The following factors have been identified to be tested:
Presentation (Acceptance Testing)
Navigation (Acceptance Testing)
Functional (Acceptance / Unit testing)
Security (Unit testing/Manual)
Performance (Stress testing)
5.1 Unit Testing using FuseBox Architecture
FuseBox (http://www.fusebox.org) is a framework defined for web applications development. The reason why it is helpful is that it relies on Component Based Software Engineering method, even though there is no mention of CBSE in its documentation! FuseBox isn’t the only candidate, Unit testing is ideal for any Component Based Software Architecture.
Details of such a system can get extremely complicated and are beyond the scope of this article, thus we will stick to the simplicity of FuseBox framework. If you’ve worked with other, more dynamic, component architectures where interfaces can by dynamically downloaded and queried (For example WebServices), then you probably already know how unit testing works.
FuseBox recommends defining its interface as a FuseCode (Its just a syntax really, similar to IDL file for CORBA/RMI). The important part of Fusedocs is the <IO> section. It clearly defined the inputs and outputs passing through a “Fuse” (Component)
”..it's possible to test a display fuse without a database. Even if the display fuse is displaying records from the database. All we need to know are what <in> variables the fuse will be given. If the fuse will be given a recordset, we simply need to simulate that incoming recordset so we can see what happens when that data comes in.
This is the case for any incoming variable. We can fake each incoming variable to simulate how the fuse will react to different incoming variables. As a tester we need to make a decision whether it's reacting properly or not.
This where the <out> variables come into play. Using fusedocs it's possible to validate whether the data being produced by the fuse matches up with the necessary <out> data in the fusedoc. ” - FuseBox People
(http://www.fusebox.org/index.cfm?&fuseaction=methodology.UnitTestFuses)
5.2 Acceptance Testing
Acceptance testing is all about testing whether we have met the initial specified requirements. It can be manual as well as largely automated, but the latter is preferred.
We make use of the comparator and the oracle definition mentioned in section 4.2, and 4.3, respectively, to automate our process of acceptance testing an entire website with 100s of pages and complex structure.
An alternative to this is the HTTPUnit which lets you code acceptance testing based on objects sitting in an HTML page. The disadvantage of this approach, first of all, is that it does not support Javascript, which is too unreasonable to restrict a developer to avoid. Besides that, unless unit tests are exhaustively coded from the desired HTML output page, you will come across large number of situations where a test will pass but the output doesn’t seem acceptable. |
|
 |
The approach described here, in Figure 5.1, is quite powerful if used correctly. We assume the standard 3-tier architecture, of which Application and Data tiers have been illustrated, the client tier is irrelevant and can be considered as the testing-tier in this case.
First of all, we need a second database test mirror, which is dynamically chosen by the reflector when a test suit queries. The Test Runner possesses an algorithm for traversing through the site structure, or implicit knowledge of URLs to query with appropriate data.
Although URLs to query may be statically hard-coded, testing of data input/output should be dynamic. All boundary values and sub-normal conditions must be tested.
The Test Suit with Comparator queries the application as requested by the Test Runner, and retrieves the output to be tested. It takes two further inputs, the expected output (oracle) defined in an HTML-type file, but contains our previously defined XML tags for defining oracle. Optionally it may also take a DTD to validate and process the XML.
At this point the Comparator has all the information it needs, i) data to be tested, ii) expected output, and iii) the knowledge of complex comparisons
Using the above method it is fairly easy to make intelligent comparisons within a dynamic test environment.
6 Recommendations
All web applications must be tested. It is recommended to think of testing mechanisms at designing stage of the process, and even write few initial tests. Acceptance tests must be generated in parallel with the specifications.
Tests should cover, but not bias towards, five of the following aspects of a applications:
- Presentation – Besides HTML validation, appearance in most browser / configuration must be manually tested.
- Navigation – Not just link validation checking, test access to all pages and content as per initial specifications.
- Functional - Unit and Acceptance testing covers this aspect.
- Performance – Covered by stress testing. Any application will fail at some stress, be reasonable, estimate the expected load and keep an extra safety margin.
- Security – This has to be carefully planned at design time, Unit and Acceptance testing are well suited, however certain types of flaws can only be revealed through manual verification.
|
|
|
|
| |
|
|
