History | Log In     View a printable version of the current page. Get help!  
Issue Details (XML | Word)

Key: WT-252
Type: Task Task
Status: Closed Closed
Resolution: Fixed
Assignee: Marc Guillemot
Reporter: Marc Guillemot
Votes: 0
Watchers: 3
Operations

If you were logged in you would be able to see more operations.
WebTest

Simplify and extend report file

Created: 20/Dec/06 14:26:14   Updated: 20/Feb/08 17:16:39
Affects Version/s: 2.1
Fix Version/s: 2.6

File Attachments: 1. Zip Archive CAPE_ReportMods.zip (143 kb)

Issue Links:
Depends
 
This issue contains:
WT-277 Provided additional and better struct... Closed


 Description   
Standard report mode generates a xml file and saves all received responses in the same directory. This works fine but...

1) report format should be simplified
for instance
<step>
  <parameter name="description" value="Enter the application"/>
  <parameter name="label" value="login"/>
  <parameter name="taskName" value="clickLink"/>
  <result>
    <completed duration="1668"/>
   </result>
    ...
could be simplified in something like
<step taskName="clickLink" description="Enter the application" status="completed" duration="1668">
  <parameter name="label" value="login"/>

making xslt selectors easier to write

2) Additional information could be provided in the report like:
- failure cause (more readable than stack trace - which should still be provided -)
- step locations
- unevaluated property values
- javascript warnings & errors

3) it doesn't scale very well
the result of each single test is appended to the result file (what requires complete parsing each time) producing huge result files for larger test suites.
A solution could be to create a subdirectory per <webtest> wich could contain the xml result file as well as all received responses. This would imply modification in the generation of the html reports but would bring more flexibility: possibility to simply delete all results of a single <webtest>, to add the results of an other one, ...

Such changes wouldn't impact user of the standard WebTest stylesheet for report generation but could cause problem to the one who have customized the reports. Depending of the number of persons impacted different ways can be followed:
- just perform the switch when it's ready, users of customized xslt can upgrade once they've adapted their reports
- provide the current report generator separately (for instance on the community site)
- maintain both report modes

Comments are welcome.

 All   Comments   Change History      Sort Order:
Comment by Denis N. Antonioli [21/Feb/07 08:08:50]
(1) would break all existing stylesheets. Although the present format may not be ideal (?), it is working.

Breaking code just for beauty's sake is a very bad thing!

Comment by Marc Guillemot [21/Feb/07 14:32:08]
According to discussion on the mailing list, custom stylesheet are seldom therefore not so many users would be impacted.

Comment by Marc Guillemot [21/Feb/07 15:18:55]
Additional motivation for these changes: current xsl takes huge time (hours) on very large result files (~100 webtest à 1000 steps) to generate the html reports. Furthermore recent improvements of the standard xsl have made this task even more time intensive.

Comment by Denis N. Antonioli [21/Feb/07 19:05:52]
I lost a lot of time (and that means weeks, not hours) changing existing tests to follow the changes we foolhardily made to webtest.

Staying compatible with previous versions should belong to our goals.

The present report format is part of the API of webtest and it fulfills its function.

I'm opposed to pure cosmetic changes such as described by (1)

Comment by Denis N. Antonioli [21/Feb/07 19:06:18]
On (2) I agree, this could be done on a backward-compatible way.

Comment by Denis N. Antonioli [21/Feb/07 19:07:34]
On (3), yes, this is sometime useful, and it can also be solved in a backward-compatible way.

Comment by Marc Guillemot [22/Feb/07 10:15:31]
The order of the 3 points exposed here isn't related to their significance. (2) or (3) are the motivations and IF compatibility needs to be broken, then (1) should be considered too.

This is not the right place to make such declaration about broken compatibility in your tests. If you have issues, please bring them to the mailing list.

I only partly agree with your sentence "Staying compatible with previous versions should belong to our goals. ". It is incomplete. Otherwise it means that a bad decision can never been fixed. Personally I recognize that I make (many) errors but because I recognize them, I can fix them.

Comment by Marc Guillemot [22/Feb/07 10:20:05]
According to the discussion on the mailing list, the WebTest users that adapted the reporting for their own needs are extremely rare. I think this has different reasons (absolutely not sorted by significance):
- the stylesheet provided with WebTest is already quite good
- not so many people are comfortable with XSLT
- the xml report format is over complicated what causes many selectors in XSLT too be over complicated too (see WebTest standard stylesheet)
- WebTest standard stylesheet doesn't provided appropriate extension points allowing to simply extend the stylesheet

Comment by Marc Guillemot [20/Feb/08 17:16:39]
(1) and (3) are now implemented and (2) is partially implemented. A new issue could be opened to extend current implementation of (2) when needed, but I close this issue now.