Calligra/Testing

Blackbox testing

This section is dedicated to black box testing at the moment. There are two sets of test files. The first one is used to test input filters by comparing the output of the calligraconverter tool against the expected output. These are feature based files prepared by a testing team sponsored by Nokia and are primary intended to filter developers. The corresponding test tool is a shell script applying XSL templates. The second set is comprised of real world documents and it is used in combination with the cstester and visualimagecompare tools as described below. The purpose is to test both the filter and the layout engine and to display differences in document layout in means of visual output.

Getting the test files

Checkout test files from the KDE subversion repository:

$ svn co svn+ssh://[email protected]/home/kde/trunk/tests/calligratests

Download test files for cstester:

$ wget http://www.valdyas.org/~boud/documents.tar
$ wget http://www.valdyas.org/~boud/documents_pdf.tar

Filter tests

There are two important directories in calligratests at the moment, the import and the interoperability directory, which have the following 1st level structure of interest (list of subdirectories containing tests):

import
|-- excel
|-- msword
|-- powerpoint

interoperability/
|-- presentations
|-- spreadsheets
|-- wordprocessing

Each of the listed directories contains a Makefile. To start tests, make sure the import/share/bin directory is in your $PATH environment variable and run make.

cstester

It's more convenient to use the cstrunner tool. Please check the following file present in your working copy of calligra for further instructions and details: calligra/devtools/cstester/README.txt

cstrunner

A tool to run cstester in multiple processes. It creates a report of files terminated by a signal. This can happen as a result of a bug in calligra or when the allowed amount of time or CPU usage has been exceeded. The md5.txt file providing the ms5sums of generated thumbnails is created in each output directory. The following scripts are used by cstrunner. Make sure calligra/devtools/cstester/scripts is included in your $PATH environment variable.

cstwrapper.sh - Limits the resources cstester is allowed to use and makes sure the correct exit code is returned to cstrunner. cstmd5gen.sh - Generates md5sums for a given thumbnail directory and stores them into the md5.txt file.

cstrunner expects the following parameters:

$ cstrunner <docDir> <resultDir> <concurrentProcesses>

To run over all documents in the current directory and put the thumbnails in to ../result/sha1 with 4 concurrent processes:

$ cstrunner . ../result/sha1 4

Apply your changes and start cstrunner again:

$ cstrunner . ../result/sha2 4

To create the input file for visualimagecompare use the cstmd5diff.sh script having the following parameters:

$ cstmd5diff.sh <documents dir> <previous result dir> <current result dir>

To display changes between outputs from the previous two runs of cstrunner:

$ cstmd5diff.sh . ../results/sha1 ../results/sha2 > md5-sha1-sha2.log
$ visualimagecompare ../results/sha1 ../results/sha2 md5-sha1-sha2.log

visualimagecompare

List of supported commands :

PageUp, n	Move to the next page of the current document
PageDown, b	Move to the previous page of the current document
Up	  	Move to the previous document
Down		Move to the next document
Left		Switch to the left tab
Right		Switch to the right tab
Space		Switch between tabs

Static Code Analysis

Our idea is: use more than one tool to maximize scope of tests.

Following tools are available for Static C/C++ code analysis:

• Coverity - Boudwijn registered Calligra as open source project and tested once. TODO: Details
• PVS-Studio - free version for certain period of time kindly donated to Calligra, maintainer: Jaroslaw Staniek
• Prevent by Parasoft - status unknown
• Klocwork - status unknown, volunteers needed
• Clang Static Analyzer - free/open source, status unknown, volunteers needed

If you want to help, contact the mailing list [email protected].