Jump to content

Calligra/Testing: Difference between revisions

From KDE Community Wiki
← Older edit
Uzak (talk | contribs)
17 edits
Jstaniek (talk | contribs)
KDEDevelopers, spam-ninja, trusted
4,545 edits
 
(12 intermediate revisions by 3 users not shown)
Line 1: Line 1:
== Preface ==
== Blackbox testing ==


This page is dedicated to black box testing at the moment.  There are two sets of test files.  The first one is used to test input filters by comparing the output of the ''calligraconverter'' tool against the expected output.  These are feature based files prepared by a testing team sponsored by Nokia and are primary intended to filter developers.  The corresponding test tool is a shell script applying XSL templates.  The second set is comprised of real world documents and it is used in combination with the ''cstester'' and ''visualimagecompare'' tools as described below.  The purpose is to test both the filter and the layout engine and to display differences in document layout in means of visual output.   
This section is dedicated to black box testing at the moment.  There are two sets of test files.  The first one is used to test input filters by comparing the output of the ''calligraconverter'' tool against the expected output.  These are feature based files prepared by a testing team sponsored by Nokia and are primary intended to filter developers.  The corresponding test tool is a shell script applying XSL templates.  The second set is comprised of real world documents and it is used in combination with the ''cstester'' and ''visualimagecompare'' tools as described below.  The purpose is to test both the filter and the layout engine and to display differences in document layout in means of visual output.   


== Getting the test files ==
=== Getting the test files ===


Checkout test files from the KDE subversion repository:
Checkout test files from the KDE subversion repository:
Line 16: Line 16:
</nowiki>}}
</nowiki>}}


== Running tests ==
=== Running tests ===


=== Filter tests ===
=== Filter tests ===


There are two important directories in calligratest at the moment, the ''import'' and the ''interoperability'' directories, which have the following 1st level structure of interest (because those contain tests):
There are two important directories in calligratests at the moment, the ''import'' and the ''interoperability'' directory, which have the following 1st level structure of interest (list of subdirectories containing tests):
{{Output|1=<nowiki>
import
|-- excel
|-- msword
|-- powerpoint
</nowiki>}}


{{Output|1=<nowiki>
{{Output|1=<nowiki>
Line 28: Line 34:
|-- wordprocessing</nowiki>}}
|-- wordprocessing</nowiki>}}


{{Output|1=<nowiki>
Each of the listed directories contains a Makefile.  To start tests, make sure the ''import/share/bin'' directory is in your $PATH environment variable and run make.
import
|-- excel
|-- msword
|-- powerpoint
</nowiki>}}
 
Each of the listed directories contains a Makefile.  To start tests, make sure the ''import/share/bin'' directory is in your PATH environment variable and run make.


=== cstester ===
=== cstester ===


It's more convenient to use the cstrunner tool.  Please check the following file present in your working copy of calligra for further instructions and details: calligra/devtools/cstester/README.txt
It's more convenient to use the cstrunner tool.  Please check the following file present in your working copy of calligra for further instructions and details: ''calligra/devtools/cstester/README.txt''


==== cstrunner ====
==== cstrunner ====
A tool to run cstester in multiple processes.  It creates a report of files terminated by a signal.  This can happen as a result of a bug in calligra or when the allowed amount of time or CPU usage has been exceeded. The md5.txt file providing the ms5sums of generated thumbnails is created in each output directory.  The following scripts are used by cstrunner.  Make sure calligra/devtools/cstester/scripts is included in your $PATH environment variable.
A tool to run cstester in multiple processes.  It creates a report of files terminated by a signal.  This can happen as a result of a bug in calligra or when the allowed amount of time or CPU usage has been exceeded. The md5.txt file providing the ms5sums of generated thumbnails is created in each output directory.  The following scripts are used by cstrunner.  Make sure ''calligra/devtools/cstester/scripts'' is included in your $PATH environment variable.


cstwrapper.sh - Limits the resources cstester is allowed to use and makes sure the correct exit code is returned to cstrunner.
''cstwrapper.sh'' - Limits the resources cstester is allowed to use and makes sure the correct exit code is returned to cstrunner.
 
''cstmd5gen.sh'' - Generates md5sums for a given thumbnail directory and stores them into the md5.txt file.
cstmd5gen.sh - Generates md5sums for a given thumbnail directory and stores them into the md5.txt file.


cstrunner expects the following parameters:
cstrunner expects the following parameters:
{{Input|1=<nowiki>
{{Input|1=<nowiki>
$ cstrunner docDir resultDir concurrentProcesses
$ cstrunner <docDir> <resultDir> <concurrentProcesses>
</nowiki>}}
</nowiki>}}


To run over all documents in the current directory and put the thumbnails in to ../result/sha1 with 4 concurrent processes.
To run over all documents in the current directory and put the thumbnails in to ../result/sha1 with 4 concurrent processes:
{{Input|1=<nowiki>
{{Input|1=<nowiki>
$ cstrunner . ../result/sha1 4
$ cstrunner . ../result/sha1 4
Line 63: Line 61:
</nowiki>}}
</nowiki>}}


To create the input file for visualimagecomapre use the cstmd5diff.sh script having the following parameters:
To create the input file for ''visualimagecompare'' use the ''cstmd5diff.sh'' script having the following parameters:
{{Input|1=<nowiki>
{{Input|1=<nowiki>
$ cstmd5diff.sh <documents dir> <previous result dir> <current result dir>
$ cstmd5diff.sh <documents dir> <previous result dir> <current result dir>
Line 74: Line 72:
</nowiki>}}
</nowiki>}}


==== visualimagecompare ====
=== visualimagecompare ===
 
List of supported commands :
 
{{Input|1=<nowiki>
{{Input|1=<nowiki>
PageUp, n Move to the next page of the current document
PageUp, n Move to the next page of the current document
Line 84: Line 85:
Space Switch between tabs
Space Switch between tabs
</nowiki>}}
</nowiki>}}
== Static Code Analysis ==
Our idea is: use more than one tool to maximize scope of tests.
Following tools are available for [http://en.wikipedia.org/wiki/Static_program_analysis Static C/C++ code analysis]:
*Coverity - Boudwijn registered Calligra as open source project and tested once. TODO: Details
*PVS-Studio - free version for certain period of time kindly donated to Calligra, maintainer: Jaroslaw Staniek
*[http://www.parasoft.com/static-analysis Prevent] by Parasoft - status unknown
*[http://www.klocwork.com/products/insight/source-code-analysis/ Klocwork] - status unknown, volunteers needed
*[http://clang-analyzer.llvm.org/ Clang Static Analyzer] - free/open source, status unknown, volunteers needed
*[http://cppcheck.sourceforge.net Cppcheck] - free/open source, status unknown, volunteers needed
If you want to help, contact the mailing list [email protected].

Latest revision as of 19:59, 5 June 2014

Blackbox testing

This section is dedicated to black box testing at the moment. There are two sets of test files. The first one is used to test input filters by comparing the output of the calligraconverter tool against the expected output. These are feature based files prepared by a testing team sponsored by Nokia and are primary intended to filter developers. The corresponding test tool is a shell script applying XSL templates. The second set is comprised of real world documents and it is used in combination with the cstester and visualimagecompare tools as described below. The purpose is to test both the filter and the layout engine and to display differences in document layout in means of visual output.

Getting the test files

Checkout test files from the KDE subversion repository: 

$ svn co svn+ssh://[email protected]/home/kde/trunk/tests/calligratests

Download test files for cstester: 

$ wget http://www.valdyas.org/~boud/documents.tar
$ wget http://www.valdyas.org/~boud/documents_pdf.tar

Running tests

Filter tests

There are two important directories in calligratests at the moment, the import and the interoperability directory, which have the following 1st level structure of interest (list of subdirectories containing tests): 

import
|-- excel
|-- msword
|-- powerpoint

interoperability/
|-- presentations
|-- spreadsheets
|-- wordprocessing

Each of the listed directories contains a Makefile. To start tests, make sure the import/share/bin directory is in your $PATH environment variable and run make.

cstester

It's more convenient to use the cstrunner tool. Please check the following file present in your working copy of calligra for further instructions and details: calligra/devtools/cstester/README.txt

cstrunner

A tool to run cstester in multiple processes. It creates a report of files terminated by a signal. This can happen as a result of a bug in calligra or when the allowed amount of time or CPU usage has been exceeded. The md5.txt file providing the ms5sums of generated thumbnails is created in each output directory. The following scripts are used by cstrunner. Make sure calligra/devtools/cstester/scripts is included in your $PATH environment variable.

cstwrapper.sh - Limits the resources cstester is allowed to use and makes sure the correct exit code is returned to cstrunner. cstmd5gen.sh - Generates md5sums for a given thumbnail directory and stores them into the md5.txt file.

cstrunner expects the following parameters: 

$ cstrunner <docDir> <resultDir> <concurrentProcesses>

To run over all documents in the current directory and put the thumbnails in to ../result/sha1 with 4 concurrent processes: 

$ cstrunner . ../result/sha1 4

Apply your changes and start cstrunner again: 

$ cstrunner . ../result/sha2 4

To create the input file for visualimagecompare use the cstmd5diff.sh script having the following parameters: 

$ cstmd5diff.sh <documents dir> <previous result dir> <current result dir>

To display changes between outputs from the previous two runs of cstrunner: 

$ cstmd5diff.sh . ../results/sha1 ../results/sha2 > md5-sha1-sha2.log
$ visualimagecompare ../results/sha1 ../results/sha2 md5-sha1-sha2.log

visualimagecompare

List of supported commands : 

PageUp, n	Move to the next page of the current document
PageDown, b	Move to the previous page of the current document
Up	  	Move to the previous document
Down		Move to the next document
Left		Switch to the left tab
Right		Switch to the right tab
Space		Switch between tabs

Static Code Analysis

Our idea is: use more than one tool to maximize scope of tests.

Following tools are available for Static C/C++ code analysis:

  • Coverity - Boudwijn registered Calligra as open source project and tested once. TODO: Details
  • PVS-Studio - free version for certain period of time kindly donated to Calligra, maintainer: Jaroslaw Staniek
  • Prevent by Parasoft - status unknown
  • Klocwork - status unknown, volunteers needed
  • Clang Static Analyzer - free/open source, status unknown, volunteers needed
  • Cppcheck - free/open source, status unknown, volunteers needed

If you want to help, contact the mailing list [email protected].

Retrieved from "https://community.kde.org/index.php?title=Calligra/Testing&oldid=38016"