Kexi/Junior Jobs/Design and perform Kexi-LibreOffice benchmarks

Status: UNASSIGNED, Difficulty: MEDIUM

Proposed and mentored by Jstaniek (talk) 14:13, 26 March 2014 (UTC)

The Goal

Document pros and cons of Kexi vs LibreOffice Base using performance/stability/resource consumption benchmarks.

Rationale

For many reasons opinions about software is often subjective. We would like to identify following in Kexi:

• strengths,
• potential for further development,
• most important areas to improve.

The Task

In order to do a Kexi-LibreOffice comparison:

1. Design performance and stability benchmarks
2. Perform the benchmarks

Below we call Kexi and LibreOffice Base as a product.

Requirements

1. Include single-user usage only
2. One version of each product should be selected, and it should be as new as possible
3. Intersection of features and parameters present in both products should be covered by the benchmark
4. One database project should be designed for one product
5. In each database, a number of objects of type (table, query) should be designed
6. Tables should be filled with appropriate amount data
7. Database projects built for the two products should be as equivalent as possible
8. Design documents and project files should be made available under the GNU FDL license
9. Design documents should be put inline on this wiki page, in a subsection
10. Tests should be reproducible given the same hardware, so relevant scripts or recipes should be provided
11. Benchmarks should be performed on at least two operating systems, each on different hardware architecture/model (e.g. with differences in hardware performance of disks, RAM, amount of memory)
12. The operating systems have to be both Linux-based (because Kexi for other OSes may not be tested enough at the moment)
13. User and system time should be measured whenever possible
14. It may be possible that alteration to product's configuration or even source code for the needs of this benchmark. Therefore, it is preferred to have Kexi built from source code so it can be easily adapted; this is not required to LibreOffice Base.
15. Release or ReleaseWithDebugInfo (not Debug or DebugFull) build or equivalent should be used to build product(s)
16. Memory-related benchmarks should be designed and performed too
17. KDE's System Activity "Detailed Memory Information" too should be used for memory reports. Read this to see how memory usage comparison may look.
18. Benchmark report should include:
    1. Reference to the benchmark design documents and test projects/data
    2. Detailed hardware information (types of components)
    3. Operating system information (version, distribution, versions of relevant dependent packages, e.g. Qt or SQLite)
    4. Detailed information about both product: their relevant configuration, version, source code patches (if applied)
    5. Result for each test for each product; median of time (or memory or other resource) for N runs for should be computed, N >= 10
    6. Remarks about accuracy of the test and limitations
19. Future expansion of the benchmark should be documented, e.g. new features to benchmark