Kexi/Junior Jobs/Design and perform Kexi-LibreOffice benchmarks

Status: UNASSIGNED, Difficulty: MEDIUM

Proposed and mentored by Jstaniek (talk) 14:13, 26 March 2014 (UTC)

The Goal

Document pros and cons of Kexi vs LibreOffice Base using performance/stability/resource consumption benchmarks.

Rationale

For many reasons opinions about software is often subjective. We would like to identify following in Kexi:

• strengths,
• potential for further development,
• most important areas to improve.

The Task

In order to do a Kexi-LibreOffice comparison:

1. Design performance and stability benchmarks
2. Perform the benchmarks

Below we call Kexi and LibreOffice Base a product.

Requirements

1. One version of each product should be selected, and it should be as new as possible.
2. Intersection of features and parameters present in both products should be covered by the benchmarks.
3. Only include single-user usage scenarios.
4. Only include scenarios available through using the products. For example tests cannot require command-line use of SQLite database.
5. One database project should be designed for one product.
6. In each database project, a number of objects of type (table, query) should be designed.
7. Tables should be filled with appropriate amount data.
8. Database projects built for the two products should be as equivalent as possible.
9. Design documents and project files should be made available under the GNU FDL license.
10. Design documents should be put inline on this wiki page, in a subsection.
11. Tests should be reproducible given the same hardware, so relevant scripts or recipes should be provided.
12. Benchmarks should be performed on at least two operating systems, each on different hardware architecture/model (e.g. with differences in hardware performance of disks, RAM, amount of memory).
13. The operating systems have to be both Linux-based (because Kexi for other OSes may not be tested enough at the moment).
14. User and system time should be measured whenever possible.
15. Disable any unnecessary applications and service that may consume resources during the benchmark: HTTP, other databases, screen savers, other applications etc.
16. Unless concurrent behavior is tested, all tests should be performed sequentially without a need to run more than one product at a time.
17. It may be possible that alteration to product's configuration or even source code for the needs of this benchmark. Therefore, it is preferred to have Kexi built from source code so it can be easily adapted; this is not required to LibreOffice Base.
18. Release or ReleaseWithDebugInfo (not Debug or DebugFull) build or equivalent should be used to build product(s).
19. Memory-related benchmarks should be designed and performed too.
20. Cold and warm start of applications loading tested projects should be measured too (user/system time).
21. KDE's System Activity "Detailed Memory Information" too should be used for memory reports. Read this to see how memory usage comparison may look.
22. Benchmark report should include:
    1. Reference to the benchmark design documents and test projects/data.
    2. Detailed hardware information (types of components, amount of resources).
    3. Operating system information (version, distribution, versions of relevant dependent packages, e.g. Qt or SQLite).
    4. Detailed information about both product: their relevant configuration, version, source code patches (if applied).
    5. Result for each test for each product; median of time (or memory or other resource) for N runs for should be computed, N >= 10. The form should be a table and a chart; use the ODS format.
    6. Remarks about accuracy of the test and limitations.
23. Proposals for future expansion of the benchmark should be documented, e.g. new features to benchmark.

Hints

• Study publicly explained benchmarks of database or desktop software to identify areas tested and methodologies.