Kexi/Junior Jobs/Design and perform Kexi-LibreOffice benchmarks

Status: UNASSIGNED, Difficulty: MEDIUM

Proposed and mentored by Jstaniek (talk) 14:13, 26 March 2014 (UTC)

The Goal

Document pros and cons of Kexi vs LibreOffice Base using performance/stability/resource consumption benchmarks.

Rationale

For many reasons opinions about software is often subjective. We would like to identify following in Kexi:

• strengths,
• potential for further development,
• most important areas to improve.

The Task

In order to perform a Kexi-LibreOffice comparison:

1. Design performance and stability benchmarks
2. Perform the benchmarks
3. Provide results

Student should provide schedule with all subtasks to mentor for approval.

Below we call Kexi and LibreOffice Base a product.

Requirements

1. One version of each product should be selected, and it should be as new as possible.
2. Intersection of features and parameters present in both products should be covered by the benchmarks.
3. Only include single-user usage scenarios.
4. Only include scenarios available through using the products. For example tests cannot require command-line use of SQLite database.
5. One database project should be designed for one product.
6. In each database project, a number of objects of type (table, query) should be designed.
7. Tables should be filled with appropriate amount data.
8. Database projects built for the two products should be as equivalent as possible.
9. Design documents and project files should be made available under the GNU FDL license.
10. Design documents should be put inline on this wiki page, in a subsection.
11. Tests should be reproducible given the same hardware and OS is used, so relevant scripts or recipes should be provided.
12. Benchmarks should be performed on at least two operating systems, each on different hardware architecture/model (e.g. with differences in hardware performance of disks, RAM, amount of memory).
13. All the operating systems have to be Linux-based. That is because Kexi for other OSes may not be tested enough at the moment.
14. User and system time should be measured whenever possible.
15. Disable any unnecessary applications and services that may consume resources during the benchmark: HTTP, other databases, screen savers, other applications etc.
16. When running the tests, use separate user account dedicated to performing the tests.
17. Unless concurrent behavior is tested, all tests should be performed sequentially without running more than one product at a time.
18. It is allowed to alter product's configuration or source code for the needs of tests, for example to intercept some operations or output additional debugging data. Therefore, it is preferred to have Kexi built from source code so it can be easily adapted; this is not required but allowed for LibreOffice Base. Mentor does not offer support to the student regarding building of LibreOffice.
19. Release or ReleaseWithDebInfo (not Debug or DebugFull) build type or equivalent should be used to build product(s).
20. Memory-related benchmarks should be also designed and performed.
21. Cold and warm start of applications should be measured too with and without loading of tested projects (user/system time).
22. KDE's System Activity "Detailed Memory Information" too should be used for memory reports. Read this to see how memory usage comparison may look.
23. Benchmark report should include:
    1. Reference to benchmark design documents, test projects and test data.
    2. Detailed hardware information (types of relevant components, amount of resources).
    3. Operating system information (version, distribution, versions of relevant dependent packages, e.g. Qt or SQLite).
    4. Detailed information about products: their relevant configuration, version, source code patches (if present).
    5. Results for each test per product; individual N runs and median (of time or memory or other values) should be computed, N >= 10. Use tabular form and a chart; use the ODS format for publication.
    6. Remarks about accuracy of the tests and limitations, proposals for future expansion of the benchmark, e.g. new features to test.

Hints

• Study publicly explained benchmarks of database or desktop software to identify areas tested and methodologies.