Kexi/Junior Jobs/Design and perform Kexi-LibreOffice benchmarks: Difference between revisions
< Kexi | Junior Jobs
Created page with "'''Status: UNASSIGNED''', Difficulty: MEDIUM Proposed and mentored by ~~~~ __TOC__ ==The Goal== Document pros and cons of Kexi vs LibreOffice Base using performance/stabilit..." |
|||
| Line 31: | Line 31: | ||
#Design documents and project files should be made available under the GNU FDL license | #Design documents and project files should be made available under the GNU FDL license | ||
#Design documents should be put inline on this wiki page, in a subsection | #Design documents should be put inline on this wiki page, in a subsection | ||
# | #Tests should be reproducible given the same hardware, so relevant scripts or recipes should be provided | ||
#Benchmarks should be performed on at least two operating systems, each on different hardware architecture/model (e.g. with differences in hardware performance of disks, RAM, amount of memory) | #Benchmarks should be performed on at least two operating systems, each on different hardware architecture/model (e.g. with differences in hardware performance of disks, RAM, amount of memory) | ||
#The operating systems have to be both Linux-based (because Kexi for other OSes may not be tested enough at the moment) | #The operating systems have to be both Linux-based (because Kexi for other OSes may not be tested enough at the moment) | ||
#[http://en.wikipedia.org/wiki/Time_%28Unix%29#User_Time_vs_System_Time User and system time] should be measured whenever possible | #[http://en.wikipedia.org/wiki/Time_%28Unix%29#User_Time_vs_System_Time User and system time] should be measured whenever possible | ||
#It may be possible that alteration to product's configuration or even source code for the needs of this benchmark. Therefore, it is preferred to have Kexi built from source code so it can be easily adapted; this is not required to LibreOffice Base. | #It may be possible that alteration to product's configuration or even source code for the needs of this benchmark. Therefore, it is preferred to have Kexi built from source code so it can be easily adapted; this is not required to LibreOffice Base. | ||
| Line 42: | Line 41: | ||
#Benchmark report should include: | #Benchmark report should include: | ||
##Reference to the benchmark design documents and test projects/data | ##Reference to the benchmark design documents and test projects/data | ||
##Detailed hardware information | ##Detailed hardware information (types of components) | ||
##Operating system information | ##Operating system information (version, distribution, versions of relevant dependent packages, e.g. Qt or SQLite) | ||
##Detailed information about both product: their relevant configuration, version, source code patches (if applied) | ##Detailed information about both product: their relevant configuration, version, source code patches (if applied) | ||
##Result for each test for each product; median of time (or memory or other resource) for N runs for should be computed, N >= 10 | |||
##Remarks about accuracy of the test and limitations | |||
#Future expansion of the benchmark should be documented, e.g. new features to benchmark | |||
Revision as of 14:18, 26 March 2014
Status: UNASSIGNED, Difficulty: MEDIUM
Proposed and mentored by Jstaniek (talk) 14:13, 26 March 2014 (UTC)
The Goal
Document pros and cons of Kexi vs LibreOffice Base using performance/stability/resource consumption benchmarks.
Rationale
For many reasons opinions about software is often subjective. We would like to identify following in Kexi:
- strengths,
- potential for further development,
- most important areas to improve.
The Task
In order to do a Kexi-LibreOffice comparison:
- Design performance and stability benchmarks
- Perform the benchmarks
Below we call Kexi and LibreOffice Base as a product.
Requirements
- Include single-user usage only
- One version of each product should be selected, and it should be as new as possible
- Intersection of features and parameters present in both products should be covered by the benchmark
- One database project should be designed for one product
- In each database, a number of objects of type (table, query) should be designed
- Tables should be filled with appropriate amount data
- Database projects built for the two products should be as equivalent as possible
- Design documents and project files should be made available under the GNU FDL license
- Design documents should be put inline on this wiki page, in a subsection
- Tests should be reproducible given the same hardware, so relevant scripts or recipes should be provided
- Benchmarks should be performed on at least two operating systems, each on different hardware architecture/model (e.g. with differences in hardware performance of disks, RAM, amount of memory)
- The operating systems have to be both Linux-based (because Kexi for other OSes may not be tested enough at the moment)
- User and system time should be measured whenever possible
- It may be possible that alteration to product's configuration or even source code for the needs of this benchmark. Therefore, it is preferred to have Kexi built from source code so it can be easily adapted; this is not required to LibreOffice Base.
- Release or ReleaseWithDebugInfo (not Debug or DebugFull) build or equivalent should be used to build product(s)
- Memory-related benchmarks should be designed and performed too
- KDE's System Activity "Detailed Memory Information" too should be used for memory reports. Read this to see how memory usage comparison may look.
- Benchmark report should include:
- Reference to the benchmark design documents and test projects/data
- Detailed hardware information (types of components)
- Operating system information (version, distribution, versions of relevant dependent packages, e.g. Qt or SQLite)
- Detailed information about both product: their relevant configuration, version, source code patches (if applied)
- Result for each test for each product; median of time (or memory or other resource) for N runs for should be computed, N >= 10
- Remarks about accuracy of the test and limitations
- Future expansion of the benchmark should be documented, e.g. new features to benchmark