View source for Calligra/Flow/Usability Survey

In 2006 a Usability Survey was done. This page contains the restored results.

=Introduction & Background=

The purpose of this study was to get insights into Kivio's major user base, their tasks and needs. Learning more about the usage of diagramming tools, the context of diagram creation and application, as well as the required preciseness were important questions in this scope.

Based on this information, the survey should allow us to illustrate typical usage patterns, and to learn about feature relevance and priorities. It should help us to determine if the features planned in the scope of Kivio's redesign are important to users.

Also, the survey aimed at determining the users' opinions about Kivio and evaluate the differences to other diagramming applications. Reasons for prominence and gaps should be identified and considered in the future work on Kivio.

Finally, the survey should give us widespread information about users of diagramming applications which may be used for persona creation. This includes where they live, their age and gender, education and how experienced they are with computers and with creating diagrams.

=Survey Methods=

The survey was publically announced on planetkde, kdedevelopers and several Linux- and Windows forums. Furthermore, private invitations were sent to known Kivio users. The survey addressed not only actual Kivio users, but anybody who occasionally creates diagrams, no matter if using a software tool or paper and pen.

It was expected that the choice of these publication channels would influence the type of participants towards a rather technical user base. As it is assumed that the majority of Kivio's actual user base is technically oriented this fact was appreciated.

After the public announcement on June 16th the survey was online for two weeks, until June 30th.

For the survey, the UCCASS survey tool was utilised. All in all, approximately 35 questions were asked, partly limited choice, partly Likert-scales, partly free-form comments. They addressed information about user demographics, general computer experience, diagramming purpose, habits and experience, Kivio experience and evaluation, experience and evaluation regarding other diagramming tools, and finally an importance rating of several planned features for the Kivio rewrite.

The complete list of questions can be seen in the survey preparation section.

=Survey Results=

==Result Summary==

===Demographics===

* In the survey, 257 people from 30 nations participated. 
* The majority was between 18 and 34 years old. 
* From the 257 participants, 5 were female. 
* Almost 50% of the participants had finished a university degree, 35% were still students or dropped out. 6% had a doctoral degree. 

===Diagramming Basics===

* As expected the participants were rather technically-oriented "power users" who create diagrams once a month or even more often. They rated themselves to be very experienced with computers, and spent between 26 and >50 hours at a computer per day. A great part of the participants reported to develop themselves. 
* With regard to diagrams, they mostly created flowcharts, but also UML, networking diagrams, database diagrams, electrical circuits or visualised their thoughts. Especially UML, electrical circuits and (relational) database diagrams were later named very often in the free-form text comments. 
* As they reported, diagrams mostly consist of 6-15 elements, 2-3 levels of hierarchy and use 4-7 different shapes. But also larger diagrams with 16-30 or even 31-60 elements were named. More than 6 levels of hierarchy or 13 different shapes were extremely seldom. 
* About half of the diagrams were created for formally presenting information to others, one third for personal usage. 20% were meant as work input to others. 
* More than half of them was created for university or the participants' job (mostly IT-related), about 20% they created for themselves. 

===Kivio Users===

* One third of the participants was currently a Kivio user. 
* More than 80% of them used version 1.5 and have not been using Kivio longer than a year. 
* Only about 17% felt experienced or very experienced, more than half inexperienced or very inexperienced. 

inexperienced 	- 	13 	

14.94%
	- 	36 	

41.38%
	- 	22 	

25.29%
	- 	14 	

16.09%
experienced 	- 	2 	

2.30%
Total Answers 	- 	87

* Regarding satisfaction, 35% were indifferent, the rest was equally portioned to the two poles, but only few people chose a extreme rating. 

unsatisfied 	- 	5 	

5.75%
	- 	27 	

31.03%
	- 	32 	

36.78%
	- 	22 	

25.29%
satisfied 	- 	1 	

1.15%
Total Answers 	- 	87

===Positive Comments:===

* Kivio users especially like about Kivio that it is very easy to use and "basic", and provides a basic set of shapes. 

* They like the integration with KDE and KOffice in particular. This is interesting, as former Kivio user complained about the insufficient integration with other KOffice components that allows for a direct interaction between the applications. 

* For many Kivio users, it was important that Kivio is Open Source Software. 

* They liked certain features like gridlines and snap-to-grid, customizability, the window layout (one main window), and they appreciated Kivio's speed. 

===Negative Comments:===

* Kivio users were especially unsatisfied with the quality and choice of shapes, as well the lacking possibility to reuse stencils or add stencil sets. When analysing the reasons why other people stopped using Kivio, the shapes were also a frequently heard reason. 

* Connectors showed to be another problematic area: While the preciseness of the currently available connectors was perceived as insufficiently, the participants missed several features like auto-routing or bent connectors. 

* Several participants had a hard time to find different export options or did not find them at all. Others complained about the quality of the exported diagrams. Especially SVG export was missed. 

* Rotating and flipping of elements is not possible, which was also a reason why other participants had stopped using Kivio. 

* Apart from that, there were some interaction issues, two participants missed gradients and transparency, and some did not like the rendering of diagrams. 

===Feature Requests:===

* Accordingly, feature requests covered a larger choice of shapes, mostly Database and UML. 

* As well, they asked for custom creation of stencils and stencil sets. This might be combined with a free-hand drawing tool. Others asked for better import options of stencils via SVG and image insertion. 

* The choice of connectors should be increased, "smart" connectors should be added and there should be an option to create custom connectors. 

* Export facilities should be improved, especially SVG was named over and over again. 

* Flip and rotate, as well as better auto-align faetures were asked for, as well as in-line editing. 

* Regarding the page layout, some requested headers and footers, while others did not like Kivio's page-orientation and preferred to have a free drawing board. They preferred to do layouting when the diagram is finished. The page-orientation was also a frequent reason why other participants did not use Kivio any longer (see below). 

* Two participants wished to have an option to link objects inside the diagram, and to other files or locations. 

===Diagram Shape Requests:===

* Database, Advanced UML, Circuits, Network Diagrams, Mindmaps and some special diagram shapes were asked for. 

* Especially, the participants wanted "modern" and clean shapes that look equally good in the thumbnail preview and on the panel. 

===Other Applications:===

* Besides Kivio, these participants also made use of Pen and Paper (28%), Dia, Visio, Umbrello, Power Point or Open Office Impress (each ~12-14%). 

* When comparing Kivio to Dia, Kivio users mostly commented better on Kivio: They preferred its usability and its integration with KDE. Some features were missed, however. 

* Compared to Visio, Kivio was criticised for its stencil sets and rendering. 

===Users of Other Applications===

====Types of Applications====

Of the 170 participants who made use of other applications, most claimed to make use of Dia, Visio, OpenOffice.org, Umbrello and Pen and Paper.
Dia 	- 	47 	

14.16%
Visio 	- 	44 	

13.25%
SmartDraw 	- 	3 	

0.90%
Omnigraffle 	- 	3 	

0.90%
Umbrello 	- 	39 	

11.75%
Power Point or Open Office Impress 	- 	40 	

12.05%
Pen and Paper 	- 	92 	

27.71%
Other 	- 	64 	

19.28%
Total Answers 	- 	332

All in all, the users of other diagramming applications were slightly more satisfied with that application than the Kivio users: On a five-point scale, the majority was between indifferent to satisfied, as compared to the rather equalised distribution of Kivio's satisfaction measure (see above).
unsatisfied 	- 	13 	

7.65%
	- 	27 	

15.88%
	- 	55 	

32.35%
	- 	63 	

37.06%
satisfied 	- 	12 	

7.06%
Total Answers 	- 	170

====Former Kivio Experience====

60% of the users of other applications had tried Kivio before. They had stopped using it by various reasons.

* The selection of shapes was the number one reason to fall back to another application. Missing UML or database stencils were one aspect, the rendering quality the other main one. 

* Again, the options to connect stencils were rated as insufficient. Auto-routing, curved connectors and missing preciseness of alignment were the major problems reported. 

* For these users, Windows compatibility was a very important factor. They often reported that they need to exchange flow charts at work or university. The missing compatibility was a major reason not to use Kivio. 

* Again, export facilities, as well as SVG import/export were listed. 

* Importantly, the number of complaints about fixed paper size instead of a flexible work space were very high among the former Kivio users, while only few of the current Kivio users reported problems with that. It seems that for users who require a flexible space management, this limitation is a reason to move away from Kivio. 

* The final rendering was not evaluated sufficiently by some participants. 

* Templates that help users to get started in an easy way were missing (or the ability to create templates oneself). 

* Some other issues comprised rotation, moving of objects, printing or text layout. 

===Importance of Different Features===

On different rating scales, the participants were asked to rate the importance of various features for them.

As important to very important, the majority of participants rated the opportunity to edit points along connector lines. Almost equally important was an option to automatically arrange and align shapes on the page or relative to each other.

More equally distributed among the levels of importance, but with a clear tendency towards a high relevancy, were the following features: Creating your own custom stencils or shapes and group them into sets. Linking shapes to other objects. Using elements other than plain text in the labels for shapes.

A tendency towards a lower importance showed the features to assign gradients to stencils, as well as transparency.

==Detailed Results and Discussion==

The detailed results can be seen on the Appendix A page.

=Conclusion and Follow-up Work=

As expected, the majority of participants was technically oriented. Mostly, they reported to create diagrams for university or their IT-related job. Accordingly, the produced diagrams ranged from flowcharts, electrical circuits, UML, networking or database diagrams.

The most valuable insights with regard to user requirements were extracted from the free-form comments. Here, the participants were asked to list the advantages and disadvantages of Kivio or their other preferred diagramming application. All in all, the comments were very detailed and rich of information.

In a great part of these free-form comments, Kivio was valued for its KDE and KOffice-integration, and its ease of learning and using. However, this ease of learning was criticised by others who experienced a lack of certain features.

In this scope, a number of features was named repeatedly:

   1. The quality and choice of shapes was too limited, especially for UML, several Database diagrams and electrical circuits.
   2. The connectors showed to be problematic, especially with regard to alignment preciseness, curving, auto-routing and special shapes.
   3. Transforming shapes, especially rotating and flipping was missed.
   4. More flexible export options, especially SVG and EPS were asked for. Some groups asked for a closer integration with other KOffice tools.
   5. The page-oriented work space of Kivio showed to be problematic for several groups of users. Others missed an option to format a page with headers and footers.
   6. Windows compatibility was one of the major reasons, why certain user groups did not make use of Kivio any longer. 

Another aspect that was named over an over again was the rendering quality of Kivio. Certain user groups were not satisfied with the look of the produced diagrams - regardless of the design of the shapes.

In accordance with the free-form comments, the importance rating of planned features showed that users set a higher value on functionality that affects the basic creation of the tool: Editing points along connector lines and automatically arranging and aligning shapes were rated as the most important features, followed by an option to create your own custom stencils, linking shapes to other objects, or using elements other than plain text in the labels for shapes. Of a lower importance were features to polish a diagram such as assigning gradients to stencils and transparency.

Based on these results, we suggest the following next steps:

   1. Feature priority list: Create a feature priority list. (El 17:50, 10 August 2006 (CEST): ready for review)
   2. Validate Use Cases: Validate the list of use cases along the survey results and feature priority list. (El 17:50, 10 August 2006 (CEST): ready for review)
   3. Sketch Personas: Identify and sketch the three major user groups of Kivio. (El 00:46, 21 September 2006 (CEST): ready for review)
   4. Reconsider the overall interaction paradigm: Devise a design to integrate stencil and connector management, to obtain a flexible more flexible work space, etc.