GSoC/2018/StatusReports/FerenczKovács: Difference between revisions

Latest revision as of 14:01, 9 August 2018

Plotting of live MQTT data

Currently, LabPlot has some support for the plotting of live data, which can be read from file/named pipe, TCP socket, UDP socket, local socket and serial port. The goal of this project is to provide support for the plotting of data received via MQTT protocol from a certain broker. LabPlot currently doesn’t provide any support for MQTT, even though it would be quite important for LabPlot to provide this kind of support, since it is a scientific data plotting software and MQTT protocol is widely used to transmit certain sensor data. This project would increase LabPlot’s usability regarding analyses of scientific data, monitoring data collected by sensors, that is transmitted by MQTT protocol, or even make LabPlot usable in the field of Internet of Things or Smart Home Appliances.

Mentor: Fábián Kristóf

Project Goals

Support data received via MQTT protocol
Provide functionality to configure the MQTT connection
Adapt already existing features to MQTT protocol and also implement new ones

Work Report

Community Bonding Period

Investigating/analysing deeply already existing features for live data sources provided by LiveDataSource, ImportFileWidget, LiveDataDock, AsciiFilter

Week 1 - Week 4 (May 14 - June 10)

The very first steps were digging deeply into MQTT protocol and Qt's MQTT API to save up time later. Then designing the Import File Widget followed, in order to be able to provide the necessary functionalities.

The next task was to update the Import File Widget. Using the widget the user can set:

the MQTT client's host and port to the host and port of the broker, which the user wants to connect to,
the username and password (if authentication is required by the broker)
the client ID.

After successfully connecting to the broker, every active topic is added to a combo box. The user can choose from these topics the ones the MQTT client will subscribe to. Subscribing and unsubscribing to a topic is also implemented.

The next logical step is to offer a preview of the data, that is received via MQTT messages, so the user is able to see what kind of data is transmitted in every topic that the user subscribed to.

The next step was adapting already existing options and functionality of live data sources to MQTT protocol. Here's a list of the options I adapted to MQTT:

Reading type
- From end
- Till the end
Update type
- Periodically
- On new data
Keep last N values
Update now
Pause/Continue
Update interval
Sample rate

For more details about what has been accomplished and videos about the implemented features in use check out my blog post

Week 5 - Week 8 (June 11 - July 8)

The biggest problem was that these topics are completely INDIVIDUAL, so they may send totally different amount of data (this amount of data may differ from message to message as well, given one topic) and this hasn't made possible putting the data of these messages in the same container (spreadsheet). The idea used for the solution came from my mentors. 3 new classes were implemented, in order to represent better the tree like structure of the topics:

MQTTClient which contains the information about the QMqttClient which connects to the broker and manages it, manages the connection, contains the reading and update options, presented in the previous post, is responsible for subscribing and unsubscribing
MQTTSubscription which represents the subscriptions made by the client. This is the core of the tree like structure, because if the subscribed topic contains wildcards, then every topic from which a message was received by the subscription will be added as its child
MQTTTopic which is responsible for saving the data in columns according to the reading and update options of the MQTTClient. This is basically a spreadsheet containing at least 2 columns:
- value: the data received from the message
- timestamp: the time when the message was received in order to be able to plot the values of a single topic, without using any other topic's values
- one optional index column

This not has only solved the problem of sets of messages with not equal amount of data, but also we no longer have to enforce synchrony for the topics, since they are handled individually.

The next change is that, now we use a QTreeWidget to list the available topics, in order to represent them in their natural tree structure. Searching for "roots" is also possible for better navigation, the QTreeWidget scrolls to the given root item. Another upgrade of the ui is that its content is now scrollable, this change was needed because the amount of ui items needed for the MQTT setup has become quite large.

The next feature is setting the will message of the client. The user can choose the will topic from the topics of the MQTTSubscriptions. The user can set the QoS, retain flag and payload as well. For the payload we implemented 3 variants:

Own message: the user can set any message using a QLineEdit
Last message received (from the will topic)
Statistics: several options which can be selected by list of checkboxes (for instance Minimum, Maximum, Median, Variance, etc.)

The user can also set when and how to set/update the will message. There are 2 options: by click and after a set time interval.

Another implemented feature is subscribing and unsubscribing to topics during runtime in LiveDataDock. If we subscribe to a topic that "contains" one that we are already subscribed to, this can only happen if the new contains # or + wildcard(s) and it includes the "old" topic. If this happens we don't throw away the data which was received until that moment by the to be deleted MQTTSubscription, but we "reparent" the MQTTTopics of the "old" MQTTSubscribtion to the new subscription.

The last big feature which was implemented is saving and loading the objects of the newly implemented classes.

The preview of data in ImportFileWidget was also improved, now it works with messages containing different amount of data as well. There was added another option, the user can choose from interpreting the retained messages or not.

For more details about what has been accomplished and videos about the implemented features in use check out my blog post

Week 9 - Week 13 (July 9 - Aug 14)

The main focus was on improving the quality of the code, cleaning, optimizing and properly documenting it. And also making it more comestible for other developers.

The first step was searching for bugs and then fixing them. In order to do this properly, I implemented a unit test for the main MQTT related features. This proved to be useful since it helped discover several hidden bugs and errors which were all corrected. The main features, that tests were developed for, are: checking if a topic contains another one, checking if two topics are "common topics" (meaning they only differ at only one level, and are the same size), managing messages, subscribing&unsubscribing.

The main improvements were related to algorithms needed for subscribing and unsubscribing. This process is now more optimal, and I really hope that is bug-free, since a lot of time was spent on testing every possible scenario for this feature. Not only the algorithms were improved regarding this feature, but the user interface as well. Now we use two tree widgets instead of a tree widget and a list widget (used for listing subscriptions). A new feature is that not only the subscription is listed, but if the subscription contains wildcards, then every topic contained by the subscription and present in the topic tree will be added as children of the subscription, in order to make the user's life easier. Searching for root subscriptions in the tree widget is possible just like in the case of the topic tree widget.

Another improvement is dealing properly with multiple MQTTClient objects. Now it works fine, and the user can easily control each of the MQTTClients using the LiveDataDock. Another bug/absurdity was fixed. Namely, the user could add more MQTTClients with the same hostname, which is quite illogical (since the user can control every topic of a broker with a single MQTTClient). Another minor visual improvement is that an icon for MQTTClient and MQTTSubscripiton was added.

For more details about what has been accomplished, videos about the implemented features in use and a final demo video check out my blog post.

My Work

Contact

Email: [email protected]

@@ Line 18: / Line 18: @@
 Investigating/analysing deeply already existing features for live data sources provided by [https://github.com/KDE/labplot/blob/master/src/backend/datasources/LiveDataSource.cpp LiveDataSource], [https://github.com/KDE/labplot/blob/master/src/kdefrontend/datasources/ImportFileWidget.cpp ImportFileWidget], [https://github.com/KDE/labplot/blob/master/src/kdefrontend/dockwidgets/LiveDataDock.cpp LiveDataDock], [https://github.com/KDE/labplot/blob/master/src/backend/datasources/filters/AsciiFilter.cpp AsciiFilter]
+=== Week 1 - Week 4 (May 14 - June 10)===
+The very first steps were digging deeply into [http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.pdf MQTT protocol] and [http://doc.qt.io/QtMQTT/qtmqtt-module.html Qt's MQTT API]  to save up time later. Then designing the Import File Widget followed, in order to be able to provide the necessary functionalities.
-== Branches where I'm working ==
+The next task was to update the Import File Widget. Using the widget the user can set:
+* the MQTT client's host and port to the host and port of the broker, which the user wants to connect to,
+* the username and password (if authentication is required by the broker)
+* the client ID.
-*[https://github.com/KDE/labplot/tree/gsoc2018_mqtt LabPlot MQTT]
+After successfully connecting to the broker, every active topic is added to a combo box. The user can choose from these topics the ones the MQTT client will subscribe to. Subscribing and unsubscribing to a topic is also implemented.
-== Important Links ==
+The next logical step is to offer a preview of the data, that is received via MQTT messages, so the user is able to see what kind of data is transmitted in every topic that the user subscribed to.
+The next step was adapting already existing options and functionality of live data sources to MQTT protocol. Here's a list of the options I adapted to MQTT:
+* Reading type
+** From end
+** Till the end
+* Update type
+** Periodically
+** On new data
+*Keep last N values
+* Update now
+* Pause/Continue
+* Update interval
+* Sample rate
+For more details about what has been accomplished and videos about the implemented features in use check out [https://kfereneczgsoc2018.blogspot.com/2018/06/labplot-getting-support-for-mqtt.html my blog post]
+=== Week 5 - Week 8 (June 11 - July 8)===
+The biggest problem was that these topics are completely INDIVIDUAL, so they may send totally different amount of data (this amount of data may differ from message to message as well, given one topic) and this hasn't made possible putting the data of these messages in the same container (spreadsheet). The idea used for the solution came from my mentors. 3 new classes were implemented, in order to represent better the tree like structure of the topics:
+*MQTTClient which contains the information about the QMqttClient which connects to the broker and manages it, manages the connection, contains the reading and update options, presented in the previous post, is responsible for subscribing and unsubscribing
+*MQTTSubscription which represents the subscriptions made by the client. This is the core of the tree like structure, because if the subscribed topic contains wildcards, then every topic from which a message was received by the subscription will be added as its child
+*MQTTTopic which is responsible for saving the data in columns according to the reading and update options of the MQTTClient. This is basically a spreadsheet containing at least 2 columns:
+**value: the data received from the message
+**timestamp: the time when the message was received in order to be able to plot the values of a single topic, without using any other topic's values
+**one optional index column
+This not has only solved the problem of sets of messages with not equal amount of data, but also we no longer have to enforce synchrony for the topics, since they are handled individually.
+The next change is that, now we use a QTreeWidget to list the available topics, in order to represent them in their natural tree structure. Searching for "roots" is also possible for better navigation, the QTreeWidget scrolls to the given root item. Another upgrade of the ui is that its content is now scrollable, this change was needed because the amount of ui items needed for the MQTT setup has become quite large.
+The next feature is setting the will message of the client.  The user can choose the will topic from the topics of the MQTTSubscriptions. The user can set the QoS, retain flag and payload as well. For the payload we implemented 3 variants:
+*Own message: the user can set any message using a QLineEdit
+*Last message received (from the will topic)
+*Statistics: several options which can be selected by list of checkboxes (for instance Minimum, Maximum, Median, Variance, etc.)
+The user can also set when and how to set/update the will message. There are 2 options: by click and after a set time interval.
+Another implemented feature is subscribing and unsubscribing to topics during runtime in LiveDataDock. If we subscribe to a topic that "contains" one that we are already subscribed to, this can only happen if the new contains # or + wildcard(s) and it includes the "old" topic. If this happens we don't throw away the data which was received until that moment by the to be deleted MQTTSubscription, but we "reparent" the MQTTTopics of the "old" MQTTSubscribtion to the new subscription.
+The last big feature which was implemented is saving and loading the objects of the newly implemented classes.
+The preview of data in ImportFileWidget was also improved, now it works with messages containing different amount of data as well.
+There was added another option, the user can choose from interpreting the retained messages or not.
+For more details about what has been accomplished and videos about the implemented features in use check out [http://kfereneczgsoc2018.blogspot.com/2018/07/support-for-mqtt-has-evolved-in-labplot.html my blog post]
+=== Week 9 - Week 13 (July 9 - Aug 14)===
+The main focus was on improving the quality of the code, cleaning, optimizing and properly documenting it. And also making it more comestible for other developers.
+The first step was searching for bugs and then fixing them. In order to do this properly, I implemented a unit test for the main MQTT related features. This proved to be useful since it helped discover several hidden bugs and errors which were all corrected. The main features, that tests were developed for, are: checking if a topic contains another one, checking if two topics are "common topics" (meaning they only differ at only one level, and are the same size), managing messages, subscribing&unsubscribing.
+The main improvements were related to algorithms needed for subscribing and unsubscribing. This process is now more optimal, and I really hope that is bug-free, since a lot of time was spent on testing every possible scenario for this feature. Not only the algorithms were improved regarding this feature, but the user interface as well. Now we use two tree widgets instead of a tree widget and a list widget (used for listing subscriptions). A new feature is that not only the subscription is listed, but if the subscription contains wildcards, then every topic contained by the subscription and present in the topic tree will be added as children of the subscription, in order to make the user's life easier. Searching for root subscriptions in the tree widget is possible just like in the case of the topic tree widget.
+Another improvement is dealing properly with multiple MQTTClient objects. Now it works fine, and the user can easily control each of the MQTTClients using the LiveDataDock. Another bug/absurdity was fixed. Namely, the user could add more MQTTClients with the same hostname, which is quite illogical (since the user can control every topic of a broker with a single MQTTClient). Another minor visual improvement is that an icon for MQTTClient and MQTTSubscripiton was added.
+For more details about what has been accomplished, videos about the implemented features in use and a final demo video check out [http://kfereneczgsoc2018.blogspot.com/2018/08/labplots-mqtt-in-finish-line.html my blog post].
+== My Work ==
+*[https://github.com/KDE/labplot/tree/gsoc2018_mqtt Project's branch (we've merged it already in master)]
+*[https://github.com/KDE/labplot/commits?author=kovacsferencz98 My commits on LabPlot's master branch]
+== Other Important Links ==
 === Proposal Link ===