Necessitas/Java/Redesign

Overview drawing

Base technique

At QCS we found out that it is very impractical to have that many Java classes in sourcecode form in each application. Simply because every class becomes part of the public interface. This will cause a huge maintenance burden. I'd simply say it would not work. We need to move large parts of the code into places that are independent from the application. This however strongly required the need to add classes from a foreign APK file to the application's classpath. We were not sure whether Android would allow this but it indeed does. As a proof of concept we wrote a demo[0] application which can load any (!) APK or DEX file and create an instance of any class and call its toString() method. The POC worked in the way we wanted on a rooted but also on a normal locked device.

With this technique at hand we can do the following:

• Move all code that is not really necessary for app startup into separate APK files
• Define an interface between App and Loader which can handle updates (that allows for incompatible changes in the App code over time)

App

This is the source code that is part of each Android-Qt application. The user *may* change little pieces of the code as long as it does not violate the interface to the Loader or Ministro. What the user can do is calling any methods of the public interfaces to the loader for example. We will mark the pieces of code that the user *must* not remove or modify otherwise. This whole thing will be as small as possible! Whenever there is the need to change the App code in an incompatible way we need to introduce a new Loader interface. E.g. we will have a LoaderVersion1 interface for now. When it changes the App code will ask for LoaderVersion2. The Loader's responsibilities will be explained below.

Loader

The Loader connects methods from the App with the Android Activity class. It will implement all available methods in order to gain maximum control over the App code without letting the App decide by itself. The Loader is published by us and can as such be updated over time allowing *us* to fix bugs, introduce new feature to *existing* applications without changing them. There might be applications with different versions of the starter code in it. Each of those version requires a specific interface in the Loader. The most up to date version of the Loader will understand (=implement) and offer all interfaces. That way old and new application start code is always compatible with the Loader. If a Loader cannot satisfy the requested interface of an application then this means that the current version of the Loader is too old and it will update itself. (The Loader will technically be part of Ministro, when Ministro is installed. If the user is a developer and chose 'use local libs' then the loader will also be copied to a well known directory and directly included into the Apps classpath. As such at runtime of an App the Loader APK will be part of the App's process space!)

Ministro

Ministro does what it does right now: It downloads the libs (if necessary), provides the locations of the libraries and tells the app that it can start. Unlike now the interface between App and Ministro will become upgrade compatible. While changes will happen less frequently it is very important that we allow and can deal with incompatible changes to the interface between Ministro and the App.

A Ministro implementation does not share any code with the Loader and can theoretically support any Loader version we want. It is as independent from the Loader as it is from Qt code itself.

Bridge

These are all the classes that interact in some way with the C++ code of the Android-Qt port. Unlike now they will be a private interface. Meaning that we can do *any* change we want in them. For each supported Qt version (4.8, 5.0 etc.) there will be an accompanying bridge. The bridge may even consist of multiple APKs if we want to save space (e.g. if QtMobility is not used, then the classes for it do not need to be available as well).

The Loader needs to know a class in the Bridge APK that it instantiates and calls a few known methods on. The methods will actually do the 'System.loadLibrary' call. The background for this is that by doing so we load the native libraries with the correct classloader (that of the Bridge APK). This in turn will allow the native code to resolve any of the private classes in the Bridge APK removing the need to fumble with classloading issues in native code (which is ugly and unneeded).

Since there is only one Bridge APK per Qt version we can hardcode the name of the known class in the Loader. However this is a purely private interface that we can break and change at any time. We have full control over it.

I actually do not see that much changes to the Bridge classes are needed. The native code might even become simpler because no complicated classloading stuff needs to be done.

Qt

These are the native Qt libraries. No change needed. Except that any code can *assume* the existence of certain bridge classes and can resolve class names at will.

Caveats

Android-Qt changes

From a development perspective this is quite a task. The Java code changes are not so difficult (for me at least) but I have no clue how we can modify the Android-Qt build to also compile Java classes, dexify and make an APK out of them. Because this is what we need. When you compile Qt for Android you'll need a Java compiler, the DEX tool and something that creates an APK for you. IOW this is where I need you help.

This problem is conceptually solved. The build process will be enhanced to support compiling Java classes via QMake.

Sharing code

I plan to use real Java interfaces in order to define the public interface. An alternative approach would be to rely on reflection only. However that will make the interface very opaque and the code will be difficult to read for anyone with only a small knowledge of Java. Having interfaces however means we will need to share bits of sourcecode between the projects (namely the interface file).

The way to do it is to simply copy the interface files (sometimes Java, sometimes AIDL) into the App code. Touching those interfaces and modifying them in the App only is obviously a crime. At a later point we can decide to compile the crucial classes into a Jar file and add it to the classpath of the App. That will have the same runtime consequences but prevents unwanted modifications to the files.

Proof of concept - Java

This repository contains a proof of concept implementation in pure Java. There are three Ministro implementations: One is a first generation Ministro and the other two are a 2nd generation of which only one can satisfy two generations of the Loader. There are also three Applications: One is a 1st generation app, the other two are 2nd generation app with different requirements on the Loader generation.

The first generation app can work together with all Ministro implementations. This is because the newer ones supports the older interface.

To see it all working download and unpack the file, run './compile' from the top folder, then go into the 'app_v1', 'app_v2' and/or 'app_v2_2' folder and run the scripts 'run-against-v1', 'run-against-v2' and/or 'run-against-v2_2'.

All the source files contain extensive documentation at the beginning of the file which explains it purpose in the framework. For a good understanding I suggest the following order of reading: v1, app_v1, v2, app_v2, v2_2, app_v2_2. Many files are unchanged in the higher generation implementations. So don't worry it is not that much text.

Proof of concept - Android

With a better understanding for the problem an Android PoC is being developed. The sourcecode is available from this repository. There is an App called appv1, a Ministro implementation and a Loader.

Implementation thoughts

App and Loader

After implementing an Android PoC it turned out that the code for the App wrapper can be really nice and small. There is some code which asks Ministro for the location of the Loader APK and classname of it. In a 'use local libs' scenario that location is hardcoded and the App does not need to ask. The code doing the interaction is currently an Activity that we call Starter. When the Loader instance is created the Starter activity will transfer to a 2nd Activity which is the App. This Activity class looks very weird because it overrides every onXXX method and all it does in each is to call a variant of the method on an instance called ActivityDelegate. The ActivityDelegate is something that the App created using the Loader.

The ActivityDelegate also receives something when being created:

• the App's main activity (to call any method it wants)
• an object which allows calling the super-implementation of all the onXXX methods of the App's activity
• the resource id for the main View

With these three things available the ActivityDelegate can completely implement all the functionality of an Android application without being a subclass of Activity.

Some onXXX methods require that the super implementation is being called (e.g. onCreate()). With the delegation modell this would be impossible because in Java calling a super implementation from a foreign source is not allowed. The way to get around this is to add new methods which do nothing more than calling the super implementations. However since we also cannot see new API in the App from the Loader we need to wrap them and hide them behind an interface that the Loader defined. Conveniently that is the same that the App uses to delegate all its derived onXXX methods. Yeah, I know. It sounds difficult. Look at the code, in practice it is simple. :)

Nice side effect of the above is that methods that are normally of 'protected' scope have been widened to be of 'public' scope. Without that the Loader would not be able to call them.

Ministro

Implementing Ministro as a PoC showed that hardcoding the Ministro-App interface is needed but at the same time the Loader versions supported by Ministro can be completely free. If an App asks for the Loader version 5 we can try looking this version number up in an online repository. If we succeed and download that loader we can start the app. If not Ministro needs to tell the calling App that it cannot fulfil the request.

In the PoC I did a very simple approach. Instead of using an online repository I compiled the Loader into an APK and threw it into the 'res/raw' directory of Ministro. I then declared file and classnames of the Loader via 'res/strings.xml' and added code that sets up a little repository for the Loader at the start of Ministro. I suggest that for the beginning we should implement our new Ministro like this. Since we're not going to have many Loader implementations.

Ministro-App interface

The way we achieve an upgradable Ministro-App interface is dead simple. The interface exposed by Ministro features a 'checkCompatibility(int ministroLevel)' method. This method will always be there. After a successful call of say 'checkCompatibility(15)' the caller can also assume the existence of a method with the name 'serve(ICallback15)'. The calls are done using Android's IPC system. In order for it to work Ministro and the App need to have the same (or compatible) AIDL files.

For it to work we make strong use of Java's binary compatibility. E.g. if the App has an older interface missing a few function we can perfectly call into a newer implementation which adds new methods.

If we have it the other way round: The App is newer and expects the existence of 'serve(ICallback7000)' then the call to 'checkCompatibility(7000)' will safely tell the App that it cannot call the first method. And that is our only requirement.

Ministro results

Another API decision revolves around the answer from Ministro to the App. Instead of having a dedicated method with lots of arguments in ICallbackXX there will be a method which is called 'finished' and that takes a IResult object. IResult consists of lots of getter methods only (e.g. getLoaderPath()). The interface for that object will be extended over time in a binary compatible way. This makes the introduction of new functionality very easy.