Neon uses a Jenkins continuous integration system to build its packages

The Setup

drax.kde.org is the master server, owned by Blue Systems and administrated by KDE Sysadmins. It runs a Jenkins instance which is a Continuous Integration website at build.neon.kde.org that has many jobs to build the packages and run other functions, either on demand or at pre-scheduled intervals.

The code behind build.neon is from pangea-tooling which also runs the code for DCI Debian CI, KCI Kubuntu CI, ACI Appstream CI, and MCI Mobile neon Plasma CI.

The Jenkins jobs farm off the hard build work to 4 DigitalOcean slave servers. It runs most jobs inside a Docker container to give a fresh build environment.

After a checkout of pangea-tooling add the submodule for the CI config with git submodules update. This adds https://github.com/blue-systems/pangea-conf-projects.git which contains the files that list the jobs to be made.

To use the scripts to access Jenkins you will need to set ~/.config/pangea-jenkins.json using access key available inside Jenkins to administrators User (top right menu) -> Configure -> API Key

{
    "server_ip":"46.101.206.233",
    "server_port":"8080",
    "username":"jriddell",
    "password":"1234abcd1234abcd1234abcd"
}

The setup of the machines is maintained in pangea-kitchen which uses Chef to set up the servers with software all configured.

For more information see pangea-tooling/Getting-Started

The Packaging

Our packaging is kept at packaging.neon.kde.org Git archives, see Neon/Git.

The packaging is for .deb packages and the Git repos contain a single debian/ directory which defines how the .deb is made. We try to keep the packaging in sync with Debian pkg-kde team's Git repositories and keep the diff as small as possible with them.

Neon/unstable is for Developer Edition Unstable Branches, its packages are combined with master branches from the KDE project.

Neon/stable is for Developer Edition Stable Branches, its packages are combined with stable branches from the KDE project which are defined in overrides/base.yaml. Stable branches also includes branches released as Beta (so the name is not quite logical).

When a project makes a new (non-bugfix) release you should merge Neon/unstable into Neon/stable and update the stable branch in overrides.

Neon/release is for User Edition, the code gets built with release tars.

Neon/mobile is used by mobile CI, and not available in all repos. This branch have patches applied which are required only for Plasma Mobile.

When moving files between packages in the same source package you can use the variables (<< ${source:Version}~ciBuild) for your Breaks/Replaces where ~ciBuild gets replaced on merge into Neon/release.

See [New Repositories] for new packages.

The [build overrides] files is used to define jobs which need a paticular branch or tar to build from.

We don't use debian/changelog files, they just add merge conflicts and we already log changes in Git changelog.

The repos are kept in sub-directories which are the same as Debian pkg-kde team uses. Ones we add are neon-packaging/ for stuff we package but don't expect Debian to use, neon/ for distro specific packages such as neon-settings and forks/ for repos packaged elsewhere we want to base on.

The Jobs

The Jenkins jobs are created by running the pangea-tooling script jenkins_jobs_update_nci.rb. This creates some manual Jobs specified in the script such as the ISO jobs but mostly uses factories to create batches of jobs based on archives. Use NO_UPDATE=1 to speed up running of it by not updating git checkouts.

The YAML files in pangea-conf-projects define what jobs get created.

For each package there is a parent MultiJob which runs some sub jobs.

parent job this is set to checkout the relevant archive from KDE Git as source/ (for Developer Editions), then check out the relevant archive from KDE neon Git as packaging/. It then runs a number of child jobs...

src will create the source package. For User Edition this means running uscan to use the debian/watch file to download the relevant tar, for Dev Editions it uses the source the parent job checked out. It then builds the source package.

bin job will extract the source, install the build dependencies and compile the package. It finishes by checking the output from lintian and fails on any errors, you can override errors with lintian-overrides files in the normal .deb packaging method (see dh_lintian). It also checks for any list-missing files and fails if there are any, override by adding a debian/not-installed files. It also fails if cmake reports build-dependencies it needs, override with debian/meta/cmake-ignore.

adt job runs Debian's test framework autopkgtest. See Ubuntu guide for some details. It runs adt-run on the binaries which installs them and runs the relevant test suite as defined in debian/tests/. It doesn't fail if tests fail.

pub job will upload to aptly, see The Archive below.

lintqml job will scan for QML dependencies which have not been satisfied by the package dependencies, it will print a JSON output of any missing QML modules it requests. The packager should add these to the packaging manually and rebuild. Any false positives can be overridden, see Kubuntu/CI/QMLIgnore.

snap job will package it up as a Snappy Snap package. This is experimental, you can see the output at distribute.kde.org.

Other Jobs

watcher jobs are made for packages in User Edition. They use debian/watch files to check for new releases and if one is found add a new changelog entry, merges from Neon/stable, then runs the release build job. See man uscan for info on watch files.

It will fail it is finds an "unstable" line in the watch file as we don't include these in User Edition.

It will mangle the watch file to use the neon-sftp-bridge we have running on Drax, this uses ssh to expose the contents of download.kde.org even when they are hidden from the web server so we can get previews of tars before they are released. When building unreleased packages make sure not to snapshot them into User until they get released. Consider disabling the Snapshot job to avoid mistakes.

mgmt jobs run various management tasks. mgmt_tooling is run whenever there is a commit made to pangea-tooling, it blocks new jobs being started while it's running which can be a blocker so you can cancel it if you check the commits to pangea-tooling are not relevant. It fails if ruby testing fails. It runs mgmt_docker which updates the docker images used for builds. mgmt_pause_integration can be run manually and just blocks jobs from starting, remember to kill is when you're done. mgmt_jenkins_prune_archives and mgmt_jenkins_prune_logs clear some space on the server and run periodically. mgmt_progenitor is run daily and runs mgmt_build_xenial_release and stable/unstable. It deploys code onto slaves and then deploys docker images.

iso jobs builds the installable ISOs. See Neon/InstallableImages. The Dev Editions are run daily and the User Edition is run weekly.

The Archive

archive.neon.kde.org is our .deb package archive. For your sources.list you need one of the following lines.

deb http://archive.neon.kde.org/dev/unstable xenial main
deb http://archive.neon.kde.org/dev/stable xenial main
deb http://archive.neon.kde.org/user xenial main

It is an aptly instance and may be running the Blue Systems Aptly fork.

Admins can access it using the repo console from pangea-tooling: ./ci-tooling/ci/repo_console.rb --gateway ssh://darwini:9090

Repo.list
repo = Repo.get("unstable_xenial")
repo.packages()

This makes available the Aptly-Api code using the Ruby GEM written by Harald and Rohan https://github.com/KDEJewellers/aptly-api/

pangea-tooling/ci-tooling/nci/repo_cleanup.rb can be run to delete packages other than the latest one and save some disk space on darwini

User Repo

To allow for extra QA the packages built for User Edition are uploaded to the secret release repo

deb http://archive.neon.kde.org/release xenial main

You can test this manually and when happy run mgmt_snapshot to copy the packages to user repo. This will first run mgmt_daily_promotion_xenial_release (slow takes ~30 mins) which installs existing packages and attempts to upgrade them to new packages, if there are any problems it'll stop the snapshot. It also runs mgmt_appstream-generator which creates the Appstream data files used by the archive.