Windows/Build/emerge

emerge is a tool to build the KDE sources and its third-party requirements on MS Windows. It is the easy way to build KDE on MS Windows.

Introduction

Emerge is a tool that can build the different parts of KDE and its dependencies under Windows. We created this tool to automate and simplify the build process under Windows. We try to build all packages that we offer in the KDE installer with emerge. That has some advantages for us:

• it is easy for people to join us:

Before emerge it was quite some work to set a system up for development. There were some quirks, which were documented in some mailing lists, but you had to remember them or you ran into an already solved problem again, etc. Now to get a development machine you need a Windows computer, need to install Python and Subversion and do the emerge checkout. Then execute emerge to build what you want to build. This is easy for developers coming from Windows to KDE, and also for KDE developers coming to Windows.

• it is easy for us to do (nightly/continuous/release/reproducable/...) builds:

With emerge you can build the whole software stack (low-level libs, Qt, kdelibs, things above that) with only one command. You can start that build, and some hours later you can check if it worked, or if something broke. So we can spot problems easier and earlier. We can also start with a "naked" Windows computer without any other installed software and bootstrap kde on it. That ensures, that no hidden dependencies on some pieces of software sneak in, because then the builds on a "naked" computer would break and show the problem.

• it is easier to collaborate:

We can test the same emerge build description for a package on different Windows versions/computers before we do binary releases. People can also add build descriptions for new packages to the Subversion repository.

This emerge tool was inspired by the Gentoo emerge tool.

Set up the environment

Root directory

Create a directory if possible in your harddrive's root e.g. C:\kderoot or D:\kderoot (You will need this PATH later). This directory will contain the whole kde installation later. We will refer to it as %KDEROOT%.

Python interpreter

emerge.bat invokes an emerge.py script written in Python programming language, so you first need to install a Python Interpreter. The python installation directory will be added to the PATH later by %KDEROOT%\etc\kdesettings.bat script.

Subversion client

The latest source code for Windows emerge and the rest of KDE is stored in a repository created and managed using the Subversion version control tool. You need a Subversion client for the first checkout. There are at least two applications:

• a command line client, available at subversion.tigris.org (required by emerge to get the source code from KDE Subversion repository), aimed at developers or power users accustomed with the command line,
• GUI program like TortoiseSVN, optional, useful for displaying differences between various versions of files in the repository in a graphical way.

Note 1: If you experience problems with the checkout of Qt (Subversion doesn't work correctly) please remove any other Subversion binaries out of the path that you do have. The different versions of the Apache portable runtime (APR) are incompatible!

Note 2: Make sure to use a copy of Subversion that was built on Windows so that checked-out files do not use UNIX line endings. If you check out with UNIX line endings, the patch program will fail when attempting to apply a patch whose line endings don't match the system's.

Check out the emerge tool

The source code of the emerge tool and the recipes for creating KDE packages are located at svn://anonsvn.kde.org/home/kde/trunk/kdesupport/emerge, which is an URL based on Subversion-specific svn protocol.

You need to check out the source code from the emerge Subversion directory into a new directory, which in this example we will call %KDEROOT%.

Check out using the 'svn' command

• Option 1: With the svn command line tool, you can accomplish this with the following commands:

  cd %KDEROOT%

  • if you will only use anonymous (read-only) access to the KDE svn repository:

    svn co svn://anonsvn.kde.org/home/kde/trunk/kdesupport/emerge

  • or, if you plan to use write access (commit) to the KDE svn repository
    • via https:

      svn co --username yourusername https://svn.kde.org/home/kde/trunk/kdesupport/emerge

    • via a puTTY tunnel using your existing account & OpenSSH private key:

      svn co svn+putty://svn.kde.org/home/kde/trunk/kdesupport/emerge

This would result with:

Error validating server certificate for 'https://svn.kde.org:443':
 - The certificate is not issued by a trusted authority. Use the
   fingerprint to validate the certificate manually!
Certificate information:
 - Hostname: svn.kde.org
 - Valid: from Wed, 11 May 2005 09:08:21 GMT until Sat, 09 May 2015 09:08:21 GMT
 - Issuer: SVN, KDE e.V., Nuernberg, Bavaria, DE
 - Fingerprint: xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
(R)eject, accept (t)emporarily or accept (p)ermanently? 

enter p here to permanently accept the certificate:

Authentication realm: <https://svn.kde.org:443> KDE SVN account
Password for 'yourusername': ***************
A    emerge\kdeenv.bat
A    emerge\portage
A    emerge\portage\kdesupport
[....]

The password and cache for the certificates is saved in %APPDATA%\Subversion\auth directory.

Check out using the TortoiseSVN

• Option 2: If you use TortoiseSVN:
  1. right-click on your %KDEROOT% folder and select SVN Checkout... command from the context menu,
  2. paste svn://anonsvn.kde.org/home/kde/trunk/kdesupport/emerge URL into the URL of repository text box (replace with https://svn.kde.org/home/kde/trunk/kdesupport/emerge for read-write access)
  3. add \emerge to the folder name in the Checkout directory box and click OK to continue
  4. if you picked the read-write access, you will be asked for accepting the SSL certificate of the SVN server (click "Premanent") and then for username and password. For convenience select "Save authentication" checkbox too (the password and cache for the certificates will be saved in %APPDATA%\Subversion\auth directory).

After the checkout you should have the directory %KDEROOT%\emerge. If you don't, you move your emerge directory to that location.

Configure the emerge tool

1. Create the directory %KDEROOT%\etc.
2. Copy the file %KDEROOT%\emerge\kdesettings-example.bat as %KDEROOT%\etc\kdesettings.bat and change its contents according to your needs. The options are described in the rem lines in the file itself.

The kdesettings.bat script will be called by the main kdeenv.bat script.

Note 1: Be sure that you neither have the msys/bin nor the cygwin/bin in your path. If so you have to definitely remove it.

Note 2 from a user: The applications gimp, inkscape and graphviz are also a problem. To make sure that there's nothing wrong I stripped my path to contain only what I needed to build.

Running emerge

To use emerge you need to start a console window and point that to %KDEROOT%\emerge. For example:

C:
cd \%KDEROOT%\emerge

Then you have to execute

kdeenv.bat

This tells emerge about your environment settings (e.g. paths). It will load your configuration from %KDEROOT%\etc\kdesettings.bat. It should not give any error messages, otherwise emerge will not work as expected. The output should look similar to this one (of course with your paths):

C:\kderoot\emerge>kdeenv.bat
kdesettings.bat executed
KDEROOT     : C:\kderoot
KDECOMPILER : mingw
KDESVNDIR   : C:\kderoot\svn
PYTHONPATH  : C:\python25
DOWNLOADDIR : C:\kderoot\download

C:\kderoot\emerge>

Now you should be able to use emerge. Type

emerge --help

to get some help on usage.

Setting up a compiler

Currently emerge supports both the MinGW and MS Visual C++ (msvc) compilers. We did not add dependencies for the compilers, so you have to make sure to install a compiler by yourself. There are three ways to set up a compiler for emerge. We assumed you have set KDECOMPILER variable properly in the %KDEROOT%\etc\kdesettings.bat.

In the following sections you can find information on how to install or reuse an existing compiler.

Install the MinGW compiler with emerge

Let emerge install the MinGW compiler:

To install the MinGW ("Minimalist GNU for Windows") compiler with emerge, type

emerge mingw

and wait until it is finished.

If you encounter an error like

Assertion failed: hunk, file ../patch-2.5.9-src/patch.c, line 354

try to edit line 51 of file mingw-x.y.z.py (%KDEROOT%\emerge\portage\dev-util\mingw) by adding the --binary after -p1 parameter. The line should then look like this:

cmd = "cd %s && patch -p1 --binary < %s" % \

This is probably because the different line break types (Linux vs Dos) in the files and the bug in patch.exe. Althought this is an ugly hack and should be fixed somewhere else, it works for the current versions of patch (2.5.9) and mingw (3.4.5).

Install MS Visual C++

Read here.

Point to an existing MinGW installation

• Point emerge to an existing MinGW installation:

This option is not recommended for now, because it only adds one more point of failure, and does not gain something in comparison to the option above.
NOTE from a user: be sure that path to \mingw\bin has been set correctly, by default it is pointing to: %KDEROOT%\mingw\bin which does not apply to most installations. If you see an error about cc1plus not being found, either add MinGW's \libexec\gcc\mingw32\3.4.5 to your PATH (in command line set PATH=%PATH%;path\to\directory) variable or copy the contents of this directory to MinGW's bin directory. The prior is preferred.

Under Vista, the mingw directory may need to be moved to c:\ in order to compile properly.

Point to an existing MS Visual C++ installation

You need to point emerge to an existing msvc installation. To do that, execute vcvarsall.bat before running %KDEROOT%\emerge\kdeenv.bat. In recent versions this is run automatically for you from kdeenv.bat if configured properly in kdesettings.bat. Check your kdesettings.bat file to know if you need to run this manually or not.

Notes related to Vista:

• Note that for debug builds MS Visual Studio 2005 Service Pack 1 is required due to the use of manifest files and using pre-built packages for some dependecies.
• Notes related to Vista: If you open the command prompt under Vista by right clicking and running as administrator, you don't get the UAC issues with Vista trying to unsuccesfully run patch as an installer in a seperate environment. You may want run Visual Studio with administrative rights anyway under Vista, as this is recommended by Microsoft (perhaps Visual Studio 2008 would not force you to do that...).

What emerge does

emerge will fetch Windows versions of numerous UNIX-like utilities and libraries from the Internet, putting them in kderoot\bin, then get the Win32 support files, then Subversion, then Perl and the Qt libraries, etc.

Then emerge compiles the Qt libraries, this takes hours.

emerge package performs the separate actions --fetch, --unpack, --compile, --install, --manifest, and --qmerge.

emerge command line options and settings

There are some options that can be used when building with emerge.

Command line switch	Command line argument		Description
-v	EMERGE_VERBOSE		This option sets the verbosity level. Currently the highest verbosity level is 3 (-v -v -v). A verbosity level of 0 should give no output and equals to -q. You can set EMERGE_VERBOSE=3 instead in the environment of the commandline or within your kdesettings.bat file.
--nocopy	EMERGE_NOCOPY		This very useful option suppresses copying the sources from the local Subversion tree to a directory within the build directory. It shouldn't be used while packaging; in the other cases it reduces the amount of harddisk used though and removes the copying time. You can set EMERGE_NOCOPY=True or =False instead.
--offline			This option suppresses the update step of the local tree - which needs some time. Be aware though that you have to have existing sources already if you want to use this option.
-t	EMERGE_BUILDTESTS		This option enables or disables KDE4 buildtests for KDE modules. Other packages will not change. Use EMERGE_BUILDTESTS=True or =False.
--buildtype=	Debug		This option enables full debugging mode for the build. Recommended if you plan to debug the runtime or provide more valuable feedback to developers about software defects. You can also change the 'set EMERGE_BUILDTYPE=RelWithDebInfo' line in the kdesettings.bat file.

Emerge works, now what?

• Once you have emerge --help working, try emerge --print-installable to get a list of valid package names. If you're unsure which are the packages you want, try selecting one from the main "kde" group (e.g. kdegames).

Hints

Updating packages

• Once you have packagename built, type emerge --unmerge packagename --noclean --target=svnHEAD packagename to update packagename from the Subversion and compile it without removing the build dir.

Altering locale and country settings

• To change locale for all users within the KDE environment, edit KDEROOT/share/config/kdeglobals file and add:

[Locale] Country=** Language=** Replace ** with your lowercase alpha-2 country code, e.g. pl for Poland. You can edit your $HOME/.kde/share/config/kdeglobals file instead to alter your local settings, not for all users.

Notes

emerge can mostly cooperate with the kdewin-installer but we're currently still working on some packages which are packaged in a wrong way. It is not recommended to use another layout then installer for directory_layout in the kdesettings.bat anymore (see that file for more detailed information).

emerge creates lots of files in \kderoot\tmp during build. After a package is successfully installed (check \kderoot\etc\portage\installed or the directory \kderoot\manifest\), you can delete its temporary directory.

Windows emerge is derived from the Gentoo portage system, but we are currently not enforcing compatibility. If you have questions about that please contact us at the channel #kde-windows on irc.freenode.net.

Vista issues

• jstaniek 12:02, 15 January 2008 (CET): UAC has infamous heuristics that make programs like patch.exe treat as installers and try to run them with admin rights (!). This heuristics can be tricked by renaming patch.exe to something like pch.exe (example) but we did not want to add item to our infrastructure. Instead it is possibleto turn off the heuristics (see the screenshot here in the security blog calling the heuristics 'severe hole in the design of UAC'). If you happen to disable the UAC, as many annoyed users and devs do (msvc demands admin rights anyway!), patch.exe should already work for you as in older Windows. Alternatively you may want to disable UAC for admins only, but this makes no sense if you are the only user of your machine and use only the admin account.
• This wiki page lists instructions on how to use program manifest to disable privilege elevation for a single binary and makes patch play nice with UAC. This should eventually be integrated to emerge scripts.