View source for Amarok/Development/BatchMode

[[Category:Amarok]] [[Category:Development]]
Starting in Amarok 2.1, Amarok has the ability to run its collection scans in "Batch Mode."

Batch mode scanning lets you decouple the scanning from Amarok.  Instead of having to have Amarok open to run the scan, you run the scan from the command line and save the output.  This makes scanning from cron or from scripts easy.  Another huge advantage is that if your music resides on a remote machine, you can scan it locally to that machine, where access is fast, instead of over the network on your client machine.

This actually entailed a change in the way Amarok was packaged.  Previously, to use Amarok's collection scanner, the entire Amarok package had to be installed.  Now, most distributions should be building the utilities (currently just the scanner, but soon some other utilities as well) as a separate package, usually called amarok-utils or amarok-utilities.  These were split out in order to remove all KDE (and therefore X11) dependencies from these utilities.  This means you can perform batch mode scans on a headless, X11-less machine!

Just like in Amarok, there are two types of scans: full and incremental.  How to perform each is listed below.  Keep in mind that this is intended to be scripted, so while these steps can be performed by hand, they are a bit tedious (but very easy to script).

If you run into issues or find bugs, please file a bug report at bugs.kde.org and be sure to CC mitchell @t k d e dot o r g

Note: $KDEHOME is normally .kde or .kde4, depending on distribution.

== Full scan ==

You're going to run amarokcollectionscanner with the following options:

-b -> batch mode -- this lets the scanner know it's running in batch mode.

-r -> recursive -- tells the scanner to scan recursively.  This flag is optional; you don't have to scan recursively if you don't want.  You can append the output from multiple runs on specific directories together instead if you like.

--rpath "<path>" -> relative path.  This flag is optional; only use it if on the machine you are scanning your music, the path to your music is different than on the machine that will run Amarok.

This option takes some explaining.  Let's suppose that on the server holding your music, the music is at /media/Music.  So you would run the collection scanner from /media; this would therefore be your path.  However, suppose that this is shared via NFS to your laptop that is going to run Amarok, and on the laptop it is located at /mnt/Music.  You therefore want the scanner to output paths relative to /mnt instead of /media.  So if you pass in --rpath "/mnt", it will subsitute /mnt for the current working directory at the time of the scan.

So let's suppose you have the following scenario:

* On your server, the music is located in /media/Music
* On the client, this music is mounted at /mnt/Music

Here are the steps to performing a full scan.

# On the server, cd to /media
# Run: amarokcollectionscanner -b -r --rpath "/mnt" Music > amarokcollectionscanner_batchfullscan.xml (note: this '''must''' be the name of the output file)
# Transfer this file to the client.  Note: the client will consume this file, so keep a backup if you like)
# The client should put it in their $KDEHOME/share/apps/amarok directory
# In Amarok, run a full rescan.  Amarok will consume the amarokcollectionscanner_batchfullscan.xml file and read its contents instead of launching its own collection scanner.

Another scenario:

* On your server, one music directory is located in /media/Music, and another at /home/jeff/Music
* On your client, you have the /mnt/Music mount as before, and also have /home/jeff/Music mounted from the server in the same directory /home/jeff/Music directory on the client.

Here are the steps you would use:

# On the server, cd to /media
# Run: amarokcollectionscanner -b -r --rpath "/mnt" Music > ~/amarokcollectionscanner_batchfullscan.xml (note: this '''must''' be the name of the output file)
# On the server, cd to /home/jeff
# Run: amarokcollectionscanner -b -r >> ~/amarokcollectionscanner_batchfullscan.xml (Yes, the correct behavior in this case is to append the output to the existing file.  The resultant file is not proper XML, but the scanner knows how to split the various XML chunks out.)
# Transfer this file to the client.  Note: the client will consume this file, so keep a backup if you like)
# The client should put it in their $KDEHOME/share/apps/amarok directory ($KDEHOME is usually .kde or .kde4)
# In Amarok, run a full rescan.  Amarok will consume the amarokcollectionscanner_batchfullscan.xml file and read its contents instead of launching its own collection scanner.

== Incremental Scan ==

Please read the full scan section first so you have an idea of what Amarok is doing.  One notable difference between a full and incremental batch scan is that the incremental scan does not support the --rpath option, which means that either you must run it on the same computer as where Amarok will be run, or you must have the same path structure to your music.

Every time that the scanner runs while Amarok is open, it writes a file to $KDEHOME/share/apps/amarok called amarokcollectionscanner_batchincrementalinput.data.  This file contains a listing of all of your defined directories in your collection, and this is the file that lets us perform the batch incremental scan.

You're going to run amarokcollectionscanner with the following options:

-b -> batch mode -- this lets the scanner know it's running in batch mode.

-i -> incremental -- this lets the scanner know it's in incremental mode.

-r -> recursive -- this does special things in batch incremental mode...

Normally, when performing an incremental scan, if you have selected to scan recursively inside Amarok, the scan manager inside Amarok builds a list of directories that already recursively contains all known directories, and sends them to the scanner.  If the scanner doesn't know what to do about a new folder, it sends a D-Bus query to Amarok to ask whether that folder exists in the scope of the collection.  Since we're running with the assumption that Amarok isn't available, the -r flag will end up controlling what the scanner does in this instance.  If you specify -r, it will read new directories.  If you don't, it won't.  Note that in this case, the scanner will think it's a new directory every time that it's run, and keep reading the files over and over.  I don't think this is a problem, but haven't heavily tested it.  Do let me know.

There's one final bit: when running in batch incremental mode, you don't pass in a folder to the scanner.  Instead, it expects the first folder argument to be the full patch of the amarokcollectionscanner_batchincrementalinput.data file.  In normal Amarok usage the scan controller checks mtimes of folders to decide whether they need to be rescanned; in batch incremental mode, the scanner the mtime of this file to determine whether something has changed.  So, each time you run the scan, you should touch that file to update the mtime, so that future scans are incremental instead of additive.

Here are the steps you would use:

# Run: amarokcollectionscanner -b -i /my/home/$KDEDIR/share/apps/amarok/amarokcollectionscanner_batchincrementalinput.data >> /my/home/$KDEDIR/share/apps/amarok/amarokcollectionscanner_batchincrementalscan.xml (note: this '''must''' be the name of the output file)
# Run: touch /my/home/kdedir/share/apps/amarok/amarokcollectionscanner_batchincrementalinput.data
# Repeat steps 1 and 2 indefinitely...
# In Amarok, run an incremental rescan, or "update" (this happens automatically when Amarok is opened).  Amarok will consume the amarokcollectionscanner_batchincrementalscan.xml file and read its contents instead of launching its own collection scanner.