Baloo: Difference between revisions

From KDE Community Wiki

Latest revision as of 08:03, 5 May 2024

Baloo is the file indexing and file search framework for KDE Plasma, with a focus on providing a very small memory footprint along with with extremely fast searching.

User documentation

User documentation for search types and document properties that Baloo indexes

Ways to communicate

Mailing List: [email protected] (info page)

IRC Channel: #kde-devel on Libera Chat

Phabricator project: https://phabricator.kde.org/project/view/261

Top bugs and feature requests

Bugs: https://bugs.kde.org/buglist.cgi?bug_severity=critical&bug_severity=grave&bug_severity=major&bug_severity=crash&bug_severity=normal&bug_severity=minor&bug_status=UNCONFIRMED&bug_status=CONFIRMED&bug_status=ASSIGNED&bug_status=REOPENED&list_id=1629910&priority=VHI&priority=HI&product=frameworks-baloo&query_format=advanced

Feature requests: https://bugs.kde.org/buglist.cgi?bug_severity=wishlist&bug_status=UNCONFIRMED&bug_status=CONFIRMED&bug_status=ASSIGNED&bug_status=REOPENED&list_id=1629911&priority=VHI&priority=HI&product=frameworks-baloo&query_format=advanced

Indexing limitations

Baloo uses the file metadata extractors in KFileMetadata to get information about each file it indexes. This means for a file's content to be indexed

the file must have a recognizable MIME type
KDE must have an extractor for that MIME type. Use the command-line utility kmimetypefinder5 to determine a file's mime type.
- Due to a glib bug, the MIME type of HTML files can change from text/html to application/x-extension-html. The KDE file metadata extractors don't recognize the latter. That bug has a workaround to reset the MIME types to the usual values.
KFileMetadata uses the aging utilities catdoc, xls2csv, catppt to index content of files using the Microsoft Office Word, Excel, and PowerPoint file formats (source), and these utilities have undocumented limitations (bug 438455).
- catdoc home page
- Debian's bug list for catdoc; RedHat's bug list for catdoc
KFileMetadata does not index file names or file contents in ZIP archives.
KFileMetadata does not index the contents of Open Document Format files that are ZIP archives, nor does it index "flat" Open Document Format files that are complex XML files.

Other limitations:

Baloo doesn't index text files (those whose MIME type is detected as "text/something") over 10 MB (source).
The KFileMetadata extractor for text attempts to convert text to Unicode. If the file uses another encoding, such as iso-8859-1, any file contents after the first character that is invalid in Unicode will not be indexed (bug 440537). You may find the -i option to the file command-line utility useful; it tries to infer the character set of a file, e.g. file -i path/to/myfile.txt. You can use the iconv command-line utility to report invalid encodings and convert encodings to UTF-8.
If a file's modification time is January 1 1970 ("zero" in the Unix epoch) or earlier, baloo will reindex it each time it starts (or you run balooctl check) (bug 456108), and balooshow will be confused about the file's "Mtime" if it is before January 1 1970. As a workaround you can change th e modification time to something after 1970, e.g. touch -m --date=2022-01-01 path/to/myfile.
Some users report that baloo doesn't properly index some files extracted from zip or JAR files. A workaround is to clear them from baloo's index then reindex them. with balooctl clear /path/to/file then balooctl index /path/to/file .

Other Baloo pages here

Information may be obsolete.

Using Baloo

Baloo is not an application, but a daemon to index files. Applications can use the Baloo framework to provide file search results. For example, Dolphin's Content search can use Baloo.

KDE System Settings > File Search provides an intentionally limited number of settings. You can make additional adjustments in Baloo's configuration file.

There is a command-line search utility, baloosearch

Plasma 6 versions

In some Linux distributions that ship Plasma 6, the latest command-line utilities may have a 6 appended, thus balooctl6, baloosearch6, and balooshow6.

balooctl

balooctl is a CLI command to perform certain operations on Baloo. Enter balooctl --help in a terminal app such as userbase:Konsole to list its available subcommands.

@@ Line 1: / Line 1: @@
 [[File:Mascot konqi-support-search.png|thumbnail|right|Help [[Konqi]] find what he wants!]]
-[https://projects.kde.org/projects/frameworks/baloo Baloo] is the file indexing and file search framework for KDE Plasma, with a focus on providing a very small memory footprint along with with extremely fast searching.
+Baloo is the file indexing and file search framework for KDE Plasma, with a focus on providing a very small memory footprint along with with extremely fast searching.
+== User documentation ==
+[https://github.com/KDE/baloo/blob/master/docs/user/searching.md User documentation] for search types and document properties that Baloo indexes
+== Ways to communicate ==
 :Mailing List: [email protected] ([https://mail.kde.org/mailman/listinfo/kde-devel info page])
-:IRC Channel: #kde-devel on freenode
+:IRC Channel: [https://web.libera.chat/ #kde-devel on Libera Chat]
 :Phabricator project: https://phabricator.kde.org/project/view/261
-== Top issues ==
+== Top bugs and feature requests ==
-* DB size grows forever; deleted files are never removed: https://bugs.kde.org/show_bug.cgi?id=353874
+'''Bugs:''' https://bugs.kde.org/buglist.cgi?bug_severity=critical&bug_severity=grave&bug_severity=major&bug_severity=crash&bug_severity=normal&bug_severity=minor&bug_status=UNCONFIRMED&bug_status=CONFIRMED&bug_status=ASSIGNED&bug_status=REOPENED&list_id=1629910&priority=VHI&priority=HI&product=frameworks-baloo&query_format=advanced
-* Files in plasma vault are indexed and available when the vault is closed: https://bugs.kde.org/show_bug.cgi?id=390830
+<br/><br/>
-* Baloo's file metadata extractor doesn't run sandboxed: https://phabricator.kde.org/D8532
+'''Feature requests:''' https://bugs.kde.org/buglist.cgi?bug_severity=wishlist&bug_status=UNCONFIRMED&bug_status=CONFIRMED&bug_status=ASSIGNED&bug_status=REOPENED&list_id=1629911&priority=VHI&priority=HI&product=frameworks-baloo&query_format=advanced
-* Baloo DB is entirely rewritten for every write, introducing delays and SSD lifespan issues when DB size grows large: https://bugs.kde.org/show_bug.cgi?id=373021
-* Baloo doesn't notice/re-index renamed files saved over other ones: https://bugs.kde.org/show_bug.cgi?id=397242
-* Need better error handling for lmdb errors: https://bugs.kde.org/show_bug.cgi?id=368557
+== Indexing limitations ==
-* Baloo's initial indexing introduces IO-related UI lag and delays: https://bugs.kde.org/show_bug.cgi?id=400704 & https://bugs.kde.org/show_bug.cgi?id=356357
+Baloo uses the file metadata extractors in [https://invent.kde.org/frameworks/kfilemetadata KFileMetadata] to get information about each file it indexes.
-* No apparent effect after removing a folder from the index; need to restart or kill all Baloo processes: https://bugs.kde.org/show_bug.cgi?id=373430
+This means for a file's content to be indexed
-* baloo_file doesn't get killed when you stop indexing or force a re-index: https://bugs.kde.org/show_bug.cgi?id=353559
+* the file must have a recognizable MIME type
-* Baloo should index NTFS disks: https://bugs.kde.org/show_bug.cgi?id=399885
+* KDE must have an extractor for that MIME type. Use the command-line utility <code>kmimetypefinder5</code> to determine a file's mime type.
-* Baloo should have a user-facing option to index source code: https://bugs.kde.org/show_bug.cgi?id=380893
+** Due to a [https://gitlab.gnome.org/GNOME/glib/-/issues/2511#note_1293471 glib bug], the MIME type of HTML files can change from  <code>text/html</code> to <code>application/x-extension-html</code>. The KDE file metadata extractors don't recognize the latter. That bug has a workaround to reset the MIME types to the usual values.
-* Baloo crashes a lot in various places:
+* KFileMetadata uses the aging utilities <code>catdoc</code>, <code>xls2csv</code>, <code>catppt</code> to index content of files using the Microsoft Office Word, Excel, and PowerPoint file formats ([https://invent.kde.org/frameworks/kfilemetadata/-/blob/master/src/extractors/officeextractor.cpp#L20 source]), and these utilities have undocumented limitations ([https://bugs.kde.org/show_bug.cgi?id=438455 bug 438455]).
-** https://bugs.kde.org/show_bug.cgi?id=367480 <tt>PostingCodec::decode()</tt>
+** [http://www.wagner.pp.ru/~vitus/software/catdoc/ catdoc home page]
-** https://bugs.kde.org/show_bug.cgi?id=357206 <tt>PostingDB::get()</tt> or <tt>WriteTransaction::commit()</tt>
+** [https://bugs.debian.org/cgi-bin/pkgreport.cgi?repeatmerged=no&src=catdoc Debian's bug list for catdoc]; [https://bugzilla.redhat.com/buglist.cgi?quicksearch=catdoc RedHat's bug list for catdoc]
-** https://bugs.kde.org/show_bug.cgi?id=361186 <tt>PositionDB::get()</tt> or <tt>WriteTransaction::commit()</tt>
+* KFileMetadata does not index file names or file contents in ZIP archives.
-** https://bugs.kde.org/show_bug.cgi?id=397644 <tt>DocumentTimeDB::get()</tt>
+* KFileMetadata does not index the contents of Open Document Format files that are ZIP archives, nor does it index "flat" Open Document Format files that are complex XML files.
-** https://bugs.kde.org/show_bug.cgi?id=390823 <tt>IdTreeDB::get()</tt>
-** https://bugs.kde.org/show_bug.cgi?id=372880 <tt>IdFilenameDB::get()</tt>
-** https://bugs.kde.org/show_bug.cgi?id=395902 <tt>DocumentTimeDB::del()</tt>
-** https://bugs.kde.org/show_bug.cgi?id=393583 <tt>PostingDB::del()</tt>
-** https://bugs.kde.org/show_bug.cgi?id=372655 <tt>IdFilenameDB::del()</tt>
-** https://bugs.kde.org/show_bug.cgi?id=385056 <tt>PendingFileQueue::enqueue()</tt> or <tt>FileWatch::slotFileDeleted()</tt>
-** https://bugs.kde.org/show_bug.cgi?id=389679 <tt>Transaction::commit()</tt> or <tt>lmdb::mdb_txn_commit()</tt>
-** https://bugs.kde.org/show_bug.cgi?id=380111 <tt>IdTreeDB::put()</tt>
-** https://bugs.kde.org/show_bug.cgi?id=395888 <tt>DocTermsCodec::decode()</tt>
-** https://bugs.kde.org/show_bug.cgi?id=374239 <tt>DocumentIdDB::size()</tt>
-** https://bugs.kde.org/show_bug.cgi?id=384133 <tt>DocumentIdDB::fetchItems()</tt>
-** https://bugs.kde.org/show_bug.cgi?id=393472 <tt>PostingDB::iter</tt> or <tt>PostingDB::prefixIter</tt> when invoked from KRunner
-** https://bugs.kde.org/show_bug.cgi?id=403475 <tt>putDifferentialVarInt32()</tt> or <tt>PositionCodec::encode()</tt>
-== Useful pages ==
+Other limitations:
-* [[Baloo/Architecture | Architecture]]
+* Baloo doesn't index text files (those whose MIME type is detected as "text/''something''") over 10 MB ([https://invent.kde.org/frameworks/baloo/-/blob/master/src/file/extractor/app.cpp#L143 source]).
-* [[Baloo/Configuration]]
+* The KFileMetadata extractor for text attempts to convert text to Unicode. If the file uses another encoding, such as iso-8859-1, any file contents after the first character that is invalid in Unicode will not be indexed ([https://bugs.kde.org/show_bug.cgi?id=440537 bug 440537]). You may find the <code>-i</code> option to the <code>file</code> command-line utility useful; it tries to infer the character set of a file, e.g. <kbd>file -i ''path/to/myfile.txt''</kbd>. You can use the <code>iconv</code> command-line utility to report invalid encodings and convert encodings to UTF-8.
-* [[Baloo/Debugging]]
+* If a file's modification time is January 1 1970 ("zero" in the Unix epoch) or earlier, baloo will reindex it each time it starts (or you run <kbd>balooctl check</kbd>) ([https://bugs.kde.org/show_bug.cgi?id=456108 bug 456108]), and <code>balooshow</code> will be confused about the file's "Mtime" if it is before January 1 1970. As a workaround you can change th e modification time to something after 1970, e.g. <kbd>touch -m --date=2022-01-01 path/to/myfile</kbd>.
-* [[Baloo/NepomukPort]]
+* [https://discuss.kde.org/t/how-do-i-troubleshoot-baloo/2830/12 Some users] report that baloo doesn't properly index some files extracted from zip or JAR files. A workaround is to clear them from baloo's index then reindex them. with <kbd>balooctl clear ''/path/to/file''</code> then <kbd>balooctl index ''/path/to/file''</kbd> .
-* [[Baloo/Tasks]]
-* [[Baloo/XapianAlternatives]]
+== Other Baloo pages here ==
-* [[Baloo/XapianProblems]]
+Information may be obsolete.
+{{Special:PrefixIndex/{{FULLPAGENAME}}/}}
 == Using Baloo ==
-Baloo is not an application, but a daemon to index files.  Applications can use the Baloo framework to provide file search results.
+Baloo is not an application, but a daemon to index files.  Applications can use the Baloo framework to provide file search results. For example, [[Dolphin]]'s Content search can use Baloo.
+KDE System Settings > File Search provides an [http://vhanda.in/blog/2014/04/desktop-search-configuration/ intentionally limited number of settings].  You can make additional adjustments in [[Baloo/Configuration | Baloo's configuration file]].
+There is a command-line search utility, <kbd>baloosearch</kbd>
-An [http://vhanda.in/blog/2014/04/desktop-search-configuration/ intentionally limited number of settings] are provided via the KDE System Settings "Desktop Search" icon.  Other adjustments can be made directly to the [[Baloo/Configuration | configuration file]].
+=== Plasma 6 versions ===
+In some Linux distributions that ship Plasma 6, the latest command-line utilities may have a <kbd>6</kbd> appended, thus <kbd>balooctl6</kbd>, <kbd>baloosearch6</kbd>, and <kbd>balooshow6</kbd>.
 == balooctl ==
-<code>balooctl</code> is a CLI command to perform certain operations on Baloo. The operations are indicated by '''arguments'''. All available arguments can be listed by issuing <code>balooctl --help</code> in terminal app, such as [[userbase:Konsole]].
+<code>balooctl</code> is a CLI command to perform certain operations on Baloo. Enter <code>balooctl --help</code> in a terminal app such as [[userbase:Konsole]] to list its available subcommands.
-To be noted, argument <code>checkDb</code> is only meant to be used by developers for debugging. All it does is to print some messages but not to sanitize the database. And the execution time grows exponentially with the number of indexed files (see [https://bugs.kde.org/show_bug.cgi?id=380465#c5 Comment 5 under bug #380465]).
+See also [[Baloo/Debugging]].