dm.incrementalsearch is an efficient low level search execution engine. Its
primary purpose is to incrementally perform searches involving and, or and
not queries over ZODB.BTrees.
Incrementally means here that hits are determined one at a time: the first
hit, then the second hit, then the third hit, etc. Therefore, the first few
hits can be determined extremely fast. But even if all hits need to be determined,
the incremental execution of subqueries can lead to speedups of several orders for
some query types (especially those dominated by specific and queries).
Queries involving large or subqueries are difficult to optimize in the standard
way. But often they can be replaced by incremental filtering. With this technique,
a subquery is removed from the original search, the modified search executed and
the result filtered by the removed subquery. incrementalsearch supports
incremental filtering and thereby can again gain serveral orders of speedup for
otherwise difficult to treat query types.
The primary concept is that of an ISearch (incremental search). This is
conceptionally a sorted list, computed incrementally (or lazily). The elements of
this list are the ISearch's hits. The ISearch's keytype determines the type of the
list elements. Currently supported are OBJECT (comparable Python objects), INT
(Python 32 bit integers) and LONG (Python 64 bit integers).
|