Description

Annoy (Approximate Nearest Neighbors Oh Yeah) is a C++ library with Python bindings to search for points in space that are close to a given query point. It also creates large read-only file-based data structures that are mmapped into memory so that many processes may share the same data.

There are some other libraries to do nearest neighbor search. Annoy is almost as fast as the fastest libraries, (see below), but there is actually another feature that really sets Annoy apart: it has the ability to use static files as indexes. In particular, this means you can share index across processes. Annoy also decouples creating indexes from loading them, so you can pass around indexes as files and map them into memory quickly. Another nice thing of Annoy is that it tries to minimize memory footprint so the indexes are quite small.

Why is this useful? If you want to find nearest neighbors and you have many CPU's, you only need to build the index once. You can also pass around and distribute static files to use in production environment, in Hadoop jobs, etc. Any process will be able to load (mmap) the index into memory and will be able to do lookups immediately.

We use it at Spotify for music recommendations. After running matrix factorization algorithms, every user/item can be represented as a vector in f-dimensional space. This library helps us search for similar users/items. We have many millions of tracks in a high-dimensional space, so memory usage is a prime concern.

Summary of features

• Euclidean distance, Manhattan distance, cosine distance, Hamming distance, or Dot (Inner) Product distance
• Cosine distance is equivalent to Euclidean distance of normalized vectors = sqrt(2-2*cos(u, v))
• Works better if you don't have too many dimensions (like <100) but seems to perform surprisingly well even up to 1,000 dimensions
• Small memory usage
• Lets you share memory between multiple processes
• Index creation is separate from lookup (in particular you can not add more items once the tree has been created)
• Native Python support, tested with 2.7, 3.6, and 3.7.
• Build index on disk to enable indexing big datasets that won't fit into memory (contributed by Rene Hollander)

Additional Information