This looks like it could be a very useful resource.

Molecular Similarity Search Benchmark (MssBenchmark) on GitHub https://github.uconn.edu/mldrugdiscovery/MssBenchmark these can be run on your local machine or on a HPC.

Currently supports

• Balltree
• Bruteforce/Exhausive search
• Chemfp 1.6.1
• the standard modulo-OR-compression algorithm, or folding
• Min-Hash
• DivideSkip
• Hnsw
• Onng
• Panng
• Pynndescent
• Risc
• SW-graph
• VPtree

They also have ChEMBL and Molport as test datasets.

Requires

• ansicolors==1.1.8
• docker==2.6.1
• h5py==2.7.1
• matplotlib==2.1.0
• numpy==1.13.3
• pyyaml==3.12
• psutil==5.4.2
• scipy==1.0.0
• scikit-learn==0.19.1
• jinja2==2.10
• h5sparse==0.1.0