Benchmarking your Image Analysis

In past Tips I have covered the importance of using 12 bit images and the use of thresholding for identifying the objects of interest in your image.  In the future, I plan on expanding on the above concepts to do things like cell counting, cell cycle and live/dead assays.

A common question in automated image analysis is: Does the protocol I created in the image analysis program accurately measure what I want it to measure, across a large data set of images.

This Tip is to share with you a resource that can help you determine a common reference point - the image set's "correct" answer - and then measure how closely the your image analysis protocol matches that.  This answer is commonly referred to as the ground truth or gold standard.

The resource is the Broad Bioimage Benchmark Collection, a collection of annotated biological image sets for testing and validation.  This collection covers four types of ground truths and contains thousands of images for validation uses. The ground truths the collection covers are:


         Foreground/background (i.e. which pixels are objects and which pixels are background)

         Outlines of objects (i.e. which pixels belong to which objects)

         Biological labels (i.e. experiments that have been prepared with control samples for which the expected biological result is known)

This resource is provided by the Broad Institute, a collaboration between scientists from Harvard and MIT working on new approaches to cancer and human genetics.

Here are their weblinks:

Home page


Data sets

Benchmarking methodology

Please let me know your thoughts about this resource and let me know of any questions you may have.