Pilot
European
Image
Processing
Archive
The PCCV Project:
Benchmarking Vision Systems
Overview
Tutorials
Methodology
Case studies
Test datasets
Our image file format
HATE test harness
Information
General links
Conferences
Mailing lists
Research groups
Societies
Techniques (CVonline)
Software
Image databases
Other stuff
Linux on ThinkPad
We are asked surprisingly often why we believe performance characterization is important for computer vision: after all, people have been researching vision for around 40 years without worrying about it too much.
In the early days of vision research, people were mostly just exploring what could be achieved in the processing and analysis of images; so papers mostly reported that such-and-such an approach worked on a particular image or set of images. Of course, things have moved on now, a wealth of algorithms have been explored, and we know what does and doesn't work.
Or do we? Do we know that (say) John Canny's edge detector, the de facto standard for most computer vision work, truly is best? Do we know how well it finds corners in images that have a signal-to-noise ratio of about 10? Do we know that subsequent stages in an object recognition scheme are able to cope with corners that may be 3 pixels out in their position? We think you'll find that the answers to all these questions is "no." In that case, how can anyone expect to connect together a series of processing steps and have any confidence as to when the resulting system will and won't work?
Could it be that the well-known fragility of computer vision systems to unseen types of data has something to do with this?
Does performance characterization somehow solve all these problems? If we just run our algorithms against a database of images, will the result magically be more robust? Of course it won't, and we aren't pretending that it will. But what performance characterization can do is give us information about the conditions in which a particular technique is likely to succeed and fail, so that we can do something to accommodate (say) image data that don't fulfill the requirements for success.
In fact, our ultimate goal to put the development of computer vision techniques on a sound theoretical footing. But we can't get there in a single leap; we must take a series of steps, each of which is tractable and follows on from the previous one. Understanding how to characterize performance is just the first step along that road.