Paper Reading #21 - Automatically Identifying Targets

Comments:
Comment 1
Comment 2

References:
Title: Automatically Identifying Targets Users Interact with During Real World Tasks
Authors: Amy Hurst, Scott E. Hudson, and Jennifer Mankoff
Venue: IUI 2010, Feb. 7-10 2010

Summary:
In this paper, the authors describe a method of gathering user click data in an accurate, device-agnostic way. They do this by using a hybrid method of kernel-based tracking combined with image identification.

Their user data gatherer, called CRUMBS, works in two levels. At the lower level, a series of different data gatherers reports on what they each think is what the user clicked on. For example, some of the low-level gatherers include Microsoft's Accessibility API, an image difference checker, as shown above, and a template checker. Then, at the high level, a machine learning interface decides based on all the data gathered from the low-level gatherers to make a final decision on what the user clicked on.

With their method, they reported a 92% correct click identification rate, which they mention is higher than using only the accessibility API. Furthermore, they mention if they captured a larger portion of the screen on a click (they currently grab only a 300x300 space), they could get an even larger portion of clicks correct.

Discussion:
I think that CRUMBS could be a very useful tool to use when testing how users make use of your program. If the collection data from other sources is as bad as they say in the paper, then it is very difficult to gather real usage information for programs, and this could help.

One thing I am curious about though is if this information gatherer must be turned on and off manually or if it does so automatically. Otherwise, it might gather usage information in a clicking space where it isn't needed, such as in a video game.

(Image courtesy of: this paper)