Using "Human-in-the-loop" in an Adaptive System: An Evalutation Study of the ConCall System

Abstract

The rapid development of new information technology has brought forward the possibilities of efficient collaborative work. People can seek in vaster information spaces, and are the target of a tidal wave of information surging by every day through email, newsgroups, news sites, etc. The access to information is escalating and we feel overwhelmed. We call it information overload. One way of solving this problem would be to reduce the information that passes through each individualâs sphere. The aim would be to create a system that could filter through the incoming information in an intelligent way so that we reduce the flow but at the same time get through all the information relevant and interesting to the user of such a system. This master thesis presents an evaluative study of the ConCall system, where we take a look at how to use the best of both human and machine to solve this problem. ConCall is an adaptive system, implementing the EdInfo ideas which is to combine human expertise with machine intelligence in order to achieve a high quality of filtered information to its end users. ConCall was built up to be a call for paper and participation filtering system targeting researchers as users. The study of ConCall was an experimental evaluation aiming both to look at the functionality and utilities in ConCall and show that this concept works. The study was also one of the last steps in a bootstrapping circle with the intentions to be a steppingstone to the start of the next circle of development. The study showed that a filtering system like this could be both useful and was desired as a help to sort through the continuos stream of incoming calls for papers and participation as an alternative to what most of the participants used today: unstructured streams of calls coming in through e-mail. The study also showed that recommendations are preferred having colleagues and friends as senders. Another interesting result concerned a dependency between the motivation in users and the filtering performance in means of precision and recall. Lessons learned from this study had to do both with setting up experimental situations and the difficulties of developing adaptive systems.