Findings from these activities

• cognitive methods of evaluation were found to be useful in providing input into system design and fine-tuning of computer algorithms in order to produce system outputs more compatible with human cognitive processes (particularly for system components tested for automatically generating operative reports for ICU staff)

• although there is considerable individual variance in the way that human physicians judge relevance of articles (in relation to real patient cases), there are essential aspects of relevance judgments that cut across subjects and need to be considered in design of computer algorithms for automating this cognitive task

• differences in strategies for judging relevance of articles were found across different levels of expertise, with experts applying less information in such judgements than residents. Implications for system design include development of methods for presentation of information fine-tuned to individuals' level of medical expertise

• analysis of questions generated in real medical situations can be used to drive selection and design of the information retrieval components of DLI2 (i.e. as a basis for system design, fined-tuned to the type of questions that arise in environments such as the ICU)

• methods of usability engineering (including video based usability testing) were successfully applied in comparing different Web-based classifications - we are exploring this in order to improve the content and organization of medical information presented over the WWW (based on empirical data from study of human preferences in information classification)