BACKGROUND: Increasing recognition of the need for continuing professional development (CPD) for health professionals has created an inevitable expansion in course provision. One criterion of a successful course demands that CPD must meet local health service needs, as well as the personal and professional requirements of the participants. At a more general level, a key pedagogical requirement of any educational course is that the content and process must meet the stipulated aims and objectives, and this requires appropriate evaluation procedures. Within health care CPD, very little course evaluation routinely takes place, possibly because the area is relatively new, and secondly because relevant evaluation may be difficult to operationalize. AIMS: The current study reports a small exploratory pilot investigation into the use of a valid and reliable training needs analysis questionnaire as a means of evaluating a nurse practitioner (NP) training scheme for practice nurses (PN). METHODS: Using a before/after design, the instrument involved the participants in a pre and post course self-assessment on a range of tasks. These assessments were made according to three criteria: how critical each task is considered to be to the effective delivery of the respondent's current PN role; how critical each task is considered to be to the effective delivery of the role of the NP; and how well each task is currently performed. Various comparisons of the ratings provide a considerable amount of information that may be of use to the nurse manager in planning NP programmes. Of particular relevance to the current paper, however, are the comparisons between salience of tasks for the NP role and performance on these, before and after the course. This provides an assessment of the extent to which the course has met educational requirements. RESULTS: The present study demonstrated that overall, the course reduced skill deficits in all but seven of the tasks. Of these, six related to research and audit and the seventh to clinical examination of patients. CONCLUSION: The study is discussed in terms of the implications of the technique for course development and planning to take account of local needs, as well the instrument's reliability and validity, for use in this way.