Abstract: There has - since the development of simulator based maritime training - been a demand the measurement of the effect of the training. Tools used to obtain these measures are often called -assessment techniques". This paper will describe a repertoire of different assessment techniques. Some of these are well known and already in use. A special focus will be held on an in the maritime domain new and very promising methodology for measurement of crew stress. The methodology can - if it is developed and refined - be used as an assessment technique to evaluate the effect of Crew Resource Management training. The methodology will be used in the European project SPIN-HSV, and it is based on psycho physiological measurements of brain activity (EEG). The concluding recommendation is a composite use of different human factors parameters as indicators of increase in human performance (knowledge and skills) and a composite use of different methodologies for measurement of these parameters.

　

 There has - since the development of simulator based maritime training - been a demand for measurement of the effect of the training and an answer to the question: Does the training really work? Is the knowledge and/or skills of participants improved during training or are they unchanged or - even worse - aggravated? Tools to investigate and answer these questions and used to obtain these measures are often called "assessment techniques".

　

 This paper will describe a repertoire of different assessment techniques from subjective measures based on observations and questionnaires to objective measures based on psycho physiological techniques such as the example Heart Rate Variability or electric activity of the brain (EEG). Some of these are already in use. A special focus will be held on an in the maritime domain new and very promising methodology for measurement of crew stress. The methodology can - if it is developed and refined - be used as an assessment technique to evaluate the effect of the human factors oriented Crew Resource Management training. The methodology will be used in the European project SPIN-HSV, and it is based on psycho physiological measurements of brain activity (EEG).

　

 The purpose of maritime simulator based training is seen from the perspective of the student or participant - the attainment of further knowledge and/or skills in various disciplines e.g. ship handling or Crew Resource Management. Assessment techniques are used to evaluate whether a goal is reached or not. The simulator is well suited for training of skills, but traditional assessment techniques from other learning contexts are usually designed for measurement of knowledge.

　

 Two measures are required in the evaluation of any kind of learning process: An initial measure and a concluding measure. The initial measure should be taken in the simulator immediately before training (but after familiarization with the simulator), and the concluding measure should be taken immediately after training. The results should be compared and tested for significance.

　

 The initial measure is used as a reference point to which the concluding measure can be compared. It is expected, that a comparison of the concluding measure and the initial measure show, that the level of knowledge and/or skill is significantly higher after training than before (see figure 1). The training has had no effect if the level is unchanged (see figure 2), and the effect has been negative if the concluding level of skill and/or knowledge is lower than the initial level (see figure 3).

　

　

 This type of measurement is called "intervention study" because the subject of the study is the effect of the intervention. In this case the intervention is training, but it could as well has been for example introduction of new procedures or new instruments or technology.

　

　

　

2.1 Significance

　

 Test for significance is done according to standard procedures. If the results are found to be insignificant, no conclusion. about the effect of the training can be drawn. The obtained results could be purely random fluctuations.

　

2.2 Confounding variables

　

 When we would like to evaluate the effect of training, the problem about confounding variables is always relevant (see figure 4). To eliminate the problems related to confounding variables we must ensure, that the training is the only intervention or influence the participants are subject to. We must be sure, that we use exactly the same test, exactly the same subjects and that all other variables than the training is eliminated.

　

　

 One example could be experience. This is in particular a problem if the time distance between the initial measure and the concluding measure becomes too large. The initial measure should be taken immediately before training begins and the concluding measure should be taken immediately after the training is finished. Otherwise it is impossible to determine if a change in performance is due to general increase in experience or due to the training . Another example could be a mixture of training and increased salary. If salary_ were increased it could lead to increase of motivation and thereby better performance.

　

2.3 The principle of composite use of methods

　

 The overall quality of the assessment - especially the validity - can be enhanced by composite use of methodologies. The methods described in this paper covers a variety of methodologies from subjective measures based on observations and questionnaires to objective measures based on psycho physiological techniques such as for example Heart Rate Variability or electric activity of the brain (EEG). Corresponding to that, the human factors parameters suggested as indicators of improved performance (improved knowledge and skills) also covers a wide selection, for example mental workload, situation awareness, stress and fatigue.

　

 It is most likely, that the principle of composite use of methods could emerge to be a very powerful assessment tool for evaluation of the effect of maritime simulator based training. If - for example - the effect of Crew Resource Management training is examined, more than one parameter should be used as indicator, and each of these parameters should be measured by means of more than one method or technique.