Traditional examination - oral or written of students will reveal improvements in knowledge, but it is not suitable for evaluation of skills. This type of evaluation technique can be used with great advantage in Computer Based Training programmes (see figure 5), but it will have very limited success within simulator-based training.

　

 An example of examination used as assessment technique in Computer Based Training is found in the SimFlex COLREG Courseware system from FORCE Technology (see figure 6).

　

 The student is - in this system - asked questions in the syllabus_ of learning material. The student responds with answers in a multiple-choice style. The answers are compared to a key, and the number of correct answers is calculated.

　

　

 The examination is not a true evaluation of the obtained knowledge since there is no initial measure, but the same method could be used for initial measurement. In this case the method could be used as an assessment technique.

　

　

 The traditional technique for measurement of skills is often called "instructor rating". This technique is used in simulator-based cadet training at Force Technology, Division for Maritime Industry (DMI) (see [1] for further description). The technique is based on subjective rating made by one or more instructors. The reliability and - especially_ - the validity of this method can be discussed due to its elements of subjectivity. The scale used for measurement is independent from level of difficulty. The same rating given both as initial and concluding rating will therefore represent an improvement of performance due to the increase in level of difficulty from initial to concluding evaluation. This is, however, in contradiction to the principle that the conditions of evaluation should be kept unchanged. i.e. that the same test should be used for both initial and concluding evaluation for purposes of comparison.

　

 The instructor rating is based on observations of the communication and behaviour of the participating crew. These observations will normally be audio visual (perhaps mediated through devices such as a video camera) and they will be supported by build in systems for registration of the participants use of instruments, knobs and handles for the control of the vessel - the so called "replay system" in the simulator.

　

 Even though the instructor rating is partly subjective and the reliability and validity can be questioned, it is still one of the most often used and most straightforward ways of measuring the participant performance (e.g. ship handling and Crew Resource Management performance) in the maritime simulator.

　

 The instructor rating methodology is well suited as instrument used as an assessment technique for the purpose of evaluation of the effect of simulator based training such as ship handling or basic navigation and watch keeping skills (e.g. the training of cadets, see [1] for further description), but it is not well suited as stand alone method for evaluation of the effect of Crew Resource Management training programmes due to the complexity of Crew Resource Management skills.

　

 We can conclude, that the training has served its purpose if and only if the result obtained by. the instructor rating technique shows a significant improvement from initial to concluding test.

　

 A special focus in observation of communication can be used as an assessment technique for evaluation of improvements in the safety of crew performance. The focus is on the time perspective, the tense in the communication. The underlying assumption is, that a crew communicating in future tense and about events in the future will be less exposed to accidents and incidents and less vulnerable than a crew communicating in present or past tense [2]. The reason for this is probably better anticipation. better planning and higher degree of preparedness.

　

 When this method is applied in the maritime simulator it is often done in that way, that an observer (e.g. instructor or psychologist). is recording communication sequences in a spreadsheet on a personal computer. This recording of communication is - when it is done in real time - subject to errors, and the subjectivity creates some doubt about reliability and validity. The recording can be done post hoc by use of video recordings. This will probably enhance the quality and reliability.

　

 Used as an assessment technique the tense element of communication can be used for evaluation of the effect of Crew Resource Management training. The effect would be, that the communication would change from initial being mostly Past or present tense to a concluding situation of a dominating majority of future tense communication (see figure 7).

　

　

 If the compound of communication is unchanged (or insignificantly changed) from initial to concluding measure, we will say, that the training have had no effect what so ever (see figure 8).

　

　

 Situation awareness is - according to Endsley [3] the ability to perceive information from and about the surrounding environment (e.g. visual information from looking through the window of the wheel house, information from instruments, radar etc., information communicated from other crew members or through radio), the ability to comprehend this information and the ability to anticipate future events on the basis of these information.

　

 The ability to perceive elements in the current situation (situation awareness level 1), comprehend the current situation (situation awareness level 2) and make projections of future events (situation awareness level 3) is very important in the everyday routine work on board ships of any type. Together the perception, comprehension and projection form the situation awareness of the operator, the crewmember. The situation awareness is important in the navigation of the vessel, the anti collision work and in the maintenance of a safe and efficient operation of the vessel in any matter. It is possible to find evidence from maritime accident reports, that a lack of situation awareness can actually cause accidents or contribute significantly to the causal development of them. According to a study performed by Grech & Horberry [4] 71% of all human error types on ships are situation awareness related problems. It is therefore evident, that proper situation awareness has significant effect on the overall safety of any type of maritime vessel and that a lack of situation awareness eventually can cause accidents such as groundings, collisions or structural damage to the vessel [5].

　

 Situation awareness is not always directly subject to training in simulator based training programmes, but it is certainly a very important element in Crew Resource Management training. The result of such training should definitely be enhanced or improved situation awareness - both from the perspective of the single crewmember and from the perspective of the entire crew. Crew or team situation awareness is build from good communication and proper sharing of information, which again is a basic Crew Resource Management skill. An evaluation of situation awareness could therefore act as an assessment technique for determination of the effect of a Crew Resource Management training programme.

　

 A number of methods for measurement of situation awareness have been suggested in the literature. One of them is the SAGAT methodology [6]. This methodology - which is originally designed for the purpose of testing airplane pilots in flight simulators is based on questionnaires, which is supposed to test the situation awareness in a given moment in time. The work performed by the operator, the pilot. officer on watch etc. has to be interrupted or abandoned when the questionnaires are filled in This is perfectly possible in a simulated environment where the simulation can be momentarily stopped, but it could be a threat to the safety this method is used in real environments. The method will simply generate a loss of situation awareness. Further, the method is suitable for single measurements of situation awareness, but it is not suitable for continuous monitoring of situation awareness over longer periods of time [5].

　

 The SAGAT methodology is well suited as instrument used as an assessment technique for the purpose of evaluation of the effect of a simulator based Crew Resource Management programme. We can conclude, that the training has served its purpose if and only if the result obtained by the methodology shows a significant improvement in situation awareness from initial to concluding test.

　

 Another important skill enhanced from Crew Resource Management training - apart from situation awareness - is the ability to manage mental workload and the ability to adjust behaviour and crew communication to the level of workload in the given situation whether this is high or low.

　

 The ability to cope with mental workload can be tested through psycho physiological methods, which are described later in this paper, through behavioural measures and through questionnaire based subjective methods such as the NASA-Task-Load Index (NASA-TLX) [7].

　

 The subjective questionnaire based techniques (like the NASA-TLX methodology) for measurement of mental workload are well suited as assessment techniques for the purpose of evaluation of the effect of a simulator based Crew Resource Management programme. We can conclude. that the training has served its purpose if and only if the result obtained by the methodology shows a significant decrease in subjective (perceived) mental workload from initial to concluding test due to improved workload management skills.