On behalf of the Dutch Ministry of Transport, Maritime Simulation Rotterdam (MSR) was requested to investigate the effectiveness of BRM training in the prevention of collisions and groundings. In order to assess the effectiveness of BRM training a phased approach was chosen: Firstly a review of reported accident cases and secondly a simulator based study. During the accident analyses, several BRM related clues were scored and an overview was made of the most relevant clues in relation to the accidents. During the simulator -based study, two groups of Dutch Officers (one group had previously attended a BRM Course and the other group hadn't) were assessed on a full mission simulator in order to compare the performance difference with respect to BRM clues. By means of the results of the simulator study it was possible to demonstrate which BRM related clues were improved by means of training. The next step was combining these results with the results of the accident analyses, in order to assess the potential impact of BRM training on accident reduction.

　

1.1 Back ground

　

 The number of accidents where it is acknowledged that inadequate Bridge Resource Management (BRM) is an underlying factor seems to be increasing. This can be concluded by reading the accident investigation reports made by the Dutch Shipping Council (Dutch: Raad voor de Scheepvaart). This paper gives the results of an investigation carried out in order to provide an answer to the question as to whether there is a relationship between the quality of BRM and the probability of an accident occurring. To be able lo answer this question, the Directoraat-Generaal Goederenvervoer (DGG) has initiated this research project. MSR was requested to investigate the effectiveness of BRM training in the prevention of collisions and groundings.

　

 DGG initiated this research project based on internal questions there were:

 The research questions formulated from these internal questions are the following:

　

1.3 Study Set-up

　

 In order to determine the effectiveness of BRM training a phased approach was chosen: firstly a review of reported accident cases and secondly a simulator-based study. The study started with the definition of the BRM related aspects for watch keeping personnel; thereto the BRM Courses given at MSR were used as guideline. Thereafter a review of reported accident cases and a simulator-based study were executed. During the accident analyses, the several BRM related clues were scored and an overview was made of the most relevant clues in relation to accidents. During the last phase, a simulator based study, two groups of Dutch Officers (one group had previous attended a BRM Course and the other group hadn't) were assessed on a full mission simulator in order to compare the performance difference with respect to BRM clues.

　

 Bridge Resource Management is a training programme designed to ensure effective use of personnel and equipment during vessel operations. Similar to aviation's Crew Resource Management, BRM is designed to reduce errors and omissions in bridge operations through a simple system of checks and delegation of duties. BRM emphasizes a co-ordinated effort among bridge personnel to ensure smooth, efficient, and safe operation of the vessel. The 1995 amendments to the International Convention of Standards of Training, Certification and Watch-keeping for Seafarers (STCW) [8] include a requirement for training in bridge team procedures and a recommendation for training in BRM techniques. A bridge team consists of those crewmembers responsible for the safe navigation of the ship. While onboard, the pilot becomes part of this bridge management team. The Master maintains overall responsibility for safe navigation, but the pilot serves as the advisor to the Master on local conditions.

　

 There is no standard BRM Course. In the Netherlands the institutes that are certified to teach the course are STC in Rotterdam, Nova College in IJmuiden and MSR in Rotterdam. The main difference between the programmes is the duration of the course and the amount of exercises on the simulators. For this study the MSR course was used as reference course. The MSR course is a five-day course of which 2 days college, 2.5 day simulator training and 0.5 day case study.

　

3.1 Introduction

　

 By using the technique of Content Analysis a number of accident reports issued by the Dutch Shipping Council were assessed using experienced BRM facilitators. By analyzing these reports an attempt was made to answer the following questions:

　

　

 It will be clear that these questions should be investigated with enough reliability and validity to provide a clear answer. To do so, we must have a high quality measurement instrument and sufficient data to be analyzed.

　

3.2 Methodology

　

 Within BRM Courses, a number of so called BRM clues are used to determine which human errors play a role in accidents. These clues are:

 It seems logical to incorporate these clues into the assessment method.

　

 The assessment procedure was two-fold. First, the preliminary parts of each report were scored in terms of the BRM clues. These clues were assumed to be "latent constructs" that manifested themselves as concrete behavioural descriptions in the text. Next, the involvement of a number of general factors in the incident was judged using rating scales.

　

 The assessors were asked to report the BRM content of each accident report by reading the objective information and by:

　

 If there was a large influence of BRM related factors in the accidents, this should be visible by a large number of clues counted and a perceived big importance of the BRM related aspects by the assessors. During a 4-hour session, seven experienced BRM training facilitators were trained in this procedure and were subsequently given a number of accident reports to evaluate.

　

3.3 Results

　

 The accident reports used for this study are the collisions and groundings reported in the years 1998, 1999, 2000 and 2001 up to number 16. In total 52 cases have been evaluated, 22 involving a grounding and 30 involving a collision. During the period 1998-2001 the Dutch Shipping Council published 96 reports containing all type of accidents with or on ships. Two independent assessors assessed each report.

　

 Reliability of the Assessment Procedure; To check the reliability of the scoring protocol used in the content analysis of the investigation reports, the correlation between assessor pairs was investigated. This correlation was 0.73 on average for all variables, that was, rating scale type variables and clue score variables taken together. For the rating scale type variables alone the average correlation was 0.93. Thus, it can be concluded that the correspondence between the separate assessors was satisfactory, which implies that further analyses could be done using the assessor averages.

　

 Accident Circumstances; The circumstances of the reported accidents were described in order to get an overview of the cases investigated. It was possible that environmental conditions caused or contributed to the accidents:

 Preliminary to the actual data analysis, these results indicated a great part of the accidents occurred under rather normal circumstances. Fig.1 gives an overview of the types of vessels involved in the cases studied.

　

 BRM Related Aspects and BRM Clues; Table 1 gives the average scores (1 = no role, 2 = some role, 3 = big role) for the BRM related aspects as scored from the accidents reports.

　

 The results in Table 1 indicates that according to the assessors especially teamwork on the bridge and quality of leadership played a relatively big role in the cases studied.

　

　

　

　

　

 Table 2 gives the descriptive statistics for the number of BRM clues per page. Thus the relative importance of the various clues, independent of the number of report pages, can be determined. For the statistic back-ground refer to [19]. This table shows that the three most prevalent BRM clues are: Improper conning/lookout; Violation of rules; and Breakdown in communication.

　

 Furthermore the relationship of BRM related factors and clues with other variables were investigated and the inter-correlations of the BRM clues and the BRM related aspects. The results of these are out of the scope of this paper, but can be found in the original report [19].

　

3.4 Comparison with students' registers

　

 To test the influence of having been trained in BRM aspects, the three educational facilities that offer BRM training were asked to check their registers. Only two persons involved in the accidents were known to have followed the study Ship Management, which includes BRM training. The other participants were not registered. This does not necessarily mean that the lack of BRM training caused the accidents. To sustain such a conclusion more cases need to be studied (increase the sample size). Also, more information on the educational backgrounds of the persons involved (e.g. other types of BRM Courses) and on the general Dutch population of bridge officers is needed. Hence, no firm conclusion can be drawn about the relation between a BRM training and the accidents investigated due to a lack in the registration of BRM certificates.

　

4.1 Introduction

　

 The second phase of this study was a simulator-based comparison of 29 Officers of the Watch with and without a BRM certificate. In this part of the study the following questions had to be answered:

　

4.2 Methodology

　

 Design; In order to make a correct comparison of the groups of Officers of the watch, with or without BRM certificates, it was decided to make a direct comparison of both groups under equal conditions. This means that each subject should occur under the same circumstances, and should be assessed in an equal manner. In order to achieve this, a simulator-based experiment was designed, which contained the following elements:

　

 A Standardized Scenario; All the participants had to sail the same scenario. The original plan of using a scenario, based on one of the accidents analyzed in the Content Analysis, was rejected. The main reason for this was that most accidents occurred during moderate weather in open sea and with low traffic intensity. These conditions would lead to a scenario, which does not create sufficient information within the limited time frame (2 hours) available for the assessment procedure due to the low activity level.

　

 Therefore it was decided to create a specific scenario for this assessment, taking into account the results of the Content Analysis, but with sufficient "clues" for a reliable assessment. The next demand was that the scenario should be equal for each candidate: the number of potential solutions for a certain situation should be limited, in order to keep all assessments comparable.

　

 This resulted in a scenario during which a small tanker approaches a harbour in moderate weather conditions at dawn. This tanker later receives a change of orders due to a blockage of the harbour. Thereafter, the Officer of the Watch has to reschedule the voyage based on the information he received from the agent. It was anticipated that during this process sufficient activities, related to BRM aspects, would occur.

　

 A Full Mission Bridge Simulator of MSR was used for the execution of the scenario. The bridge crew consisted of an Officer of the Watch, a helmsman and a Maritime officer (Maroff). The helmsman and the Maroff were staff of MSR. The participants were acting as Officers of the Watch. During the exercise experienced assessors observed and assessed the scores for the performance of the participants.

　

 A Standardized Scoring System; Standardized assessment forms for scoring the performance of the participants with regard to eight BRM clues and an overall score were used. The assessment was based on the results found in the Content Analysis. This means that there was a simulation scenario, which applies for BRM skills measured by the score on the eight BRM clues as described in Table 2.

　

 The assessors gave an "overall score" about the quality of the performance of the participants on the exercise. The bridge team member also gave scores as observed by him from his point of view with respect to the "quality of the bridge management", and whether the candidate gave enough "information" about the voyage preparation.

　

 Finally, the participants themselves gave their opinion about their own performance, with respect to interfering rules from the point of view of the exercise; i.e. whether the candidate was acquainted with the term Message Markers, whether he made a second voyage plan on paper and whether he asked for the missing chart of Lake Maury.

　

 Standardized Procedures; Each subject of the study should face the same conditions during the assessment. In order to secure this, detailed procedures were agreed upon. These procedures are summarised below:

　

 None of the research team was allowed to know beforehand, whether a candidate already had followed a BRM Course or not. Otherwise they would not have been un-biased, which would have influenced the scores. Any communication about the candidate's background was discouraged. Only during the debriefing was this information communicated. To prevent information transfer between the participants about the exercise, the participants were asked not to communicate with each other about the exercise. The suggestion was made that there were more scenarios available. The total duration time involved in the assessment exercise of one candidate lasted 2 hours.

　

 Research Population; A total of 35 Officers of the watch from different shipping companies (in order to establish a representative sample) sailed the scenario. During two days, 6 participants analyzed the scenario as a try-out of the entire scenario to fine-tune the assessment form, train the assessors, and optimise the testing procedures. The other 29 participants performed the exercise as part of the actual assessment. The ages ranged from 24 to 56 years, and their educational backgrounds varied accordingly. Only officers who possessed a Dutch maritime education attended the assessment.

　

4.3 Validation of the methodology

　

 The methodology was validated during two days of testing. During these days the following was checked:

 After these two days the methodology was considered correct for this study.

　

4.4 Reliability

　

 To check the inter-rater reliability (reliability in scoring between the assessors), the correlation between the assessor pairs, who each judged all the participants, with respect to the BRM clues was determined. This correlation amounted to 0.87 (P<0.01) on average for all the BRM clue scores. The relationship between the assessor pairs on the BRM final score resulted in a correlation of 0.93 (P<0.011). The assessors scored in high accordance with each other; not meaning that they scored exactly the same. The conclusion is that the correlation between the assessors was satisfactory, which implied that further analyses could be done on the assessor averages.

　

4.5 Results

　

 Of the 29 participants, 5 persons did not fit in the lest-group as they did not sail on conventional ships, and their results were left out of the analyses. This resulted finally in a test group of 24 persons of which 14 did not attend a BRM Course and 10 did at MSR.

　

 The data obtained from the assessment was analyzed in several ways:

・Score of BRM related Aspects;

・Overall score by assessors and bridge team;

・Score of BRM clues.

　

 The score of BRM related aspect was based on a check of three typical BRM related actions: A check whether the participant was familiar with the term "Message Marker", if they made a second voyage plan after the "change of order", and if they discovered that one of the charts of the area was missing. This resulted in the Table 3.

　

 Table 3 shows that 9 participants were familiar with the term Message Markers. It was interesting to sec whether or not these 9 participants have attended the BRM Course: I did not and 8 did.

　

 The second source of information are the scores given by the assessors and the bridge team member on the performance of the candidate related to BRM aspects. The assessors gave an overall score about the quality of the bridge management and the bridge team member gave a score for informing the bridge team (Table 4). Also here there was a tendency that participants with a BRM certificate performed better than participants without the BRM certificate.

　

　

　

 Summarized it can be stated that there was a difference in performance between the participants with a BRM certificate and the participants without a BRM certificate regarding BRM related aspects. The participants with a BRM certificate performed better on the variables "quality of bridge management", and "overall score".

　

 The third source of information are the BRM-clues scored during the scenarios. Below the scores on the violations of BRM clues are discussed (Table 5). The assessors counted the number of violations the participants made during the exercise on the different BRM clues.

　

 The number of violations was the highest in the BRM clues "breakdown in communication", in "improper conning/lookout" and in "uncertainty/confusion". From the results of the Content analysis we saw that the highest prevalence was in "improper conning/lookout", "violation of rules" and "breakdown in communication", and the lowest in "uncertainty".

　

　

　

 Table 5 shows that the average score of the number of violations was higher for participants without the BRM certificate, then for participants with the BRM certificate. In this case a higher score means more violations. The expectation was that participants with a BRM certificate would have lesser violations than participants without this certificate. This expectation was true for most of the clues. The clues "ambiguity" and "violations of rules" form an exception. On the clue "ambiguity" the number of violations was so low that the higher score for the participants with the BRM certificate was not significant. It was probable that so few violations were made that one violation had too big an impact. The participants with the BRM certificate score also higher on "violations of rules"; this means more violations. An explanation for this could be that the participants with the certificate feel more secure and hence take more "calculated" risks.

　

 Both groups scores were high on communication, meaning that frequently a breakdown in communication occurred during the exercise.

　

 It should be noted that the results have to be interpreted carefully as the sample size was small, with a bigger sample size, the results can differ, and probably more of the results will be significant. Hence it can only be stated that there are the following tendencies:

 These tendencies were tested by statistical techniques [19].