“Best UFO Cases” by Isaac Koi
PART 23: Quantitative criteria : Ballester/MUFON index
Introduction
The most detailed attempt made thus far to put forward a method for the quantitative assessment of UFO reports is probably the Ballester-Guasp quantification method.
In 1981, Spanish ufologist Vicente-Juan Ballester-Olmos and physicist Miguel Guasp published a book entitled “Los OVNIS y la Ciencia” (UFOs and Science). That book included a review the various systems that had been proposed for evaluating UFO reports. It also included their own proposal for such a system (see Footnote 23.01). An expanded treatment of that proposal was published in English in 1988 (see Footnote 23.02).
An article by Jerold Johnson containing a discussion of a revised version of that system was subsequently published in the 1995 edition of the MUFON training manual (see Footnote 23.03). That article have appeared on various websites (e.g. see Footnote 23.06) has also been translated into French (see Footnote 23.04).
MUFON training manual
Jerold Johnson’s article in the MUFON training manual (see Footnote 23.03) set out details of “the evaluation procedure applied to reports at headquarters level prior to their being entered into the computer file” and stated that since 1992 “reports processed into the MUFON files have been given a numeric evaluation” based on this system.
That article sets out the relevant factors at some length and mentions in the concluding section that Field Investigators should run through their own reports to “check if all the questions required for the evaluation have been answered somewhere in the report”. It states that “the MUFON report forms were designed before this formula was recognized as worthwhile and there are not specific blanks on the forms for all the required data, especially in the ‘how investigated, for how long’ category, so some Field Investigator ‘write-ins’ are necessary”.
The article proposes the calculation of an overall “score” for the report derived by multiplying the following three values (each of which ranges from zero to one) together, “representing the degree of certainty that the report indeed represents an anomalous event that happened as recorded”.
Factor 1 : “The volume and quality of the data recorded, based on the methods employed and the time spent investigating the case”.
Factor 2 : “The inherent abnormality or ‘strangeness’ of the event, making it unlikely to have a natural or conventional explanation”.
Factor 3 : “The credibility of the report, based on the reliability, maturity, and circumstances of the witnesses interviewed”.
The values to be assigned for each of these factors was set out in the article, as summarised below.
Factor 1 : “Information Quality Index”
This factor “indicates the ‘strength’ that a report has for analysis based on how it was acquired”.
This factor is similar to the [S]ource and [V]isit factors in Jacques Vallee’s SVP criteria (see PART 21: Quantitative criteria : Vallee’s SVP ratings) and the Investigation Levels proposed by Jenny Randles (see the relevant discussion towards the end of PART 20: Quantitative criteria : Hynek – Strangeness and Probability).
|
Source |
Direct Investigation |
At the site |
>= 2 hours |
1.0 |
|
< 2hours |
0.9 |
|||
|
Interview Person to person |
>= 1 hour |
0.9 |
||
|
< hour |
0.8 |
|||
|
By Telephone |
>= 1/2 hour |
0.7 |
||
|
< 1/2 hour |
0.6 |
|||
|
Indirect Investigation |
Questionnaire with follow-up |
Extensive |
0.7 |
|
|
Brief |
0.6 |
|||
|
Letter with follow-up |
Extensive |
0.6 |
||
|
Brief |
0.5 |
|||
|
Other Investigation |
Questionnaire no follow-up |
0.6 |
||
|
Letter/Narrative no follow-up |
>= 1 page |
0.4 |
||
|
< 1 page |
0.3 |
|||
|
Newspaper |
>= 500 words |
0.2 |
||
|
< 500 words |
0.1 |
|||
|
Radio/TV |
0.1 |
|||
|
Witness Relative |
0.1 |
|||
|
Verbal/Rumor/Unknown |
0.0 |
|||
While such values can probably be assigned fairly easily by the person that conducted the relevant investigation, a person faced with merely an account in a UFO book would have often have difficulty in assigning any value other than 0.0 (i.e. the value where the source is “unknown”). Few accounts of UFO reports in books and magazines indicate the nature and depth of an investigation. Perhaps it is only right that the information quality of such accounts in UFO books are assigned a very low or zero value.
Factor 2 : “Strangeness Index”
This factor indicates “the ‘abnormality’ level of a report compared to normal processes, familiar phenomena and known manufactured objects”.
It is similar to the Strangeness Rating suggested by J Allen Hynek (see PART 20: Quantitative criteria : Hynek – Strangeness and Probability).
The article suggests that “one simply counts up” the number of the following seven features that are “commonly found in sighting reports” and divides by seven:
Strangeness Feature 1 : “Anomalous appearance” (i.e. “shape or dimensions do not correlate with any identifiable flying craft”)
Strangeness Feature 2 : “Anomalous movements” (i.e. the “dynamic characteristics of the observed phenomenon” make it “impossible to receive a logical explanation”)
Strangeness Feature 3 : “Physical-spatial incongruities” (e.g. “disappearances”, “the merging of two objects into one”)
Strangeness Feature 4 : “Technological detection” (“observing and/or recording of the passage of the UFO through calibrated precision instruments”, including radar, telescopes, film or videotape)
Strangeness Feature 5 : “Close encounter” (“within 500 feet”)
Strangeness Feature 6 : “Presence of beings associated with the UFO” (“the association of presumed occupants”)
Strangeness Feature 7 : “Finding of traces or production of effects” (“lasting physical or chemical characteristics or residues left by a UFO after its disappearance, provided that there exists some testimony that the traces or effects were produced by the presence of the UFO”).
Factor 3 : “Reliability Index”
This factor indicates “the witness ‘credibility’ ”.
(I note in passing that some of the factors that relate to the credibility of a UFO report – e.g. a video recording or physical evidence– are covered within this system within the factor on “strangeness”).
The “Reliability Index” is considerably more complicated than the above factors. There are six categories within this parameter and each is assigned a “weight factor”. MUFON’s manual states that a researcher “selects the appropriate number from each category, multiplies it by its ‘weight factor’ and ultimately adds the six results together”.
A similar method of giving different weights to different factors has been described by Claude Poher (see the relevant discussion near the end of PART 20: Quantitative criteria : Hynek – Strangeness and Probability).
Credibility Feature 1 : Number of witnesses (“a sighting is more believable if it has more witnesses”)
0.0 - none or unknown
0.3 - one
0.5 - two
0.7 - three to five; "several"
0.9 - six to ten
1.0 - more than ten
(multiply by weight factor 0.25)
0.0 - not specified
0.3 - students (pre-college)
0.5 - laborers, farmers and housewives
0.6 - university students
0.7 - traders, businessmen, employees and artists
0.9 - technicians, police and pilots
1.0 - university graduates and military personnel
(multiply by weight factor 0.2)
0.0 - unknown
0.3 - friends
0.6 - family relationship; also applies to cases with a single witness
0.8 - professional relationship
1.0 - no relationship
(multiply by weight factor 0.15)
Credibility Feature 4 : Geographic relation between witnesses (“when there are multiple observers, their relative location affects the certainty of the event”).
0.0 - unknown
0.5 - together; also applies when there is a single witness
1.0 - independent (separate)
(multiply by weight factor 0.15)
0.0 - not specified
0.3 - recreational activity (walk, rest, outing, hunting, sport, at home, on vacation, etc.)
0.6 - traveling (moving, by any means)
0.8 - cultural or intellectual activity
1.0 - working (at work or on the way to or from)
(Multiply by weight factor 0.15)
Credibility Feature 6 : Age of the witness (“indicates their degree of maturity and the validity of their testimony, based on their capability”).
0.0 - unknown
0.2 - under 10 years or over 75 years
0.4 - between 10 and 17 years
0.6 - between 18 and 34 years
0.8 between 65 and 74 years
1.0 between 35 and 64 years
(multiply by weight factor 0.1)
It is notable that no references whatsoever are given in the MUFON manual in support of any assertion that the various features actually matter to the credibility of a witness or to support the values assigned. As one French ufologist (Claude Mauge) has commented, “Many values used by various authors are based more on good sense than on ‘scientific’ data” (see Footnote 23.09).
While some of the values suggested above may be justified on the basis of (to use Claude Mauge’s phrase) “good sense”, I question whether (in particular) the values for “activity at the time of the sighting” as a measure of “the opportunity for a hoax” are really justified. For example, is it really justifiable to give a report from someone that reported he was indulging in an activity falling within the category “cultural or intellectual activity” should be given a score for this feature which is nearly three times that from someone that said he was walking at the time of the sighting?
Vicente-Juan Ballester-Olmos himself has commented on the paragraphs above. He stated that my comments are "well taken" and that "we move in a subjective area here but our index is based on criteria supported by years of field work and case analysis experience, not merely an arm-chair elaboration" (see Footnote 23.12). He also mentioned that the values are followed a consideration of "previous work in this particular area of research" and referred to the bibliography for the original paper. He also commented specifically on the activity at the time of sighting, stating "it is tested that the proportion of of hoax cases is larger during a lazy, leisure activity than during a professional one, for instance, this is why we computed it that way".
The overall score (“Certainty Index”)
The “Certainty Index” (i.e. the overall score for a UFO report) is obtainted by multiplying the three factors above.
According to the MUFON manual, this provides a “measure of the overall degree of ‘certainty’ of an anomalous event behind the report” and is “often expressed as a percentage”.
The MUFON manual also suggests that “the Certainty Index might be used as a quick way to order the reports in a catalog from ‘least promising’ to ‘most promising’, while the other three parameters will indicate why each report received the value it has”.
As can be seen from the above summary, calculating the relevant total score involves several multiplications which would be difficult to perform mentally. A calculator would generally be needed.
In 2003, the MUFON Journal published an item by Terry Groff entitled “Online Javascript Certainty Index Calculator useful as an investigating tool” (see Footnote 23.07). That item decribed a free online tool helpfully developed by Terry Groff (which is still available online - see Footnote 23.08) which allows the user to merely click on relevant attributes and then ask the tool to calculate the relevant scores.
Terry Groff’s online tool simplies the relevant calculations considerably. The relevant webpage also obviates the need to remember (or keep at hand) all the relevant guidelines and values.
Actual applications of the Ballester-Guasp method
As mentioned in the introductory sections above, the method above was adopted and promoted by MUFON in the 1995 edition of MUFON’s training manual for its field investigators. Since the MUFON report forms at the time of the publication of that manual did not contain “specific blanks” for all the required data, the manual stated that “write-ins” were necessary. The relevant article in the MUFON training manual states that since 1992 “reports processed into the MUFON files have been given a numeric evaluation”
It is not clear whether the MUFON report forms were ever redesigned to contain the relevant “specific blanks”, nor how common it was (or is) for MUFON field investigators to “write-in” relevant information.
However, one of the volunteers that has contributed to MUFON's database systems has helpfully informed me that "a great number of cases that have been investigated do have this information available" and that it is part of the "final report text field" within MUFON's records. That volunteer commented that: "Unfortunately though it's difficult to do mass comparison across the entire DB to get a sense of the quality of all cases (or the cases that are missing this information) because the data hasn't been abstracted to its own numerical column" (see Footnote 23.10).
The current incarnation of the online tool mentioned above that has been developed by Terry Groff now appears to be integrated with MUFON's Case Management System("CMS") database, since it includes fields for inputting the relevant Case Management System case number (see Footnote 23.11). I am currently seeking confirmation of this.
Unfortunately, the only database of reports available on the MUFON website to members of the public appears merely to contain details of sighting submitted without any evaluation (whether numerical or otherwise). I am currently seeking clarification of the status and availability of the "final report text field" referred to above.
Jacques Vallee has referred to the Ballester-Guasp proposals in the context of a criticism of the fact that other UFO researchers “have rarely bothered” to apply some way “of assigning credibility or ‘weight’ to an oberservation”. Jacques Vallee said that there was to a notable exception to his criticisms, i.e. the “quality index” proposed by Spanish researchers Vicente Juan Ballester-Olmos and Guasp, “but it is so detailed that I have found it difficult to apply in practice” (see Footnote 23.05). Vallee has suggested that it is “important to implement a system that is simple enough to be applied quickly and with enough mnemonic value that it does not require constant reference to a user’s manual or a set of tables”.
Jacques Vallee’s actual experience of the proposed system appears to be inconsistent with the assertion made in MUFON’s manual that “The system gives reproducible numbers when evaluated by different individuals, at different times, as long as they are following the standards as published. The method is relatively ‘quick and easy’ given a calculator and a few tables and definitions extracted from the publications and kept handy as notes”. It is notable that Vallee's remarks were made prior to the developed by Terry Groff of his online calculation tool and the apparent subsequent integration of that tool with MUFON's Case Management System.
One of the points made in the MUFON training manual in relation to this system is valid in relation to any of the quantitative criteria that have been proposed by various researchers : “Avoid the urge to argue with the selection, ordering, or number value assigned to the various factors. The standards must be stable and rigorously adhered to for there to be any usefulness in comparisons of reports using the numbers at different times, by different people, possibly in different countries, but all using the same standardized evaluation procedure”.
Vicente-Juan Ballester-Olmos has mentioned that the Ballester-Guasp proposal was used by Willy Smith "in working with his UNICAT" (see Footnote 23.12).Since Jan Aldrich (a ufologist that has paper copies of the UNICAT records) has stated that UNICAT includes "strangeness values assigned by Hynek", it is currently unclear to me whether UNICAT records include strangeness values under the system proposed by Ballester-Olmos and Guasp in addition to Hynek's strangeness values, or only values under one of these approaches. Due to the range of UFO classification schemes, I am aware of at least one other signficiant database (UFOCAT) has adopted more than one classification scheme in order to provide the maximum amount of information.
Vicente-Juan Ballester-Olmos has also mentioned that German engineer Adolf Schneider "did computer work" about the Ballester-Guasp proposal "but soon left ufology and it was not complete" (see Footnote 23.12). The only mention of Adolf Schneider's name in relation to a UFO database that I have found is a brief reference to him as as a point of contact for a catalogue of 1080 UFO cases with electro-magetic and gravity effects. MUFON-CES was also mentioned in this context. I do not know if that database was in fact developed by Adolf Scheider and/or if it assigned the Ballester-Guasp values to the relevant cases.
Vicente-Juan Ballester-Olmos himself has applied the system in a book he co-authored with Juan A. Fernández entitled "Enciclopedia de los encuentros cercanos con OVNIS" (Plaza & Janés, 1987), where he "analyzed 230 UFO and 355 IFO landing reports in Spain and Portugal".
FOOTNOTES
[23.01] Vicente-Juan Ballester-Olmos and Miguel Guasp “Los OVNIS y la Ciencia” (UFOs and Science), Plaza & Janes, S.A., Barcelona, 1981, 1989 at pages 117-135. [Obtained but not translated]
[23.02] Vicente-Juan Ballester-Olmos with Miguel Guasp, “Standards in the Evaluation of UFO Reports”, in The Spectrum of UFO Research, Mimi Hynek (editor), J. Allen Hynek Center for UFO Studies (Chicago, Illinois), 1988, pages 175-182. [NOT YET OBTAINED]
[23.03] Jerold Johnson, “Ballester-Guasp Evaluation of Completed Reports”, in MUFON Field Investigator´s Manual, Walter H. Andrus, Jr. (editor), Mutual UFO Network, Inc. (Seguin, Texas), February 1995, pages 214-221.
[23.04] French translation of the article referred to in Footnote 23.03 available online at:
http://rr0.org/Documents/Pratique/Ballester-Guasp.html
[23.05] Jacques Vallee, “Confrontations” (1990) at page 218-219 (in the Appendix) of the Ballantine Books paperback edition.
[23.06] Article published, for example, on Terry Groff’s website in 2005 at:
http://web.archive.org/web/20050205000326/http://terrygroff.com/ufotools/eval/eval_calc.html
[23.07] Terry Groff, “Online Javascript Certainty Index Calculator useful as an investigating tool”, MUFON Journal, February 2003, page 12.
[23.08] Terry Groff’s “Online Javascript Certainty Index Calculator” has appeared on various websites in the last few years, including at the bottom of the following webpage:
http://web.archive.org/web/20050205000326/http://terrygroff.com/ufotools/eval/eval_calc.html#calc
[23.09] Email from Claude Mauge to Isaac Koi, 29th May 2007.
[23.10] Email from Dustin Darcy to Isaac Koi,
[23.11] Current version, as at June 2010, of Terry Groff's online tool, is at:
http://mufoncms.com/cgi-bin/bge/bge.pl
[23.12] Email from Vicente-Juan Ballester-Olmos to Isaac Koi,