Scientific research versus The Forum Report

In March of 2016, The Forum on Education Abroad, which is the standards setting body for the study abroad industry, issued the following paper, Insurance Claims Data and Mortality Rate for College Students Studying Abroad.

When Ros and I read The Forum Report, to say we were disappointed does not begin to capture our feelings. For several years, we’d been asking both the study abroad industry and state and federal legislators for transparent and comprehensive safety data. This was not the product we’d ever imagined. For one thing, there were so many issues with the paper’s research methods*. But even more, The Forum Report began with its conclusion, and Ros and I had a bad feeling about what purpose this conclusion might serve.

Key Finding: College students studying abroad are less likely to die than college students studying on campuses in the U.S.

The Forum Report’s “key finding” is based on three data sources, all of them flawed, though in different ways:

The Critical Incident Database (CID):

  • CID is The Forum’s own tool
  • CID was launched in 2012, with 2014 as its first reporting year
  • CID asks broad categories of questions (this is a good thing)
  • CID participation is voluntary (this results in something called self selection bias, which is problematic)
  • The Forum guarantees confidentiality to CID participants (this makes independent review of claims impossible, which is problematic)
  • During 2014, with Forum membership around 650 schools, 38 responded to CID (~6% participation)
  • In 2014, CID reported no deaths

The Turner Study:

  • The Turner Study is a pilot study looking at student deaths on campus
  • A pilot study’s purpose is to test feasibility for a larger study (to date, no further study)
  • Turner’s objective was to look at leading causes and rates of mortality for students at a sample of US institutions of higher education
  • Surveys were sent to 1154 American College Health Association member institutions; 166 schools returned the survey (~14%)
  • Data arrived in widely varying formats
  • Turner found an all-cause mortality rate** of 22.4

Insurance Claims Data:

  • Two large insurers of higher education, Cultural Insurance International (CISI) and HTH Worldwide (HTH), volunteered their data to the Forum
  • 2014 claims data reflected 146,898 study abroad students
  • In general, using insurance data for research intended to inform public policy is problematic for several reasons:
    • Actuarial data is geared to answer one question: How many policies must be sold to offset claims?
    • Not everyone who can make a claim does—a claim may be made through another insurer, or not at all
    • Actuarial data cannot represent populations outside the insurance pool
  • During 2014, four student deaths generated claims

These four insurance company deaths formed the basis for The Forum Report’s key finding. Using Open Doors*** data to estimate that same year’s number of students studying abroad, they calculated an all-cause mortality rate for study abroad of 13.5. The Forum then compared this number with Turner’s home campus death rate of 22.4. Thus, their claim of study abroad as safer.

Aside from fundamental pitfalls born of comparing data that is “apples and oranges” (as CID, Turner, and the insurance data are), The Forum Report fails as a scientific study for the following reasons:

  • No named author(s)
  • No abstract
  • No hypothesis
  • No review of the literature
  • No definition of terms
    • For example, how is “death” defined? Who is included? Who is excluded?
    • The definition of death can explain what we believe to be an undercount
  • A poorly written methods section****
    • No expert reader we consulted can precisely say what was done
    • For this reason, experts told us no scientist could replicate this study, which means no possibility of peer review*****
  • As listed above, there are problems with all three data sources
    • Voluntary, with low participation rates
    • Confidentiality for all
    • Data represents a limited scope of time
      • Was 2014 a representative year?
    • The authors did not question 13.5 as a reasonable all-cause mortality rate
    • Experts familiar with youth mortality research suggest this number is “implausibly low”
    • Neutrality is foundational to scientific research, yet bias permeates the entire Forum Report
      • “Key Finding” appears first
        • A scientific research paper places findings and opinions in the conclusion section
      • Potential legislation is cited by The Forum as the reason for this paper, as well as “reassurances” to the industry and future study abroad families
    • The Forum Report is published in a Forum publication, which again means no peer review

For these reasons, ProtectStudentsAbroad has concluded, though The Forum Report looks sort of like a research study, it is not. It is an “industry report”, written by the industry for the industry.

Industry reports are common to for-profit corporations, where business success depends upon budgets, quotas, and organizational control derived by powering through narratives. Industry reports are not intended to protect human safety. They are meant to drive business.

Indeed, four months after The Forum released its report, a University of Wisconsin student died. Beau Solomon was murdered while studying abroad with third party program John Cabot University in Rome.

In the aftermath of Solomon’s death, news headlines read, Despite Beau Solomon Death, Few Students Die on Study Abroad (Newsweek) and Despite Rome murder, study finds students are less likely to die abroad (Christian Science Monitor). Articles quoted study abroad senior leadership, including Forum president, Brian Whalen, and The Forum’s key finding was repeated like a tagline.

Aside from the flagrant insensitivity of such statements, The Forum Report’s underlying question of campus safety versus study abroad safety isn’t a line of inquiry bereaved families have ever pursued. Risks faced by students on campus versus students during study abroad are different; we get that. It is disappointing to us that The Forum Report did nothing to identify specific risks to study abroad students.

And so, in the aftermath of The Forum Report, using student deaths and injuries available via Internet, we are reviewing this information in an increasingly systematic way. While we do not claim PSA’s data as quantitative, or even scientific, we believe it is qualitative and notable—an ad hoc dataset compiled using available resources, including the web, volunteer experts, and the labor and knowledge of bereaved families. From our review, we would like to share a few observations.

While The Forum Report based their key finding off four student deaths, during that same calendar year we’ve identified fourteen. The following are categories of students we suspect were not included in The Forum Report:

  • High school students
  • Graduate students
  • Students who are foreign-born, but studying in the US, including US study abroad
  • Students studying abroad and not receiving credit, including internships, mission trips, religious trips, and performance trips
  • Students whose death occurred during “free-time”, including pre-, intra-, and post-program travel
  • Students whose programs were not insured by the two companies who participated in The Forum Report
  • Students attending non-Forum schools

Among our dead, injury appears to lead causes of death, specifically motor vehicle accidents, drowning and falls. Furthermore, heightened risk appears to be linked to specific demographics. For example, gender appears to be linked with manner of death. Also, certain countries of study appear to be linked with manner of death. In light of these observations, ProtectStudentsAbroad issues two recommendations.

To the study abroad industry: If fatality prevention is your profession’s premier goal, then embrace principles of transparency. To do less goes against American higher education’s historic ideals and operates as a betrayal of trust for students and their families.

To students and parents: The American publics’ lack of scientific literacy has allowed all sorts of industries, including the study abroad industry, to market sloppy science as “truth”. Until such time that the study abroad industry steps up and into the 21st century of evidence-based education******, we implore you to vigorously interrogate all claims made by programs. Ask difficult questions. Expect real answers.

The stakes are high. This work is hard. The cause is good.

~

*Research methods refers to the means by which data is gathered, and then statistically treated and analyzed.

**All-cause mortality is a measure used in research as an indicator of the safety or hazard of an intervention. For deaths, it is customary to use rates per 100,000, over a one-year period. In Turner’s study, he was measuring all deaths that occurred in his specified population, regardless the cause.

***Open Doors is a “comprehensive information resource on international students and scholars studying or teaching at higher education institutions in the U.S., and U.S. students studying abroad for academic credit at their home colleges or universities”. Under the umbrella of the Institute of International Education (IIE) and supported by a grant from the U.S. State Department, institutions of higher education voluntarily complete yearly surveys, from which an annual Open Doors publication is produced. Open Doors express purpose is to promote and grow international education; it does not collect safety data.

****The methods section of a research paper follows an exhaustive review of existing research on the topic. The methods section describes procedures used to collect data, and also details statistical treatment of that data. Methods are the nuts and bolts of a research project. When scientific methods are sound, the findings can illuminate, without bias, otherwise indecipherable connections between outcomes and a whole host of variables.

*****The peer review process begins with a research paper being submitted to editors of scholarly journals. If interested, editors refer the paper to experts in the field. Using discreet criteria, experts read and comment. Peer reviewers pay particular attention to the methods and results section, since these are most complicated and technically difficult. Methods sections should be written clearly enough that another researcher could pick up the paper and replicate the project. When well done, the peer review process serves to screen out poor research, screen in good, and in this way promote “best practices” for the entire field.

******Evidence-based education considers all aspects of education, from policy-making to classroom implementation, and applies evidence obtained from scientific research. Such research is not meant as a substitution for professional experience. Rather, it is a critical adjunct, meant to deepen and broaden fact-based knowledge.

By | 2017-06-07T02:53:04+00:00 March 13th, 2017|To begin again|Comments Off on Scientific research versus The Forum Report