Trains: CA: Panel finds flaws in high-speed-rail forecasts

From Palo Alto Online: Panel finds flaws in high-speed-rail forecasts

The California agency charged with building America's first high-speed-rail system has been using a flawed forecasting model to predict ridership for the proposed system, a peer-review panel concluded in a report that largely confirms previous criticism from transportation experts and rail watchdogs.

The five-member panel, which consists of professors and transportation experts, found that the ridership model, while "generally well founded and implemented," suffers from a series of major flaws. These include insufficient consideration of socioeconomic factors; a bias in the survey data used as a basis for the model; and a failure to distinguish between short and long trips when calculating the impact of schedule delays.

The highly technical report, which was released in late July and covers the panel's findings and recommendations during its January to March review period, confirms earlier findings from the UC Berkeley Institute of Transportation Studies and from the Palo Alto-based watchdog group Californians Advocating Responsible Rail Design (CARRD). Both groups had criticized the methodology used by the consulting firm Cambridge Systematics and argued that the California High-Speed Rail Authority's estimates of the number of people who would ride the rail system are too flawed to be used for setting policy.

The panel, which reports to rail authority CEO Roelof Van Ark, is chaired by Frank Koppelman, professor emeritus of civil engineering at Northwestern University. It also includes Kay W. Axhausen, a professor at the Institute for Transport Planning and Systems in Zurich, Switzerland; Billy Charlton from the San Francisco County Transportation Authority; Eric Miller, a professor of civil engineering at the University of Toronto; and Kenneth A. Small, a professor emeritus in economics at University of California, Irvine.

The panel calls Cambridge's ridership model "ambitious" and representing a "significant improvement in practice in several respects." But the report also notes that "there are important technical deficiencies in the model and the documentation thereof." It encourages the rail authority to lower its projections.

"The Panel has significant concerns about the model formulation, primarily with respect to specification that should have been addressed during previous work," the report states. "Pending improvements to the model, we recommend that any use of the model include some steps to make the demand forecasts more conservative, especially in forecasts for financial (investment and risk) analysis."

A rail official, meanwhile, said that the report highlighted the complex nature of forecasting.

"Essentially, the report says that this is the most ambitious and most transparent modeling exercise to have occurred in this realm and, when refined, will represent best practices for this kind of forecasting in North America. The panel asserts no bias or improper practices," spokesperson Rachel Wall said in an email.

"What the panel expresses desire for is more documentation and more testing, both of which have been provided to the panel in the time between March and today, and which will be reflected in the forthcoming reports from the panel," she said.

One flaw that the panel identified involved the ridership model's treatment of out-of-vehicle travel time, particularly the time passengers have to wait when trains are delayed. The report states that the assumptions used in the Cambridge model to calculate the "constraint on out-of-vehicle travel time" are valid only for urban trips with small headways (that is, the distance and time between trains). The report cites a study showing the passengers' behavior is much different in the "intercity market." In other words, passengers are much more likely to stomach scheduling delays if they're preparing for long trips out of town as opposed to jaunts from one neighborhood to another.

The report calls the ridership model's use of this constraint "unjustified."

The panel also found "several instances of incomplete or outdated information in the documentation," according to the report. This includes insufficient discussion of such factors as fare levels, highway and airport congestion, train frequency and analysis of how the proposed train system would impact other modes of transportation, including airlines and intercity bus services.

The report is particularly critical of the survey used by Cambridge to get data for the ridership model. The company used a technique called "choice based sampling" which targets and, as a result, over-represents a specific subset of the population (in this case airline and train travelers). The firm conducted surveys in 2005 at airports, rail stations and over the phone. This included on-board surveys on Altamont Commuter Express trains, telephone interviews of Amtrak passengers and surveys of passengers at six California airports.

The report notes that while choice-based sampling is useful for making sure "enough respondents were found to choose each of the major modes," the technique is also "known to bias estimation results unless the estimation procedure is modified to take account of this sampling."

"The method used by CSI, which was believed to be correct at the time of model estimation, has since been shown to be incorrect and a new procedure has been developed which is correct," the report states. "Future estimation work should take advantage of this new knowledge."

The panel released its findings at a time when the rail system continues to weather criticism and financial uncertainty. State Sens. Joe Simitian and Alan Lowenthal have consistently criticized the rail authority's ridership forecasts and business plans over the past two years. Other critics, including CARRD and a group of Silicon Valley economics and business executives, issued reports criticizing the rail authority's business plan and its assumptions about federal grants and private investments.

Rail authority officials could not be reached for comment Monday afternoon.