Playing With Figures

in City Rail Link by Jon C — June 1, 2011 at 7:40 pm | 9 comments

One of the most astonishing things I have noticed since running this blog has been the indecent playing around with projections, often using the latest fashionable modelling methods.
I have called it before a throw of the dice, inventing a number, doubling it and making it sound good.

The RWC 2011 is a good example of the way figures are made up.

Auckland’s official RWC2011 site opens with the unattributed bold statement:

“Rugby World Cup 2011 has the potential to generate $267 million of direct economic benefits for the Auckland region.”

Not doubt some economic research modelling company was paid a lot of money to come up with a good story that makes an impressive quote. How on earth can they get away with just throwing out this projection when they don’t even know yet how many tickets will sell (only half to date, 100 days out).

This spreadsheet nonsense gets serious when it means such playing with figures has cost us the CBD Rail Link.

One of the key elements used to torpedo Auckland’s CBD Rail Link Case was playing with the projections of how it would grow Auckland’s rail traffic - and the impact in the CBD.

Sorry to use the tired clique that build it and they come but with rail, that has been proven in recent years.
It would have been difficult to have correctly projected just how fast rail patronage would have grown over the last year but if courageous leaders had not pushed for an improvement of the rail, we would never have found out how when you build it, people flock to it.

Treasury officials closeted in the unreal world of Wellington’s government department world have never been stuck outside the Britomart tunnel for 10 minutes waiting for a berth or struggled to get on a crowded train in recent months. They may use the odd Wellington train but that’s a different story in a vastly geographically and culturally different city.

Yet they and the Ministry of Transport review had to crunch the numbers and using modelling got the bullseyes that suited the Minister who had long ago declared the case seemed over blown in its claims.

When it got to the level of the team trying to work out future costs of parking in the CBD to justify we don;t need more rail is insulting and laughable.

And in one scenario, Auckland Transport appears to have thrown in the impact of the planned Parnell heritage stop - but because the funding had not been finalised at that stage, the review team decided not to take it into account. It’s unlikely it would have really made much of a difference.

We can conclude the modelling helped:

• The review team fiddle like a dodgy tax accountant to get the figures that satisfied their end goal of keeping their ministers (Transport and Finance)  happy - both of whom publicly declared no love for the rail link.
• Or expose total incompetence by Auckland’s trusted Transport and Council staff in which case,. because their projections are so wrong, they should be sent to Siberia.

I know which one I would bet on.

Mike Lee’s blog noted last year his fears National was doing this again to Auckland - and republished a 1954 Herald cartoon which lampoons the scuttling by the National government of a previous plan to build an underground rail link through Auckland. The bound and gagged figure depicted Auckland Mayor J.H. Luxford. (from ‘Decently and in Order’ by Dr Graham Bush). The 2 casks of dynamite as “procrastination” and “dissension” demolishing Auckland’s aspirations.

Papers released by the Ministry of Transport today review the thinking they had over the months of the review and what they feed the Transport Minister as they built upa  case to question the claims that the extended tunnel would boost Auckland economically and increase the numbers using the services.

Here is what they said over that time about the patronage figures in the post-electrification era - which they insist is a very key point to saying yes or no to the Business Case.

In January the review team said:
The do minimum

“You will recall that the APB&B study team provided two memos last year with patronage forecast numbers for the post-electrification do minimum and CBD rail tunnel scenarios. In these memos, the number of passengers inbound to Britomart during the AM peak is capped at the 2024 level of 7,283.

We note that this is well short of the expected post-electrification fleet capacity for inbound AM peak trips of 25,149. This means that patronage in the Business Case do minimum scenario is not constrained by Britomart.
We have asked Auckland Transport and the APB&B study team to explain this difference and they have identified two reasons.

Firstly, they anticipate a ‘walk-up’ constraint at Britomart, i.e. they have assumed that use of Britomart will be limited by the amount of job growth within a five minute walk of the station. Secondly, they state that trains on the Western and Southern iLines are at capacity approaching Newmarket and Grafton Station and as more passengers alight at these
stations than board, there is a low number of passengers travelling to Britomart. These assumptions appear inconsistent with the patronage projections provided to us as part of the EMU project and which drove the size of the EMU fleet.
We will work with all parties to investigate these assumptions. Auckland Transport and the APB&B study team are providing us with the outputs from the modelling work and data from station surveys monitoring the number of passengers alighting and boarding at various stations on the network.”

Assumptions about employment growth in the CBD were also considered a key to the Business Case for the tunnel. The modelling assumes that the CBD will grow faster than the rest of the region.
However, in January review team told the Minister:

“CBD employment growth for the last 15 years appears to follow the overall Auckland trend and the rate of growth is materially slower than what is required to meet the levels assumed in the transport modelling. For example, if current levels continue, CBD employment by 2041 could be 15,000 – 25,000 jobs lower than the 122,000 jobs assumed in the modelling.

This presents a risk to all projects that seek to meet forecast growth in transport demand into the CBD. Of course, a range of factors could change the trend including, potentially the CBD rail tunnel. We are working with the Auckland Council to investigate this issue.”

In February the suits had this to say about patronage projections:

“The Working Group has yet to confirm that the modelling assumptions used in the
Business Case are appropriate, so you should note that figures used in the final Working Group report may change.
The transport model used to assess public transport projects in Auckland does not have a mechanism for incorporating rail or bus capacity constraints, for a given service frequency, as part of its outputs. Model outputs therefore assume unlimited train capacity. Consequently, train capacity constraints and their impacts need to be assessed outside of the model.
You will recall that the modelling used in the Business Case assumed that all morning peak patronage growth in the DMEN stopped in 2024 at 26,420 passengers across the network and 7,283 passengers into Britomart.
This is a critical assumption as it impacts on project timing and the benefits from the project.
The model runs that formed the basis for the economic appraisal in the Business Case
show the following outputs.
Assuming no capacity constraints, the DMEN continues to attract patronage until
2041 and would grow from around 20,000 morning peak period passengers across
the network in 2016 to 39,862 passengers in 2041.
The CBD rail tunnel will increase morning peak period patronage across the network
to 47,617 rail passengers in 2041.(This model output was not used in the Business Case due to the assumption that patronage does not grow beyond 26,420)
Around 60 percent of the increase in rail passengers from the project comes from a
reduction in bus passengers, with the remainder coming from vehicle passengers
and active modes. Assuming no capacity constraints, 10,712 morning peak period passengers arrive at Britomart in the DMEN in 2041. (NB This model output was not used in the Business Case due to the assumption that patronage does not grow)
As noted, the above figures do not take account of capacity constraints in the DMEN and therefore overstate the performance of the DMEN and understate the patronage increase that can be attributed to the project. They do, however, indicate the scale of change caused by the increased accessibility to the CBD and higher frequencies that the tunnel project enables.
A key task for the Working Group is to agree a methodology for identifying when capacity constraints take effect in the DMEN. This will then be used to review the Business Case assumption that constraints will stop patronage growth from 2024. Based on input from a technical consultant, we have tentatively agreed a methodology with the technical
members of the Working Group and are currently working to refine the approach.”

Sections from the modelling plots for the unconstrained DMEN and the rail tunnel options for 2041 are included, in the report to suggest that,”although the CBD rail tunnel considerably increases patronage into the core CBD, a large proportion of this is a redistribution of existing trips away from stations - such as Parnell, Grafton and perhaps Newmarket - on the CBD fringe” and they added: “We will investigate this pattern further.”

It is here that noted that the planned Parnell Station has a major effect on patronage in the do minimum, but has no funding committed.

“We have requested a modelling run without this station. The proportion of total trips into the CBD in the DMEN is lower in the future than we had assumed. Britomart currently accounts for 66 percent of inbound alightings. According to the modelling, Britomart’s share of inbound alightings reduces as Grafton, Newmarket and particularly Parnell (which we had not accounted for) grow in significance.”

The group wanted an independent review of the wider economic benefits methodology used in the original Business Case. It  commissioned the UK branch of Steer Davies Gleave (SDG) to undertake this review, adding that “SDG are currently undertaking research work for the NZTA on a methodology to extend the range of wider economic benefits captured in the Economic Evaluation Manual. ”

“Auckland Council has provided the Working Group with an internal memo responding to concerns we raised over CBD employment assumptions and their use in the Business Case and the Regional Land Transport Strategy. The memo raises a number of concerns over the wider economic benefits methodology used in the Business Case.

“It also provides evidence to suggest that a large proportion of the employment growth attributed to the tunnel in the Business Case could be explained by relatively simple factors, such as differences in the geographic area and types of employees included in employment forecasts. We have included the Auckland Council memo in the material provided to SDG for review.

“Auckland Council officers have emphasized to us that the findings in the memo do not
necessarily negate the benefits included in the Business Case, or mean the methodology is inappropriate.”

During this period, the review also got stuck into its obsession about benefit cost ratios, which seem an issue for Rail but not for Motorways although it cheekily suggests that its’ the rail supporters who are not fairly using the (awful business clique ahead) level playing field.
The review team told the Minister:

“We have previously advised you that the BCR for the project, using the standard appraisal methodology, is 1.13. The Treasury has identified, and recent information from the Study Team consultants shows, that this is not the case. The BCR of 1.13 is the government BCR, which nets fare revenue from the overall costs. The national BCR, which is comparable to State highway national BCRs and treats fare revenue as neutral. We note that national BCRs are not referred to in the Business Case.
Additional information from the Study Team shows that the project is sensitive to the
reduction in patronage that occurs as a result of increases in fare levels. We have yet to review the material, but the impacts on BCR appear to be quite large.”

On March 11 the team asked for a extension of time, saying it needed to reassess the claimed benefits and had hired Steer davis Gleave and awaited that report.

The “technical issues” outstanding were:

By March 21 the review reported its conclusions on these patronage issues:

• “At present, we expect that the project will increase total rail network trips in the 2041 morning peak period by up to 9,688, and remove up to 5,000 car trips. We expect that this would include around 7,700 additional rail trips into the CBD.
• With regard to potential capacity constraints, our assessment currently shows that demand on the Western Line will exceed the former Auckland Regional Transport Agency’s target desirable loading limit around 2027, but only on inbound trips in the morning peak hour. The Southern Line and Eastern Line do not reach the target limit. However, limited seating capacity may be an issue, with some passengers standing for up to 30 minutes on the Southern Line. Further work is needed in this area.”

You can read their reports to Steven Joyce here.

Good on the Ministry for at least releasing such material without the need for us having to make an Official information request.But it did note that word was getting out and it had already started receiving such requests so it was better to put the material available when the announcement is made. Only a few lines seem to be omitted in the papers - this was when they were discussing  three main options for completing the Review.

1 Cease work and request Auckland Transport and Auckland Council commission a new business case addressing the issues we have identified. This is not recommended as we think the evidence needs to be set out in a final report, and a new business case may not necessarily resolve the key issues.

2. Complete the review by the end of April. This option is achievable, but is likely to lead to stronger disagreement over the content of the report and presents some risks to the completion of all advice. (Blanked out) xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

3 Complete the review by the end of May. This option would provide time to moderate some areas of disagreement and improve the robustness of the analysis.6. To meet the timeframes in options two and three above, we seek an urgent decision on your preferred option. We retain our preference for an extension to the end of May, but recognise there are difficulties with this. If the issue is with the length of the extension, even an additional week or two would prove useful to complete the work.

Let’s hope this is at least the way their business is conducted in the future - open and available for proper debate.

You can read all their documents here

Full coverage:
Ministry of Transport and Treasury report

Full commissioned Auckland Council Auckland Transport report

Minister, Len Brown and Greens react

Ministry of Transport explain their decision
Auckland must fight back: And more reaction

Mike Lee calls it a major setback

Tags: CBD rail link CBD rail loop ministry of transport