Written By:
lprent - Date published:
6:10 pm, April 26th, 2012 - 73 comments
Categories: public transport -
Tags: failures, veolia transport
I have and still do deal with quite a lot with systems in my day-job. But this…. I’m aghast at the self-evident stupidity.
A power outage has shut down almost all train services across the Auckland region.
The fault at KiwiRail’s National Train Control in Wellington, which controls Auckland signals and radio control, occurred about 4pm, and it is not know how long train services will be affected.
Apart from the poor quality of journalism, this really raises more questions than it answers.
This report tends to indicate that a single point of failure in Wellington is capable of taking out a major part of the transport system in a city more than 700km away.
Now speaking purely from my old training in earth sciences, there is absolutely no way that ANY system in NZ should rely on a single location as a point of failure. The risk of earthquakes at any one location anywhere in NZ is pretty high as Christchurch showed. But Wellington / Nelson has probably the highest probability of having a major earthquake of any location in the country.
Then there are those hundreds of kilometres between there and here in Auckland. Sure the comms, power lines, and roads in NZ go either side of the volcanic plateau (at least I hope they still do). But it isn’t hard for anyone who has looked at the past history of the Taupo, Rotorua, and even those smaller cones scattered around the region, to easily imagine conditions that would cut links from one side of the north island to the other.
There should be at least ones warm backup system in the north. What country do they think they are in? Frigging Britain? This is New Zealand where you have to build a lot of redundancy into every system. Hell I run this site with several backup systems at varying degrees of warmth.
But FFS. Even if they were stupid enough to just rely on a single geographical point of failure. Then where in the hell were their backup generators?
Perhaps the journo’s should examine why a single point of failure in Wellington should take out an essential service for tens of thousands of people in Auckland.
The server will be getting hardware changes this evening starting at 10pm NZDT.
The site will be off line for some hours.
That’d be nice but I doubt that it would happen as that would require questioning the privatisation agenda.
How does criticising the operations of a state owned entity amount to ‘questioning the privatisation agenda’?
Veolia – that wonderful local Kiwi organisation?
They also look after our waste disposal in Auckland.
Roadside collection up over 100 % in 10 years. Paid for through the bags.
No corresponding reduction in rates though.
Inorganic collections – thing of the past. Take it yourself and pay big bucks.
Don’t you love the service you get when multinationals take over?
Veolia arent the people in Wellington operating the tracks/signals!!!
Yes they are the train operator but the tracks etc are ‘owned’ differently
Someone has jumped to conclusions
I was just taking it off the report in the Herald. So who does run the signal system?
Kiwirail?
Yes
But Veolia run the commuter rail in Auckland? So they are the ones that commuters should scream at for letting them walk home in the rain? They are directly responsible for ensuring the service runs well.
I think that Veolia need the incentive to push Kiwirail.
In much the same way that Vector run the power lines in Auckland, Transpower run the grid, but screaming should be directed at your own power company if they drop you into the dark. I can’t influence either Vector or Transpower as I don’ have a contract with them.but I do with Meridan.
Think of this like the ports, Lynn. Kiwirail own and run the tracks and signals. Veolia run the passenger service on them, contracted by Auckland Transport which pays a lease for access to the tracks and signals.
Auckland Transport is a CCO set up so it nominally reports to Auckland Council as well as the national transport agencies and Ministry.
Buck ‘stops’ with Minister of Transport and Auckland Mayor.
Auckland Transport would only have long term oversight of contract with Veolia- who dont run the tracks/signals. Same goes for usage of tracks from Kiwirail
Not sure how you say buck stops with Auckland Mayor because of above and he only appointed 2 out of 8 directors of Auckland Transport.
You could say the buck stops with Rodney Hide , who appointed Chairman and 6 directors of Auckland Transport.
You could say that but it is not much use in fixing the problem. We probably agree that the government has more power than the council.
Veolia Transport Auckland
Confusing info on Veolia pages between Home and About us – one says that the contract with Auckland Transport to March 2014 and other says June 2016.
Agree with this post. As I said in Open Mike. Unbelievable!
And all a terrorist or other armed and dangerous enemy of the NZ state would need to do would be to strategically lay a bomb in Wellington at a selected time and place, to throw both Auckland and Wellington into disarray.
It wouldnt matter if they did, no one in Auckland bothers to catch the thing anyway.
Hi Rob. Thanks for your “concern” about my well-being.
– signed No One.
Nether accurate, funny or relevant, got back to pulling the wings off flies Rob.
Were you looking in the mirror when you typed this statement tc.
Rob you obviously haven’t caught a train to Swanson from Britomart in the late afternoons – it’s usually standing room only.
No Jilly Bee, I am working in the late afternoon in South Auckland, so I can’t catch the train from Britomart to Swanson.
Oh, gee, a RWNJ proved wrong – again
I’m pretty convinced that as and when the Big One hits Wellington, we’ll be stuffed. Most DR systems in NZ work on a she’ll-be-right basis, even when they’ve bothered building one.
We were fortunate that very little infrastructure is centred or controlled in Christchurch. When Wellington goes, I’d suspect things like total failure of bank payments for several weeks. The people in a position to fix things, even if they survive, won’t be able to get to work through streets prowled by trigger happy foreign cops and squaddies.
Yes agreed. Everyone’s personal civil defence bag should also contain a plan to deal with this certainty. Do not rely on the public services – they are well intentioned but ………. (see EQC example)
If there is one thing we all learnt in Chch it is that simply having a plan helps immeasurably. And practising that plan multiplies that help ten-fold. And also, quite simply saves lives.
You obviously have no experience of the NZ Banking system. There is significant DR capacity and most (if not all) have operations centers in Auckland so if Wellington went there would be only minor disruptions to the rest of the country.
Hence a faulty patch lead took out interbank payments.
Believe me, I have considerable experience and have seen it all, including a “DR” system involving having duplicated servers at the other end of the room from their supposed backups.
That error was rectified within a day. You are implying that the NZ payments system would be out for several weeks if there was a serious disaster in Wellington. It is simply not the case.
I have worked in banking for a number of years including in testing DR capabilities. There is simply no way that a failure in Wellington would lead to the situation you are postulating.
It is obvious you have little understanding of the infrastructure the banks have put in place around this.
What is your considerable experience of the NZ Banking industries payment system Rich?
LPRENT for parliament!
No thanks… I have spent literally decades supporting people silly enough to want to do the job on the off-chance that my duty itch would overcome my programming obsession. I like the latter and think that I’d scare too many people (and get too enamoured of) growling at them to do the former.
Also I really really don’t wanna be nice – and the voters expect that an attempt would be made to be so.
Goodness gracious, I didn’t expect such a polite response. 🙂
I was not aware you were such an excrement exciter. 😉
Perhaps Kiwirail is being systematically sabotaged from within on behalf of the Roading Lobby et al’s glove puppets, given that this is about the fourth adverse story coming from them within a fortnight or so. Don’t forget, the head of Kiwirail was appointed by Shonkey’s Administration. Yes I know, conspiracy etc etc but…….
Yes Jim Quinn is very much the yes man……whatever Joyce wants he gets.
During the last Christchurch earthquake in December, media reported that air traffic control for the whole country was down because it was still based in that city. Who on earth would regard that as sound management?
The RWNJs – minimising the number of people and locations used increases efficiency. Doesn’t do too well when the lone place gets crushed though.
To everyone it increases efficiency. But to most people efficiency at the cost of reasonable redundancy is stupid- to extreme right-wingers, it doesn’t matter because they pay directly for the redundancies in things that effect them.
Yes and of course the buses were all full leaving the city so no alternatives and no signs at station entrances letting people walk to platforms….FFS common sense people.
Beggars belief that akl system is 100% reliant on Wellington based infrastructure but then this is kiwi rail led by ex NZ Post old boy and govt muppett Jim Quinn.
See how the media uses this as another opportunity to denigrate trains. The DomPost reports almost every late train in the capital. There is never any equivalent scrutiny of road costs, road transport issues and the constant traffic jams. There is also an ongoing campaign to build the Transmission Gully White elephant. Roads good, trains bad. So boring.
EQC in Christchurch – not a back-up plan. Not even a plan. And their entire contract was to deal with a major earthquake in any of NZ’s main population centres. Epic fail.
Fukushima nuclear power plant – all back-up plans involved the use of electricity to cool an out of control meltdown. No simple mechanical device requiring no electricity. You would think they had never heard of gravity.
It doesn’t surprise me one iota.
Don’t rely on any systems seems to be the only reliable way.
It’s absolutely unbelievable if there isn’t a backup power supply.
And also unbelievable that the so-called journalists didn’t ask if there was one.
That was the problem, the backup power supply didnt ‘switch over’.
Very poor reporting over this- usual situation when there isnt some sort of celebrity involved, then we would have all the gory details
It’s “efficient” to centralise everything and take out ‘unnecessary’ redundancy (hey why pay for back up systems when they wll go unused for 99.99% of the time???).
Shame when some little thing breaks though and everyone everywhere gets screwed simultaneously.
As someone who is largely responsible for running the technology side of a similar piece of major infrastructure… I can only agree in principle with Lynn’s original post. A large portion of our resources go into adding resilience to the system. We can never make it utterly bullet-proof in the face of every conceivable event, or sequence of events… but we can cover most of the predictable ones.
Having said that I’m reluctanct to put the boot in without knowing more of the technical details. It’s far too easy to get high and mightly about something that turns out to be a bit more complex than you initially imagined. And because most of our media is technically illiterate we’ll never get a coherent explanation out of them. I do have a back-channel into Kiwirail but I can’t count on it delivering.
Train Control is an exceedingly conservative and risk averse aspect of the system, and you don’t fiddle or upgrade the system without spending a lot of money. My educated guess is that Kiwirail has been cash and capital starved for far too many decades and the kind of redundancy you are thinking of just hasn’t been in the budget.
Should have just kept signal men on the line with mechanical signal boxes. Keep more people employed doing an important job. Those systems worked with zero fatalities for decades.
Too “inefficient” I suppose.
Zero fatalities except for the single largest train disaster in our history siler fern derailment, Wellington rail yards crash, etc etc. those dead probably looked like no-one as you whizzed past in the Porsche
I don’t expect it to be bullet-proof, but the first thing I would have expected to see from Kiwirail woud have been a statement saying that the backup systems were being activated. More than three hours afterwards the only thing I could see was that some trains were starting to run – how?
Anyway I went into a movie then. I will have a look in the morning. But it did seem somewhat fragile.
Ask Air NZ how their systems worked when their was a failure in the aircraft boarding computers. IBM who ran it had very little idea what to do when there was a problem and it took many hours to fix.
I remember a large industry in Australia ( Holden) had a motor generator/flywheel as the first part of a large battery backup. And it was tested at a quiet period every week.
Dont reply on electronics to switch over your backup power
Veolia – that wonderful local Kiwi organisation?
They also look after our waste disposal in Auckland.
Roadside collection up over 100 % in 10 years. Paid for through the bags.
No corresponding reduction in rates though.
Inorganic collections – thing of the past. Take it yourself and pay big bucks.
Don’t you love the service you get when multinationals take over?
Not Veolia . See above
They do run our local commuter system. Therefore they are the organisation responsible for making it run correctly.
So you’re not really looking for the organisation responsible for the problem are you, only for a convenient scapegoat that you can complain to.
They cant run trains without signals, the responsible thing was to stop the trains. And they dont ‘buy’ the track access rights so Kiwirail isnt responsible to them directly.
Welcome to the corporatised model of public services, where all complex organisations are sliced and diced by outsourcing and sub contracting.
Personally I have never heard of any similar incident anywhere in the world, where such a failure was caused by one central “control” unit somewhere down the line. The train infratstructure so far in place leaves a lot to be desired. I was worried when I first ever saw Britomart and the trains in use. It seems to have been a rushed and cheapo decision to build the structures as they have been. Then again NZ companies involved took out of it what they could, pushing costs up beyond reason.
Auckland has a vulnerable, partly dysfunctional train infrastructure, which needs urgent attention.
But as long as the funding and other policies are not in place, NOTHING of substance will happen. It is not much wanted anyway by Joyce, Key and English, given the new 1 billion now suddenly missing of of government coffers. Maybe that is “creative accounting” though, because due to the large opposition to asset sales, some new trump card is needed now, to forcefully “convince” the public, there is NO other way?!
I suppose the trains in Jakarta, Mumbai and so forth run better than here. What does this tell us about NZ in 2012?
LPRENT: “Now speaking purely from my old training in earth sciences, there is absolutely no way that ANY system in NZ should rely on a single location as a point of failure.”
Do you not remember the power outages in Auckland a few years ago? Was it not a similar scenario? NZ has appallingly poor infrastructure, and when Queen St in Auckland’s CBD was dug up and redone a few years back, I saw the exposed “infrastructure” for water and so forth. It was unbelievable, and this city up here still relies on a totally over aged, redundant and highly fragile network for electricity, water and gas sypply, and waste water networks are not any better.
The money went into some fancy and costly face saving projects for past mayors and Councils, who largely also feathered their own nests.
Just wait a few years down the line, and Auckland will have a total black out again, maybe no water in large parts and more. The infrastructure under the ground has been patched up, is cheap and highly vulnerable, just ask an insider engineer and so about it. This city, like much of NZ is built “Lego” style – without a solid, reliable infrastructure network in many areas. All quick deals, jobs and bucks for the companies involved, but NO long term solutions, same as in economic, trade, social and other policies. NZ will fall totally to pieces very, very soon.
Yes, please, China, China, help us out, will be the desperate scream, which Key is already trumpeting around. NZ is for sale, will not function and hundreds of thousands will leave to head offshore in the coming decades!
But its OK Xtasy, we will have an inner city train loop. That will help.
The power failures were management failures not equipment ones. Poor maintenance and perhaps poor design at otahu and poor on site work practice at newmarket. The equipment never actually failed. The city centre failure in 98? Was due to ongoing managment failures not checking state of assets. People are often the greatest weakness in infrastructure.
Note these were failures by state or city owned monopolies.
What was the cause? Human error (which is forgiveable) or cost cutting to boost profits (which isn’t)?
It wasn’t cost cutting. Transpower is fully and guaranteed funded by power users and doesn’t get much push back on its budgets. Otahu was the result of failing to check a $20 shackle. But the cause was poorly designed and managed maintenance programmes and risk assessment over many years across the organisation.
Basically their maintenance was scheduled ‘we do this every X period’ not ‘the risk of failure are generally X, but here the cpnsequences are X x Y, so we should be a bit more rigorous in our checks’. A huge lack of sophistication I think caused by the cosy complacency you can get in a fairly unaccountable monopoly.
Similar with the central auckland outage in the 90s – people got lazy and oversight and accountability was poor, so basic checks and maintenance was not done on a complex piece of gear because it always worked, until it didn’t….
With Newmarket, someone was working on the nearby substation transformers and didn’t follow correct shutdown procedure, but that was conveniently blamed on ‘old’ equipment because the transformer was 45 years old (easy to do in the age of disposable electronics but ignoring that the useful life of them is at least 70 to 80 years (most are not that old so we don’t really know what they are capable of), but soundbites are important when you are wanting to justify more cash for upgrades)
And the maintenance was reduced because of…?
This is the bit you want to avoid because you don’t want to admit that exactly the same thing would have happened in a privately run outfit. I know because I’ve seen it, railed against it. Cutting costs to boost profit is systemic to all profit driven companies. All of them, doesn’t matter if its private or state owned.
Where did I say maintenance was reduced? You seem to have this view that chucking lots of money at everything will answer everything, and that there is an ongoing conspiracy to endanger things by cutting costs indiscriminately.
In all of these cases there was no question of a lack of money. They were state monopolies with guaranteed revenues. The common issue was it was poorly directed because the management did not do their jobs and did not understand the risks they were dealing with. You can have the best maintained and best oiled machine in the world, but if Homer Simpson is the man with his hand on the switch, you are still going to have issues.
The know how to get NZ working and functioning again simply no longer exists. The social order is broken and even destroyed. Education is focussed on the elite getting good education, the rest enough to sell burgers and cut hair to/for others. Some work in offices of corporations or even councils and what else, as typists, receptionists, well there is also still some good research going on, but who for, and who uses it.
Hence the government has decided, sell NZ bit by bit.
I have resigned to the fact that NZ cannot be sold bit by bit, it either has to radically re-invent itself with immense input by locals, migrants and investors with genuine intentions for the whole NZ good, or it has to take the sour step to sell THE WHOLE COUNTRY, to hand it over to a developed economy that may actually have good intentions and is willing to invest here.
Such advanced countries could be from different places, and they have to make reasonable offers to invest here, also agreeing on social, environmental and legal standards upholding first world conditions here. There are countries that could do so, but not without a price.
So the issue is clear. Many will object to a whole sale, simply because “we play rugby” and “they play soccer”, so “bugger off”.
Others will want to cling to every inch of ground and grass, for whatever reason, whether owned by Michael Fay, an iwi or more, or by whomsoever.
While the whole world economy is on the brink (and IT IS, despite of good share news at present), maybe rething the future of NZ as a whole. The choices are getting less, and Key and consorts are only in it for themselves and their mates. They will not be interested in the investment and economic development NZ needs. Good night, dear discussers of whatever.
yeah, except Apple accounts for about 30% of that stock market increase, by itself. And its not hard to pump a market up when you print money into it and hide all the liabilities off balance sheet.
The share markets are highly distorted, only representing a day to day sentiment of gamblers, such investors and opportunists seeking the best returns in the short term. It is not at all reflecting the whole economic and fiscal realities in all countries at present, as you rightly point out, also partly due to quantitative easing.
Economics is not an easy discipline, and even the leading ones disagree on some core matters.
Yet even top investors and gamblers like Soros understand that Europe’s crisis is anything but resolved. That is not necessarily saying Europe is bad, and it is all to blame on budget blow outs due to welfare costs. That may be part of the picture in SOME countries, yet if you have a globalised economy, where some players (e.g. Mainland China) are now members of the WTO, but still to a fair degree manipulate currency values, still breach intellectual property rights, still have their own blow outs internally, like in Chongquin, previously ruled by corrupt Bo Xilai, the whole country there is also built like a house of cards.
NOTHING is safe and certain anymore, and NZ would be doing very well, to radically rethink economic policies, rather than continue to simply rely on FTAs and open borders and exchanges of goods for the future safety. English shows it is NOT working! Asset sales will aggravate the situation. NZ must focus on INTERNAL economic stabilisation and development.
The US only relies on 4 or so per centage of the whole economic activity on imports and exports, the rest is LOCAL. Their issues are LOCAL. Some countries in Europe have local issues and can resolve them, with the help of the EU commisison and banks there. But global exposure is a two edged sword, which few in NZ understand.
Those that will come with the Cuba and NOrth Korea argument are idiots, as they are ridiculing the challenges that exist.
Plus they have interest rates in the US at practically zero ( for the big banks) who wouldnt invest in even the most blue chip of stocks.
I was left high-and-dry – we were told we could catch a bus using our train ticket, except every bus going along New North Rd was packed by the time they got to Mt Albert. Finally, I got onto one and the driver said that I couldn’t use my train ticket – didn’t have any cash on me, received a lecture from the driver about how I should get a Hop card (I will once they’re integrated with trains) before I got off and walked home. In a suit. In the rain.
Fuck the convention centre – let’s sort the basic shit out first.
You got that right!
One and three-quarters years commuting between Henderson and Newmarket on that clapped out piece-of-shit train set and I had had enough and moved down here to Christchurch where I moved into an inner city unit so I could walk to work. Since then we’ve had those earthquakes and I have had no work to walk to. Ah well, I like New Zealand and all (hence my username), but it isn’t easy sometimes.
Please don’t move to Wellington Happy, I’m now worried you are like the Rain God in Hitchhikers Guide….:-)
😀
You could pay me to stay out of Wellington.
Thats why we shouldnt be paying for it, however we are paying for this useless rail loop, so good luck getting anything fixed now Rupert.
I’ve got a bit of knowledge on railway signalling, and I’ve been trying to work out what happened here. The one time I went to Train Control was an enlightening experience – you still had large sections of the network as “dark territory”, where train progress was mapped with pen and paper on a reel of graph paper. For much of the country, this is still the system, except with the help of a computer (Track Warrant Control)
However, for Auckland, it’s a spanking new Centralised Traffic Control (CTC) system, without many local panels that would kick in the event of a failure of the control from Wellington. There might be one or two local control options, perhaps at Britomart and Otahuhu/Southdown, but that’s it I think. This would be anathema in the NZR days – where most local stations had some form of backup mechanical or electrical control over signalling, and someone to run them, or at least a crew that would readily be called out to fix them. Or in many cases, it was just stations on their own, running 80 year old tablet machines, with a single wire running between them. Antiquated yeah, and it required a lot of staff to run it, but it worked.
ON the dark areas, if you are transporting a long or special load (like a house or transformer etc) across a railway crossing you have to go through an eloborate process of contacting the centre and waiting for periods because the controllers don’t know where trains are on sections of track. They know when the train left ‘point X’ but are never quite sure when they will go through the crossing you want to get across, so you have to wait until they are sure you are safe.
Perhaps Kiwirail is being systematically sabotaged from within on behalf of the Cycling Lobby et al’s lycra puppets, I say lift the rails and turn the carriageway into cycle and walking tracks.
Doesnt work as the train is the most efficient means of public transport, both in terms of carbon footprint and corridor land use. Cyclists generally love trains as they can take the bike onboard.
But of course you were just talking nonsense
So there is a back-up centre in Auckland, but it is not staffed for short term emergencies….?
http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=10801750