Tuesday, May 31, 2011

As a boater, there are times when I know our survival is 100% dependent upon the weather conditions, the boat, and the state of its equipment. As a consequence, I think hard about human or equipment failure modes and how to mitigate them. I love reading the excellent reporting by the UK Marine Accident Investigation Board. This publication covers human and equipment related failures on commercial shipping, fishing, and recreational boats. I read it carefully and I’ve learned considerably from it.

 

I treat my work in much the same way. At work, human life is not typically at risk but large service failures can be very damaging and require the same care to avoid. As a consequence, at work I also think hard about possible human or equipment failure modes and how to mitigate them.

 

Wanting to deeply understand unusual failure modes and especially wanting to understand the errors that humans make when managing systems under stress, I spend time reading about system failures. Considerable learning can be drawn from reading about the failures of engineered systems and people under stress. All disasters or near disasters yield some unique lessons and re-enforce some old ones.

 

The hard part for me is getting enough detail to really learn from the situation. The press reports are often light on details partly because general audiences are not necessarily that interested but there also may be legal or competitive constraints preventing broad publication. NASA, FAA, Coast Guard and some other government reports to get to excellent detail. One analysis of system failure I learned greatly from was Feynman’s analysis of the space shuttle Challenger disaster as part of the Rogers Commission Report.

 

I just came across another report that is not quite a Feynman  classic but it is an excellent, just-the-facts description of a large scale failure. This report, from IEEE Spectrum, titled What Went Wrong in Japan’s Nuclear Reactors outlines what happened in the eventually catastrophic disaster at Japan’s Fukushima Dai-1 nuclear facility following the Tohoku earthquake and subsequent tsunami. In this report, the terminal failures of 4 of the 6 reactors at the facility is described in more detail than other accounts of that event I’ve come across.

All disasters are unique in some dimensions. What makes Fukushima particularly unusual is these failures occurred over multiple weeks rather than the seconds to hours of many events.  This one was relatively slow to develop and even slower to be brought under control. Looking forward, I suspect Fukushima will share some characteristics with Chernobyl where mitigating the environmental damage is still nowhere close to complete nearly three decades later. In 1998 the Ukraine government obtained economic aid from the European Bank for Reconstruction and Development to rebuild the failing Chernobyl sarcophagus. It is expected that yet more work will need to be done to continue to contain dangerous radioactive substances from escaping.  Similarly, I expect the environmental impact of the Fukushima disaster will be fought for decades at great cost both economic and human.

 

In many ways Fukishima was a classic disaster where a not particularly surprising event, in this case an earthquake near Japan, started the failure and then cascading natural disaster, equipment failure, and human decisions followed to yield an outcome that every aspect of the system design sought to avoid.

 

I recommend reading the IEEE report linked below and my rough notes from the write-up follow:

·         On March 11 an earthquake registering 9.0 magnitude was experienced off the coast of Japan

·         The tsunami hit the plant destroying power distribution gear cutting off power to the Fukushima facility

·         Backup generators and switch gear were also disabled by the Tsunami

·         Reactor building integrity was maintained through earthquake and Tsunami and the three reactors that were active at that point where all shut down properly

·         Due to the power failure and the damage to distribution gear and generators, plant cooling systems were not operating at any of the reactors nor the spent fuel rod storage pools

·         Even though the nuclear reaction had been stopped in the three reactors that were operational when the tsunami hit (reactors 1, 2, & 3), considerable heat was still being created putting the reactors at risk of meltdown. Meltdown is a condition where reactor core over temperature occurs, the coolant is boiled off, the fuel rods melt and form a pool of very hot, highly radioactive fuel in the bottom of the reactor. This hot, radioactive fluid then rapidly breaks down steel and concrete in the containment vessel and possibly escapes to the environment.

·         Another area of risk from the failed cooling systems are the spent nuclear fuel rod storage pools. These pools are also housed inside the reactor buildings near the primary containment vessel where the active nuclear reaction actually takes place. Although the fuel rods are no longer contributing to a nuclear reaction, they are both highly radioactive and still producing sufficient heat that active cooling is required. Without cooling these rods can heat the storage pool to the point that it boils off the cooling water and present a risk similar to the active rods inside the primary storage vessel.

·         I find it surprising that both the spent rod storage and the shut down reactor cores don’t appear to fail safe and self-stabilize when cooling water is removed given the considerably higher than zero probability of power failure and the seriously negative impact of radioactive release to the environment.

·         Events at Reactor #1:

o   March 12, a day after the power failure, heat in the recently shutdown reactor built up until the (not circulating) cooling water began to be boiled off.

o   As the water level fell, the now exposed fuel rods reacted with the steam in the primary containment vessel, and began producing hydrogen gas

o   The pressure rose to dangerous levels in the primary containment vessel and operators decided to vent the primary containment vessel into the reactor building.

o   The vented hydrogen gas when exposed to the relatively oxygen-rich environment in the reactor building, exploded blowing the top off the reactor building

o   The explosion may have also damaged the primary containment vessel and definitely released radioactive material

o   The operators chose to pump seawater into the building in an effort to control the escalating temperature inside the reactor and to avoid core meltdown

o   March 29, radioactive water was found outside the reactor building

o   April 5, reactor core temperatures have begun to fall indicating the system is coming back into control

o   Radioactivity levels in the building are very high and operators are injecting nitrogen to reduce the likelihood of subsequent hydrogen explosions.

o   May 12, TEPCO officials confirmed that the reactor had suffered a core meltdown and the bottom of the reactor building may be leaking highly radioactive water into the environment.

·         Events at Reactor #3:

o   March 14, 3 days after the tsunami and 2 days after the roof was blown off the Reactor #1 containment building, the same thing happened on Reactor #3

o   This explosion occurred despite plant operators pumping large quantities of cooling sea water into the reactor building

o   March 17, steam begins billowing from the reactor building confirming that the primary containment vessel was damaged and releasing radioactive compounds.

o   Helicopters dumped water on the building and police water cannons were used to pour water down onto the building.

o   Water was sprayed on the building for days with some interruptions as radiations levels rose sufficiently high that work had to be stopped.

o   March 24, workers laying power cables attempting to restore power to Reactor #3 waded into highly radioactive water requiring hospitalization.

o   March 28, dangerous plutonium was detected in the environment near Reactor #3.

·         Events at Reactor #2:

o   March 15, 4 days after the tsunami, 3 days after the roof was blown off Reactor #1, and a day after the roof was blown off Reactor #3, a serious explosion occurred at Reactor #2.

o   Reactor #2 was later confirmed to have experienced at least a partial core meltdown

o   March 27, highly radioactive water discovered outside of reactor building #2.

·         Subsequently large quantities of uncontained radioactive water has been found throughout the multi-reactor plan and the turbine facilities are flooded as are the cabling tunnels between the buildings. Serious radioactive water leaks into the ocean have been detected and subsequently corrected in one case by injecting 6,000 liters of liquid glass into the ground near the leak.

·         April 4th, 11,500 tons of radioactive water is pumped into the ocean. This water is 100x above the legal safety limit but was pumped into the environment in the hope that the storage facilities can be used to contain waste water that is 10,000x time radioactive limit for environmental release.

·         The spent fuel pools at the inactive reactors 4, 5, & 6 were all slowly overheating as a consequence of there being no cooling water. The Reactor #4 cooling pool either boiled off its water or it leaked off as a result of earthquake damage. The spent fuel rods exposed to atmosphere without cooling lead to fires inside Reactor building #4

·         Outcome:

o   Fukushima now rated to be as serious as the Chernobyl having been classified as a a magnitude 7 event, the worst on the International Nuclear Event Scale. However it is still consider to have released only 5 to 10% of the radiation released by Chernobyl.

o   All residents within 20 km evacuated

o   Voluntary evacuation of all residents between 20 and 30 km.

o   Agricultural products including milk and vegetables from the region contaminated

o   Tokyo’s tap water declared unfit for infants for 1 day

o   Decades of cleanup and containment remain

 

The report: What Went Wrong in Japan's Nuclear Reactors: http://spectrum.ieee.org/tech-talk/energy/nuclear/explainer-what-went-wrong-in-japans-nuclear-reactors.

 

We all wish the situation had been avoided and, those of us involved in engineering projects whether they be life critical systems or not, need to ensure that the lessons from this one are learned well and applied faithfully to new designs. I won’t speculate on human risk in the efforts spent to mitigate this disaster but, clearly, the workers that brought these systems back under control and continue to manage the environmental impact are heroes and deserve our collective thanks. 

 

                                                                --jrh

 

James Hamilton

e: jrh@mvdirona.com

w: http://www.mvdirona.com

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

 

 

Tuesday, May 31, 2011 5:39:33 AM (Pacific Standard Time, UTC-08:00)  #    Comments [8] - Trackback
Ramblings
Tuesday, May 31, 2011 6:16:22 AM (Pacific Standard Time, UTC-08:00)
"I find it surprising that both the spent rod storage and the shut down reactor cores don’t appear to fail safe and self-stabilize when cooling water is removed"

Indeed! There appear to exist few if any nuclear reactor designs that are fail-safe in this sense (tolerating loss of power / cooling).

The other surprising thing was that the spent fuel pools are not protected by a robust enclosure at all. This means that should one of them melt down (especially #4, full of unspent fuel!), no steel or concrete would hold in the goop.

Strange design.
Frank Ch. Eigler
Tuesday, May 31, 2011 7:16:01 AM (Pacific Standard Time, UTC-08:00)
To me the key lesson, and I'm not sure its not a lesson we already knew, is that loss of power or cooling needs to lead to a stable, non-reacting, cooling state.

The cooling pools at Fukushima actually are housed inside the secondary containment building that also houses the primary containment vessel. There is a concrete containment building. However, in the event the water in the pool boils off, the spent fuel rods will get extreamly hot, cause fire, and release hydrogen which can damage the secondary containment building. It is also possible that the rods in the containment pool could melt and the nuclear reaction in the liquid could restart at which point the steel and unlikely to successfully contain the reacting fuel.

--jrh
Tuesday, May 31, 2011 7:58:06 AM (Pacific Standard Time, UTC-08:00)
"cooling pools at Fukushima actually are housed inside the secondary containment building"

Sort of: they are on top of the secondary containment concrete building. All that's *covering* the pool is the thin structure that blew out from hydrogen, meaning that the cooling pools are currently exposed to air in the damaged buildings. This is why camera footage of some of the spent fuel pools was able to be collected recently.
Frank Ch. Eigler
Tuesday, May 31, 2011 9:25:13 AM (Pacific Standard Time, UTC-08:00)
I'm pretty sure that both the pools and the primary containment vessel are fully enclosed by the secondary containment building. From the article: "The primary containment vessel and the torus are in turn encased by the secondary containment building, a large box of steel and concrete. This protective building also houses a storage pool where spent nuclear fuel is kept in cool, circulating water."

Because the roofs were blown off the buildings by the hydrogen explosion, the pools are now fully exposed.

--jrh
Tuesday, May 31, 2011 12:10:16 PM (Pacific Standard Time, UTC-08:00)
Well, I won't try any more to convince you after this. :) But note how they use the word "housed"
for the pool, and "fully enclosed" for the other stuff? The explosion did not nudge the concrete
enclosure - if it did, other stuff than the pool would also be exposed.

See also http://powerandcontrol.blogspot.com/2011/04/fukushima-16-april-2011.html
and http://www.ucsusa.org/assets/documents/nuclear_power/lochbaum-testimony-senate-energy-approps-3-30-2011.pdf

I know it's hard to believe. And yet ...
Frank Ch. Eigler
Tuesday, May 31, 2011 5:27:29 PM (Pacific Standard Time, UTC-08:00)
The subject of why things fail has always been something I find fascinating. Unfortunately the media & public are all too focused on apportioning blame, whether that’s a natural disaster, human error or component failure. The best recent example of this is the Challenger disaster. The media and many commentators blamed the O Ring, when in reality it was a systemic failure that allowed the shuttle to launch.

The world today is now driven by the 24hr media cycle and the 30 second sound bite. The Toyota Unexplained Acceleration is a classic example, most people don’t know what the end result was and the media don’t care because it’s not news any more.

The public is now losing interest in the Fukushima disaster and ask the public for the cause and the answer is the Tsunami compounded by the operator mistakes. Let’s hope that the public will be given the full picture and operators are given the credit for risking their lives and making difficult decisions, right or wrong.

On the general subject of failure, I can recommend “Human Error” by Professor James T. Reason. While most of his work has been with Aircraft & Hospitals, the principles learned in his studies can be applied to all organizations (even IT change Management). If you get a chance to hear him speak you will not be disappointed, no matter what your field.
Anthony Drew
Tuesday, May 31, 2011 6:44:45 PM (Pacific Standard Time, UTC-08:00)
Thanks Frank. I may indeed be reading to much into the phrase "housed". thanks,

--jrh
Tuesday, May 31, 2011 6:46:16 PM (Pacific Standard Time, UTC-08:00)
Thanks for the comment and the pointer to "Human Error" Anthony.

--jrh
Comments are closed.

Disclaimer: The opinions expressed here are my own and do not necessarily represent those of current or past employers.

Archive
<May 2011>
SunMonTueWedThuFriSat
24252627282930
1234567
891011121314
15161718192021
22232425262728
2930311234

Categories
This Blog
Member Login
All Content © 2014, James Hamilton
Theme created by Christoph De Baene / Modified 2007.10.28 by James Hamilton