Tuesday, October 27, 2009

A Safety Argument Against Stupid Software

Everyone is already familiar with the peculiar incident last week where we saw a Delta/Northwest Airlines A320 on a flight plan from San Diego to Minneapolis remain out of contact with ATC for over an hour, overfly its destination by 150 miles and then turn around and land over an hour late, with both pilots claiming they weren't sleeping, had merely been so thoroughly engrossed in discussion of company policy that they missed the radio calls and lost track of time.

This FlightAware track, which I can't see because I'm on stupid, stupid hotel internet, should show its path and this terse NTSB press release gives the facts.

I'm not sure whether your pilots missing their exit because of a "heated discussion" is preferable to their falling asleep. Falling asleep can be called an involuntary physical reaction to being sleep deprived, rather than deliberate neglect of responsibility. I guess the argument is that if you're awake in a discussion you're more likely to notice and correct a abnormal condition, but the facts of the case dispute that argument, seeing as they failed to answer multiple ATC calls, SELCAL signals or notice that they had reached the point at which they should have been descending.

I wasn't there, so I instead present the top ten explanations for the tardiness of flight 188:

  • 10. intense discussion of company policy
  • 9. sleeping
  • 8. playing a game of chicken on when to start the descent
  • 7. waiting for the movie to end
  • 6. hoping for some talk show appearances to jumpstart new careers as "celebrities"
  • 5. flying over the captain's girfriend's house to check for a strange car in the driveway
  • 4. trying to confirm rumours of a "Tetris mode" on the FMS
  • 3. joining the six-mile-high club
  • 2. abducted by aliens for two weeks, then replaced in the cockpit with no memory of the event, and only an hour later in Earth time
  • 1. just wanted to supplement their paycheques with a couple of hours extra pay

The NTSB, as mentioned in the press release, has pulled the cockpit voice recorder, but the variety in that plane only records the previous 30 minutes of conversation. The pilots would have known that, and whatever happened they used the 150 miles of backtracking to get their story straight. I'm guessing by passing notes, or typing on someone's PDA, while the CVR recorded only normal cockpit sounds.

There's a malfunction that occurs sometimes on those old cockpit voice recorders. I know about it because has come into play in at least one accident investigation. While the tape loops around every 30 minutes, sometimes the old track isn't erased, and you get an audio equivalent of the double exposure. I have no idea how rare this is.


Update: I can't find a link to this on the NTSB site, but here's the latest on the investigation:

************************************************************

NTSB ADVISORY

************************************************************

National Transportation Safety Board
Washington, DC 20594

October 26, 2009

************************************************************

NTSB ISSUES UPDATE ON ITS INVESTIGATION OF FLIGHT 188 THAT OVERFLEW INTENDED MINNEAPOLIS AIRPORT

************************************************************

In its continuing investigation of an Airbus A320 that overflew the Minneapolis-St Paul International/Wold- Chamberlain Airport (MSP), the National Transportation Safety Board has developed the following factual information: On Wednesday, October 21, 2009, at 5:56 pm mountain daylight time, an Airbus A320, operating as Northwest Airlines (NWA) flight 188, became a NORDO (no radio communications) flight at 37,000 feet. The flight was operating as a Part 121 flight from San Diego International Airport, San Diego, California (SAN) to MSP with 144 passengers, 2 pilots and 3 flight attendants.

Both pilots were interviewed separately by NTSB investigators yesterday in Minnesota. The following is an overview of the interviews:

  • The first officer and the captain were interviewed for over 5 hours combined.
  • The Captain, 53 years old, was hired in 1985. His total flight time is about 20,000 hours, about 10,000 hours of A-320 time of which about 7,000 was as pilot in command.
  • The First Officer, 54 years old, was hired in 1997. His total flight time is about 11,000 hours, and has about 5,000 hours on the A-320.
  • Both pilots said they had never had an accident, incident or violation.
  • Neither pilot reported any ongoing medical conditions.
  • Both pilots stated that they were not fatigued. They were both commuters, but they had a 19-hour layover in San Diego just prior to the incident flight. Both said they did not fall asleep or doze during the flight.
  • Both said there was no heated argument.
  • Both stated there was a distraction in the cockpit.
  • The pilots said there was a concentrated period of discussion where they did not monitor the airplane or calls from ATC even though both stated they heard conversation on the radio. Also, neither pilot noticed messages that were sent by company dispatchers. They were discussing the new monthly crew flight scheduling system that was now in place as a result of the merger. The discussion began at cruise altitude.

  • Both said they lost track of time.
  • Each pilot accessed and used his personal laptop computer while they discussed the airline crew flight scheduling procedure. The first officer, who was more familiar with the procedure was providing instruction to the captain. The use of personal computers on the flight deck is prohibited by company policy.
  • Neither pilot was aware of the airplane's position until a flight attendant called about 5 minutes before they were scheduled to land and asked what was their estimated time of arrival (ETA). The captain said, at that point, he looked at his primary flight display for an ETA and realized that they had passed MSP. They made contact with ATC and were given vectors back to MSP.
  • At cruise altitude - the pilots stated they were using cockpit speakers to listen to radio communications, not their headsets.
  • When asked by ATC what the problem was, they replied "just cockpit distraction" and "dealing with company issues".
  • Both pilots said there are no procedures for the flight attendants to check on the pilots during flight.

The Safety Board is interviewing the flight attendants and other company personnel today. Air traffic control communications have been obtained and are being analyzed. Preliminary data from the cockpit voice recorder (CVR) revealed the following:

  • The CVR recording was 1/2 hour in length.
  • The cockpit area microphone channel was not working during this recording. However, the crew's headset microphones recorded their conversations.
  • The CVR recording began during final approach, and continued while the aircraft was at the gate.
  • During the hours immediately following the incident flight, routine aircraft maintenance provided power to the CVR for a few minutes on several occasions, likely recording over several minutes of the flight.

The FDR captured the entire flight which contained several hundred aircraft parameters including the portion of flight where there was no radio communication from the flight crew. Investigators are examining the recorded parameters to see if any information regarding crew activity during the portion of flight where radio contact was lost can be obtained.

The Safety Board's investigation continues.

It sounds like I missed possibility #11: Computer software lessons in my list. And I hope the final report answers my greatest outstanding question on the matter: were they Macs or PCs?

38 comments:

Unknown said...
This comment has been removed by the author.
Aviatrix said...

Ah thank you. That's where I expected it to be, but it wasn't at that URL when I looked. The forwarded e-mail version I received must have come out shortly before it went up on the website.

Jim said...
This comment has been removed by the author.
Jim said...

Apparently, this is going to be filed under the same heading as texting while driving... doing something which totally dominates your attention when you are supposed to be doing job #1.

One speculates on their future job prospects. I speculate that Sam will have an opinion on that.

amulbunny's random thoughts said...

I've wondered why the CVR's haven't been updated to a chip rather than a tape. As has been said, once they hit Eau Claire and knew they screwed up they just had to keep it in the air for 30 minutes more and there would be no trace of the missing time.

With all the technology in the world,it should be a no brainer to update the CVR's to the digital age.

But what do I know? I trust the guys up front to do their job and get me to where I need to go.

Aviatrix said...

Information can be recorded on a chip, sure. I think the challenge is to record it in such a way that the recorded information survives a impact, fire, salt water immersion and other effects of an air crash. There are a couple of types of longer recording time CVRs, but there's no incentive to upgrade a CVR. It doesn't help anyone on the airplane.

Jeremy said...

Not only CAN the CVRs use a chip instead of a tape, but this particular one DID:
http://www.ntsb.gov/Pressrel/2009/091023.html

It's labeled "solid state" in that press release - meaning it doesn't have the moving parts of a chip, but uses some sort of flash memory.

But it still only recorded 30 minutes of data, apparently because that's all that was required by regulation until relatively recently. I'm sure pressure from crew unions, etc., meant that an airline wouldn't want anything more than the precise regulation.

Jamaicafest said...

Whatever the reasons for their actions the pilots may find themselves permanently grounded.

Anonymous said...

Yeah, this is a strange one... I do not understand how this could have happened and I am a middle-aged Airbus captain with 10 years in the aircraft, maybe 800 hours per year, so about 8000 hours in the A319,320,321. The big question: how did the crew not notice bank angles, course changes, and annunciations on the primary flight display and nav displays. I suspect there is more to this than being reported...

Aluwings said...

Captain Dave's excellent "Oops" blog entry mentions that dropping off the edge of an ATC frequency zone without a proper transfer to the next sector does happen from time-to-time for various reasons.

But add to this the complete lack of attention to the aircraft's flight progress... wow. Very strange.

Altogether, you've got to wonder why we're still using VHF radios - basically DC3 technology - for these key communications.

Anonymous said...

What I know, we were told that this plane wasn't to be stopped on the way to the gate after landing and after they de-planed, they aircraft was towed to a hanger. I'm going with the 6 mile high club, it makes great conversation at a Halloween Party!

LT

N6349C said...

On your question, "Mac or PC"?, I note that the error was eventually corrected and nothing crashed. Therefore, could not have been a Microsoft product.

I'm going with Mac!

BTW, welcome back.

nicnacjak said...

Maybe they were testing Windows 7 on the FMS. How did it work guys?

This is a very odd story. I'd presume based on the flight-aware track that the FMS was flying but I have to wonder why the STAR wasn't flown. The FA just records the plane on the same heading until they made contact with the ground and descended for app. Very od

Cirrocumulus said...

Talking of software, the current state of the art could give "Fifi" yet another computer to track pilot eyeball movements and work out whether she has their attention.
The question is, what should she do if they aren't?
a) Complain about it by annoying synthetic voice, sounding more like their mothers if they still don't respond.
b) Squeeze a thigh; if no response try higher up the leg.
c) Perform a barrel roll while giggling and saying "Wheeee!".

Sarah said...

It is a very strange story. I can imagine missing a hand-off from Denver center to Minneapolis center. But then to be completely "checked out" for an hour.... odd. I suppose it's possible; anyone know if there's access to the internet for the flight crew? :)

I feel a little sorry for the crew. They deserve a suspension, the captain back in the FO seat... but terminating their careers? A little harsh in my opinion, though the airline may be forced into it for PR. No metal got bent, no one got hurt. Where is the outrage over the taxiway landing in ATL? That sounds more serious to me, even though it was at 6am local.

@Nicolas, they must not have had the arrival in the FMS. after reaching RWF, it just went into heading hold.

Mac or PC? Maybe the worst case, one of each! Imagine the arguments and discussions arising from that alone.

Aviatrix said...

Doesn't heading hold seem like a poor response to reaching the end of the program without receiving further instructions? A pilot who reached a clearance limit without being able to contact ATC for further would enter a hold. I'd want that to be the default on an autopilot smart enough to do that. Perhaps after a few warnings asking me what it should do on reaching MSP.

Can an Airbus pilot confirm that heading hold is the normal response in such a situation?

Anonymous said...

I wonder about their fuel reserves. Landing an hour late could have seriously complicated matters unless they had fuel for a longish alternate leg. What was the MSP weather that day?

Security word: retch - say no mowwweerrr - oops sorry, I'll clean that up.

Sarah said...

I wondered about fuel too. I was talking to an Airbus pilot last weekend.. his opinion was they probably landed with less than 30 minutes of fuel.

Here was the weather at the time... soft IFR, 1200 overcast, occasional light rain as I recall.

METAR KMSP 220153Z 01010KT 10SM OVC012 04/02 A3004 RMK AO2 RAB15E23 SLP178 P0000 T00440022

They were cleared RWF .. SKETR3 arrival, and there is a hold charted at SKETR. ( Must be named for the MN state bird, the mosquito. ) People who know can correct me please, but I suspect radar vectors are the norm after SKETR.

dpierce said...

Some locomotives make you push a button every few minutes to prove you're awake. If you don't, Bad Things happen (brakes applied, messages to HQ, etc).

I'm waiting for someone to suggest this gets applied to airliners. ("If 15 mins of straight and level flight have elapsed, a member of the crew shall push the alert button ...")

Sarah said...

Dpierce, I've heard the Boeing 777 already has a "snooze alarm". If no controls are moved or buttons are pushed for 15 minutes an EFIS message pops up. The poster said not all buttons disabled this alarm; they made a game of figuring out which ones they were.

Here are Captain Dave's comments on his fine blog. ( tod == top of descent ).

The FMS will cause flashing verbiage to come up on the primary flight display when tod is behind you; when you get to the end of the lateral path, it will default to current heading and keep flying. There is a lot of unanswered questions... For instance, I assume the a/c was on STAR arrival into KMSP with several turns that the nav mode would have flown. (???)

I really have to get back to work. I may overshoot lunch.

Aviatrix said...

when you get to the end of the lateral path, it will default to current heading and keep flying

Ah there it is. I read his blog, of course, and somehow missed that phrase. Thanks for pointing it out. I'm guessing the STAR wasn't programmed in, that they were planning to do it after they knew for sure which arrival they were getting.

I may overshoot lunch.

LOL!

Aviatrix said...

when you get to the end of the lateral path, it will default to current heading and keep flying

Ah there it is. I read his blog, of course, and somehow missed that phrase. Thanks for pointing it out. I'm guessing the STAR wasn't programmed in, that they were planning to do it after they knew for sure which arrival they were getting.

I may overshoot lunch.

LOL!

Anonymous said...

The pilots would have known that, and whatever happened they used the 150 miles of backtracking to get their story straight. I'm guessing by passing notes, or typing on someone's PDA, while the CVR recorded only normal cockpit sounds.

That's a little harsh. We can presume that there was a shortage of good judgment on that flight deck, but there's no need to impugn the integrity of the pilots, especially in the absence of any clear information on what transpired during the incident. Making a mistake or even several mistakes is one thing, and lying about it is quite another. I'd like to think--actually, I am quite comfortable believing--that most professionals in that situation would not spend the next 150 miles of backtracking getting their stories straight.

Aviatrix said...

professionals in that situation would not spend the next 150 miles of backtracking getting their stories straight

Maybe it wouldn't take 150 miles, but I sure would have taken some time to make sure we were going to sing the same song. FWIW I do believe the story now that it has been clarified from "company policy discussion" to "figuring out the new company bidding system on our laptops." So getting the story straight could consist solely of "tell ATC it was 'company policy discussion'and then if the NTSB and company get involved, the truth, the whole truth and nothing but the truth -- but please don't tell them what I said about the chief pilot."

grant said...

re: "I'm guessing the STAR wasn't programmed in, that they were planning to do it after they knew for sure which arrival they were getting."

and

re: They were cleared RWF .. SKETR3 arrival, ...I suspect radar vectors are the norm after SKETR."

I haven't flown that exact STAR but ones similar. The FMGC (i.e. FMS) would be programmed right up to MSP initially from the database. At some point the pilots would have to edit in the actual runway in use and "connect the dots".

Some pilots/SOPs do this during initial programming to get the best fuel estimates from the system. Others just leave the destination approach stuff out initially and add it when closer to destination, when they've picked up the latest ATIS, and know more what to expect. I guess that didn't happen here. The radar track makes a straight line all the way down the STAR to MSP then just keeps on truckin' ...

Peter said...

... And the FAA has apparently just pulled their licenses as an emergency revokation. http://www.theglobeandmail.com/news/world/wayward-northwest-pilots-have-licences-revoked/article1340698/

Unknown said...

@Sarah: Termination may seem a bit harsh but consider: who was managing the aircraft during the extended period during which the pilots were both distracted from their only functions? They also seem to be in clear violation of company rules to the detriment of flight safety.

After the recent Air France disaster, there's a call for Flight Data and cockpit voice recordings to be uploaded to Mother during flight...

Voice comms are being phased out to some degree with CPDLC (Controller Pilot Data Link Comms) but this has practical limits in high intervention locations/phases.

Anonymous said...

I noticed that both those pilots are pretty high time guys on the Magic Bus. When I was junior I used to see captains doing crossword puzzles during the enroute phase of flight and thought to myself: "If I ever get that complacent/bored, it will be time to retire."

When I got senior and had been on the Airbus for 5 years I found myself doing crossword puzzles enroute to keep myself from falling into a daze (especially once the flight deck doors were locked).

I retired.

One thing I'll say about automation in aviation. We tend to automate too much of the stuff that pilots should be doing to keep involved.

Aviatrix said...

I saw that they had had their licences revoked. I think that's pretty harsh. What are you going to do in your early 50s when your means of livelihood has been taken away from you?

dpierce said...

Do lectures on aviation safety

kbq said...

More realistic: knowing that you've terminated your career and finished your family's ability to live in middle class America... end it?

I wouldn't be surprised if at least one of the pilots did so... and I'd place the blame on corporate CYA.


Kevin

Michael5000 said...

I wonder if we're quick to make assumptions because of the Go! Flight 1002 incident. It's interesting looking at the FlightTracker plot that shows that, yeah, they overshot by a ways. But from the tone of newspaper coverage, I got the impression that they were halfway to freaking Labrador before they made contact.

nec Timide said...

Termination may be a bit harsh, but consider the consequences of a wake turbulence encounter and upset.

It wasn't too long ago that (some) comments on this blog, and in other aviation punditry were willing to pillory the dead NASCAR pilots that died in the crash with no evidence at all, ignoring the fact that the previous pilot had flown an hour after detecting smoke in the cockpit, and that neither the previous pilot not the maintenance department could produce produce evidence that the snag had been reported at all.

How is it that dead crews somehow deserve their fate when this crew deserves a second chance? Our safety program, based on Transport Canada SMS, is non-punitive except when there is criminal intent, or willful negligence. You would have a hard time convincing me the latter is not the case here.

Verification word: skeri
Um... Ya.

Sarah said...

As a pilot, I sympathize with these guys, who made a single, though spectacularily egregious error which has ended their careers. It now appears it may end their flying entirely. Imagine having to go back up the whole ratings ladder, just to be able to instruct at the local fright school. Maybe I'm too sympathetic. In this case, perhaps it did cross the line from an error to negligence.

Every pilot is one failed medical from a career ending event... Still, the witch hunt aspect really bothers me.

These guys are the anti-Sullys. A certain segment of the flying public seems to want to believe pilots are super-human, heroic individuals they can thoughtlessly entrust their lives to, while abusing the flight attendants. Any fall from grace down to humanity is dealt with harshly. I've looked at the local
newspaper, which updated the story daily. As a desperate attempt to modernize, they've enbloggenized their reporting allowing comments, much like this blog. Unlike this blog, the commenters are the general public, the unwashed masses, the people not in love with aviation. And so very ignorant. The comments would blister your ears. Tarring & feathering would be a popular punishment; firing and license revocation is a poor substitute.

CPAP said...

The Mesa Airlines incident revealed that the Captain suffered from obstructive sleep apnea. He'd reported the snoring problems to his medical department and their response was (typically) totally inadequate.

Been there - done that. I was given instructions on how to hydrate my sinuses with salt water solution, and all the other usual Bravo Sierra "advice" that I could just as easily get from a dime-store magazine. This is what passes for aviation medicine?

If it turns out that the pilots of 188 did actually fall asleep as opposed to PCing their way into joblessness, this could actually go better for them as there could be an actual medical underlying cause.

fwiw.

nec Timide said...

@sara

I understand what your are saying, especially about comments to reporting. About a year ago GA pilot from this area continued VFR into IMC and ended up spreading his plane and passengers across the top of a hill, killing himself and one passenger, leaving two passengers badly injured. The news reporting was the normal low standard and comments were often ill informed rude and hurtful.

As a CASO I learn about and investigate human errors more frequently that I would have imagined before taking the post. In most cases they are simple human errors and the task is to do a threat/risk assessment and put in place, or reinforce measures to prevent the errors becoming losses. In this case there was, apparently, a policy against using personal laptops in the cockpit. There have also been years of study and training into CRM and Human Factors with clear examples of what happens when a crew becomes distracted from flying the plane. This crew should have been at the top of their game and realized that even pulling out one personal laptop was a very bad idea. I think tar and feathers is too much, but loosing their privileges and jobs sounds about right.

Verification word: remand

is there a pattern forming here.

Aviatrix said...

Just in case anyone missed the significance of the title of this blog entry, here is a blog entry by the first other person I have seen articulate the thought that inspired that title.

How ridiculous is that scheduling software?

Unknown said...

Matthew Wald at the NYT is, in my opinion, probably the best aviation reporter who's out in the general media.
He consistently puts out some very well done and intelligent pieces. He knows his stuff.

My $0.02 :-)