FAA allows US domestic flights after previous suspension.

Post Reply
Message
Author
TheGreenAnger
Chief Pilot
Chief Pilot
Posts: 3286
Joined: Fri Jul 15, 2022 11:40 pm
Location: Unfashionable end of the Western Spiral

FAA allows US domestic flights after previous suspension.

#1 Post by TheGreenAnger » Wed Jan 11, 2023 2:29 pm

The Federal Aviation Administration has lifted its suspension on all domestic US flight departures following a major IT failure.

A key system used to notify pilots and ground staff of hazards and alerts suffered a “major failure” earlier today, with FAA engineers frantically scrambling to resolve it after the outage prompted a nationwide ground stop.

The fault lay with the NOTAM (Notice to Air Missions) system, which keeps pilots and other airport staff updated about aviation hazards and airport facilities.

It stopped processing information this morning, forcing a temporary suspension of flights.

However, in its latest statement, the FAA said the ground stop “has been lifted”.

“Normal air traffic operations are resuming gradually across the United States following an overnight outage to the FAA’s Notice to Air Missions (NOTAM) system that provides safety information to flight crews,” it added.

“The agency continues to look into the cause of the initial problem.”
https://www.independent.co.uk/travel/ne ... 60139.html
My necessaries are embark'd: farewell. Adieu! I have too grieved a heart to take a tedious leave.

John Hill
Chief Pilot
Chief Pilot
Posts: 5695
Joined: Sun Aug 23, 2015 7:40 pm
Location: Aotearoa

Re: FAA allows US domestic flights after previous suspension.

#2 Post by John Hill » Wed Jan 11, 2023 7:20 pm

I remember developing NOTAM briefing systems in the '90's and just how shaky the incumbent systems were.

One system we connected to was churning out briefings with parts of NOTAMs being inserted in others. We knew what their problem was but nobody wanted to know! (Data was being corrupted by the incorrect application of the multi-part messaging procedures on the AFTN.)
Been in data comm since we formed the bits individually with a Morse key.

PHXPhlyer
Chief Pilot
Chief Pilot
Posts: 8242
Joined: Sun Jun 17, 2018 2:56 pm
Location: PHX
Gender:
Age: 69

Re: FAA allows US domestic flights after previous suspension.

#3 Post by PHXPhlyer » Wed Jan 11, 2023 11:59 pm

Corrupted file to blame for FAA aviation stoppage that delayed thousands of flights
The system was restored after it forced a halt all departures across the country.

https://www.nbcnews.com/news/us-news/us ... -rcna65243

Flights across the United States resumed Wednesday morning, several hours after the Federal Aviation Administration suffered a computer outage that forced it to halt all departures nationwide while it scrambled to resolve the issue.

The FAA said the crippling delays that affected thousands of flights appear to have been caused by a problem in the Notice to Air Missions (NOTAM) system, which sends pilots vital information they need to fly.

A corrupted file affected both the primary and backup system, a senior government official said Wednesday evening, adding that officials continue to investigate.

"The FAA is continuing a thorough review to determine the root cause of the Notice to Air Missions (NOTAM) system outage," the agency said in a statement. "Our preliminary work has traced the outage to a damaged database file. At this time, there is no evidence of a cyber attack."

Transportation Secretary Pete Buttigieg said earlier in the day that officials had not completely ruled out the possibility of a cyberattack, but so far "there is no direct indication of any kind of external or nefarious activity.”

"One of the questions we need to look at right now, and one of the things I’m asking from FAA, is what’s the state of the art in this form of message traffic?" Buttigieg told NBC News' Andrea Mitchell. "And again, how is it possible for there to be this level of disruption?”

President Joe Biden ordered an investigation after he was briefed on the situation by Buttigieg.

Sen. Maria Cantwell, D-Wa., who heads the Senate Commerce Committee that oversees the FAA, said they too will look into the matter.

“The No. 1 priority is safety," Cantwell said in a statement. "As the Committee prepares for FAA reauthorization legislation, we will be looking into what caused this outage and how redundancy plays a role in preventing future outages. The public needs a resilient air transportation system.”

Meanwhile, the top Republican on that committee, Sen. Ted Cruz of Texas, said the "FAA’s inability to keep an important safety system up and running is completely unacceptable and just the latest example of dysfunction within the Department of Transportation."

The delays came just weeks after Southwest Airlines caused travel chaos by canceling more than 2,500 of its flights during the Christmas season.

Buttigieg, in his interview with Mitchell, acknowledged the frustrations of travelers but said safety always comes first.

“This is an incredibly complex system,” Buttigieg said. “So glitches or complications happen all the time. But we can’t allow them to ever lead to this level of disruption and we won’t ever allow them to lead to a safety problem.”

The FAA lifted the ground stop around 8:50 a.m. and normal air traffic operations began resuming gradually. But by then airports across the country were already crowded with frustrated travelers and a backlog of flights.

As of noon, more than 7,300 flights within, to and out of the U.S. were delayed, according to the online flight tracker FlightAware. More than 1,100 flights were listed as canceled.

Passengers stuck at O'Hare International Airport in Chicago greeted the latest air travel disruption with a collective shrug.

"It kind of is what it is," Teresa Ziakas told NBC News. "I would rather be safe and sound than worry about being delayed by a couple hours."

Traveler Martin Johansen echoed that.

"Planes get delayed," Johansen said as he waited with his wife, Barbara. "That’s just the way life is, so you get used to it."

Any person whose flight was cancelled is entitled to a full refund, DOT guidelines state. And major domestic carriers like American Airlines, United Airlines and Delta said they would waive the fees of travelers who want to rebook flights.

The first sign that this was likely to be a massive incident came around 7:20 a.m. when the FAA sent out a tweet ordering the airlines to pause all domestic departures until 9 a.m. ET “to allow the agency to validate the integrity of flight and safety information" as it worked to restore the NOTAM system.

All flights already in the sky were safe to land, the FAA said.

“Pilots check the NOTAM system before they fly," the agency said. "A Notice to Air Missions alerts pilots about closed runways, equipment outages, and other potential hazards along a flight route or at a location that could affect the flight."

As the drama unfolded, cybersecurity experts told NBC News that the likeliest cause was a bad software update.

“Today’s FAA catastrophic system failure is a clear sign that America’s transportation network desperately needs significant upgrades," U.S. Travel Association President and CEO Geoff Freeman said in a statement. "Americans deserve an end-to-end travel experience that is seamless and secure. And our nation’s economy depends on a best-in-class air travel system.

"We call on federal policymakers to modernize our vital air travel infrastructure to ensure our systems are able to meet demand safely and efficiently," he said.

Buttigieg said in a tweet that he had "been in touch with FAA this morning about an outage affecting a key system for providing safety information to pilots."

United Airlines said earlier it had temporarily delayed all domestic flights while Southwest Airlines said it was "closely monitoring" but warned customers to brace themselves for travel troubles.

Soon, the major airports were doing the same.

"An FAA system outage is causing ground stops at AUS and other airports across the country," the Austin-Bergstrom International Airport said in a tweet.

"Arriving & departing passengers can expect delays this morning & through the day," it said, adding: "Please stay in contact with your airline & check your flight status before heading to AUS."

A number of social media users said they had been affected by the situation.

Heather Allen, 32, was meant to fly from New York City to Seattle with her fiancé to visit her family for a delayed holiday visit. She was watching a movie on her plane and still on the tarmac at John F. Kennedy International Airport when she and other passengers were told to get off their Delta Airlines flight.

She said she learned of the outage by reading the news on Twitter and had been on the plane for about an hour before she had to deplane.

“Trying to be patient, but feeling frustrated,” Allen said. She said the situation at the airport was “not currently chaotic, but could be if delays are longer.”

The flight delays in the U.S. had a ripple effect on some airports abroad. The international airport operator Aéroports de Paris, or Airports of Paris, said all flights by U.S. carriers had been delayed. It said non-American airlines were flying out as normal without interruption.

Air France said all of its U.S.-bound flights were operating as planned and were not affected by the FAA computer outage. It said it continued to monitor the situation.

"As far as we are aware, we are still operating to/from the U.S. at the moment," a spokesperson for Gatwick Airport in London said.

PP

John Hill
Chief Pilot
Chief Pilot
Posts: 5695
Joined: Sun Aug 23, 2015 7:40 pm
Location: Aotearoa

Re: FAA allows US domestic flights after previous suspension.

#4 Post by John Hill » Thu Jan 12, 2023 5:52 am

"Corrupted File" eh? I wonder how that got into the system......................................? It is unfortunate that I cannot find Annex 10 volume that relates to AFTN online!
Been in data comm since we formed the bits individually with a Morse key.

User avatar
Fox3WheresMyBanana
Chief Pilot
Chief Pilot
Posts: 12986
Joined: Thu Sep 03, 2015 9:51 pm
Location: Great White North
Gender:
Age: 61

Re: FAA allows US domestic flights after previous suspension.

#5 Post by Fox3WheresMyBanana » Thu Jan 12, 2023 8:55 pm

A corrupted file is not to blame, since a corrupted file is to be expected.
The corrupted file was the immediate trigger.
The same corrupted file in the backup for such a critical system is to blame.
Now, the question is, how did that happen? The backup is supposed to be (a) reliable and (b) independent from the primary.

PHXPhlyer
Chief Pilot
Chief Pilot
Posts: 8242
Joined: Sun Jun 17, 2018 2:56 pm
Location: PHX
Gender:
Age: 69

Re: FAA allows US domestic flights after previous suspension.

#6 Post by PHXPhlyer » Thu Jan 12, 2023 9:21 pm

The software blamed for FAA outage is three decades old and years from an upgrade, official says
This system, installed in 1993, runs the Notice to Air Missions system, or NOTAM, which sends pilots vital information they need to fly.

https://www.nbcnews.com/news/us-news/so ... -rcna65562

The software that failed and forced the Federal Aviation Administration to ground thousands of flights on Wednesday is 30 years old and not scheduled to be updated for another six years, according to a senior government official.

This system was installed in 1993 and runs the Notice to Air Missions system, or NOTAM, which sends pilots vital information they need to fly, the official said.

After the FAA was able to get planes flying again, a government official said a corrupted file that affected both the primary and the backup NOTAM systems appeared to be the culprit.

But the new revelation raised questions about why the FAA is still relying on software that was introduced the year President Bill Clinton entered the White House.

The FAA is expected to release further details later Thursday.

Earlier, Transportation Secretary Pete Buttigieg said the FAA would pinpoint the source of the problem and identify steps to prevent it from happening again.

President Joe Biden ordered an investigation after he was briefed Wednesday by Buttigieg.

Tens of thousands of travelers were left stranded Wednesday after the FAA sent out a tweet at 7:20 a.m. ordering the airlines to pause all domestic departures until 9 a.m. ET “to allow the agency to validate the integrity of flight and safety information” as it worked to restore the NOTAM system.

The FAA lifted the ground stop around 8:50 a.m., and normal air traffic operations began resuming gradually. But by then airports across the country were already crowded with frustrated travelers and a backlog of flights.

pp

User avatar
Fox3WheresMyBanana
Chief Pilot
Chief Pilot
Posts: 12986
Joined: Thu Sep 03, 2015 9:51 pm
Location: Great White North
Gender:
Age: 61

Re: FAA allows US domestic flights after previous suspension.

#7 Post by Fox3WheresMyBanana » Thu Jan 12, 2023 9:35 pm

Joe Biden ordered an investigation after he was briefed Wednesday by Buttigieg.
The FAA should issue a revised statement:
"The failure of the software was ultimately due to a succession of politicians being complete idiots, with the exceptions of Dubya, Trump, and Biden, who clearly have some bits missing."

John Hill
Chief Pilot
Chief Pilot
Posts: 5695
Joined: Sun Aug 23, 2015 7:40 pm
Location: Aotearoa

Re: FAA allows US domestic flights after previous suspension.

#8 Post by John Hill » Thu Jan 12, 2023 10:15 pm

PHXPhlyer wrote:
Thu Jan 12, 2023 9:21 pm
The software that failed and forced the Federal Aviation Administration to ground thousands of flights on Wednesday is 30 years old and not scheduled to be updated for another six years, according to a senior government official.
Software does not 'age' if anything it should get better with age as 'bugs' are found and squished. However software can deteriorate if poorly skilled persons introduce 'bugs'.

The 'corrupted' file almost certainly came from outside the system and could have originated anywhere in the world, just keep in mind that the old AFTN still goes to places you cannot call on your phone.

The problem is not that a file was corrupted, the problem is that the system could not detect and handle corrupted data.
Been in data comm since we formed the bits individually with a Morse key.

User avatar
llondel
Chief Pilot
Chief Pilot
Posts: 5913
Joined: Wed Oct 03, 2018 3:17 am
Location: San Jose

Re: FAA allows US domestic flights after previous suspension.

#9 Post by llondel » Fri Jan 13, 2023 1:01 am

I've always taken the approach that if something asks for a number 1-5, I'll always try 0, or 6, or Q or #, just to see what happens. I also now try a period '.' because if it's an input that can parse decimal numbers, that's a valid part of a number. Same with + and -. I have fond memories of breaking a "tested" system with the latter pair of symbols. Entirely by accident the first time, I was trying to enter a 0 and missed. The same with text file input, ideally it should be validated well enough to reject something bad, not crash.

User avatar
Fox3WheresMyBanana
Chief Pilot
Chief Pilot
Posts: 12986
Joined: Thu Sep 03, 2015 9:51 pm
Location: Great White North
Gender:
Age: 61

Re: FAA allows US domestic flights after previous suspension.

#10 Post by Fox3WheresMyBanana » Fri Jan 13, 2023 2:11 am

The few things I've written were all for 'amateur' users. The error checking was around half of the program. I remember building one little routine to go through all the names looking for likely mis-spellings of (especially foreign) names. Drop-down boxes are great.

User avatar
llondel
Chief Pilot
Chief Pilot
Posts: 5913
Joined: Wed Oct 03, 2018 3:17 am
Location: San Jose

Re: FAA allows US domestic flights after previous suspension.

#11 Post by llondel » Fri Jan 13, 2023 2:41 am

Fox3WheresMyBanana wrote:
Fri Jan 13, 2023 2:11 am
The few things I've written were all for 'amateur' users. The error checking was around half of the program. I remember building one little routine to go through all the names looking for likely mis-spellings of (especially foreign) names. Drop-down boxes are great.
I put together a small program for use at work, which uses a database as a back end. For certain fields it creates a drop-down from what's already been used in the database to try to avoid typos, but you can type in a brand new entry if needed, and next time it too will appear in the drop-down list. When it's names of people or projects, it's definitely useful to avoid typos.

John Hill
Chief Pilot
Chief Pilot
Posts: 5695
Joined: Sun Aug 23, 2015 7:40 pm
Location: Aotearoa

Re: FAA allows US domestic flights after previous suspension.

#12 Post by John Hill » Fri Jan 13, 2023 4:38 am

I notice American sites referring to the NOTAM Code (the 'Q' and four letters code) as the 'International Code'? Does American not use the NOTAM Code and if they do have they only recently adopted it?
Been in data comm since we formed the bits individually with a Morse key.

User avatar
unifoxos
Capt
Capt
Posts: 959
Joined: Mon Aug 31, 2015 10:36 am
Location: Twycross Zoo, or thereabouts
Gender:
Age: 78

Re: FAA allows US domestic flights after previous suspension.

#13 Post by unifoxos » Fri Jan 13, 2023 10:14 am

The first sign that this was likely to be a massive incident came around 7:20 a.m. when the FAA sent out a tweet ordering the airlines to pause all domestic departures until 9 a.m

Does one have to sign up to Twatter to fly in the US these days?
(Or to not fly?)
Sent from my tatty old Windoze PC.

Boac
Chief Pilot
Chief Pilot
Posts: 17209
Joined: Fri Aug 28, 2015 5:12 pm
Location: Here

Re: FAA allows US domestic flights after previous suspension.

#14 Post by Boac » Fri Jan 13, 2023 10:33 am

One of the problems of this 'modern' society is that a lot of 'important' news is distributed via 'social media' including government policy in the UK!

User avatar
Woody
Chief Pilot
Chief Pilot
Posts: 10245
Joined: Tue Aug 25, 2015 6:33 pm
Location: Sir Kenny Dalglish Stand
Age: 59

Re: FAA allows US domestic flights after previous suspension.

#15 Post by Woody » Fri Jan 13, 2023 10:35 am

Here’s the DM version, I’ve not proofread it yet for the usual errors :D

https://www.dailymail.co.uk/news/articl ... ineer.html
When all else fails, read the instructions.

John Hill
Chief Pilot
Chief Pilot
Posts: 5695
Joined: Sun Aug 23, 2015 7:40 pm
Location: Aotearoa

Re: FAA allows US domestic flights after previous suspension.

#16 Post by John Hill » Fri Jan 13, 2023 5:56 pm

Woody wrote:
Fri Jan 13, 2023 10:35 am
Here’s the DM version, I’ve not proofread it yet for the usual errors :D
Yah, blame the 'unknown engineer' rather than admit the truth.
Been in data comm since we formed the bits individually with a Morse key.

PHXPhlyer
Chief Pilot
Chief Pilot
Posts: 8242
Joined: Sun Jun 17, 2018 2:56 pm
Location: PHX
Gender:
Age: 69

Re: FAA allows US domestic flights after previous suspension.

#17 Post by PHXPhlyer » Fri Jan 20, 2023 3:05 am

FAA says problem that grounded flights happened when files were unintentionally deleted
https://www.nbcnews.com/news/us-news/fa ... -rcna66649

The FAA halted all departures nationwide Jan. 11 after an outage to the Notice to Air Missions (NOTAM) system.\\
Last week’s massive flight disruption that grounded thousands of planes was caused after “contract personnel unintentionally deleted files,” the Federal Aviation Administration said Thursday.

On Jan. 11, the FAA halted all departures nationwide after an outage to the Notice to Air Missions (NOTAM) system.

The FAA said in a statement that the files were deleted by accident as the contractors were "working to correct synchronization between the live primary database and a backup database."

The review was called preliminary, and the incident is still under investigation. The FAA said it has found no evidence of a cyberattack or malicious intent.

The NOTAM system sends pilots vital information they need to fly. The system was restored later that day.

On Jan. 11, the FAA sent out a tweet at 7:20 a.m. ordering the airlines to pause all domestic departures until 9 a.m.

The FAA lifted the ground stop around 8:50 a.m., and normal air traffic operations began resuming gradually.

The aviation agency said Thursday that repairs have been made to make the system more resilient, and that it will adopt other steps after the incident.

PP

Post Reply