Human technical errors.... confessions.

General Chit Chat
Post Reply
Message
Author
TheGreenAnger
Chief Pilot
Chief Pilot
Posts: 3286
Joined: Fri Jul 15, 2022 11:40 pm
Location: Unfashionable end of the Western Spiral

Human technical errors.... confessions.

#1 Post by TheGreenAnger » Fri Jan 13, 2023 1:44 pm

I was reading Woody's entry to the FAA computer systems debacle yesterday viewtopic.php?p=357587#p357587 and was apt to grimace and think "there by the grace of God".

It is very easy to make this kind of blunder and sad to say, something of the same sort of category of error was made by me not two weeks ago while doing routine system security maintenance on the database infrastructure my company runs on behalf of a very big client, where I accidently disabled my own Administrator access to the core client database.

Fortunately this did not impact the client whose access was unimpeded but for an hour, before I had managed to write some server recovery scripts, the system was totally inaccessible to me, the guardian of the database. :-? @-)

The joke of the situation was that the master admin system administrator login had purposely been disabled after an external security company had recommend it be disabled in the name of security! Had it not been for the fact that there was a programmatic way for me to re-enable this master login I would have had to request client downtime to restore the database backup and reconfigure the whole database instance causing at least half a day's client outage. One lives and learns.

This could have been the biggest human technical error I have ever made, second only to the time I managed to cut the engine of the Piper Tomahawk I was flying while leaning the engine. It all went quiet until the engine sputtered back into life after the nose was pushed down to keep the prop turning and the mixture gently restored. Nobody with me to note my bumbling mistake until my mea culpa today.

What dreadful technical mistakes have you made and got away with?
My necessaries are embark'd: farewell. Adieu! I have too grieved a heart to take a tedious leave.

User avatar
llondel
Chief Pilot
Chief Pilot
Posts: 5912
Joined: Wed Oct 03, 2018 3:17 am
Location: San Jose

Re: Human technical errors.... confessions.

#2 Post by llondel » Fri Jan 13, 2023 2:12 pm

While it does add a potential security weakness, creating a "spare" admin account that you don't normally use can go a long way to mitigating the lockout problem. It's the sort of thing where you write the account name and password on a piece of paper, put it in a sealed envelope and place the envelope in the safe. If you get run over by a bus, or there's some other good reason, the envelope can be extracted and the details used to gain system access. Then someone changes the password, writes the details on a new piece of paper and the whole thing is set up again.

Rossian
Capt
Capt
Posts: 963
Joined: Fri Sep 04, 2015 1:27 pm
Location: Morayshire Scotland
Gender:
Age: 82

Re: #2.

#3 Post by Rossian » Fri Jan 13, 2023 3:32 pm

...is that the same as Eccles* having the time written on a piece of paper so that he can always tell someone else the time, should they ask?

The Ancient Mariner
* for the under 70s age group, a character in a radio comedy programme "The Goon Show"

TheGreenAnger
Chief Pilot
Chief Pilot
Posts: 3286
Joined: Fri Jul 15, 2022 11:40 pm
Location: Unfashionable end of the Western Spiral

Re: #2.

#4 Post by TheGreenAnger » Fri Jan 13, 2023 4:18 pm

Rossian wrote:
Fri Jan 13, 2023 3:32 pm
...is that the same as Eccles* having the time written on a piece of paper so that he can always tell someone else the time, should they ask?

The Ancient Mariner
* for the under 70s age group, a character in a radio comedy programme "The Goon Show"
We keep all our passwords encrypted in an online bit-locker that uses either password and/or and biometric credentials for access, so in principle we should not be able to lock ourselves out of that but, hey, who knows! ;)))

What say you Seagoon? :)
My necessaries are embark'd: farewell. Adieu! I have too grieved a heart to take a tedious leave.

User avatar
llondel
Chief Pilot
Chief Pilot
Posts: 5912
Joined: Wed Oct 03, 2018 3:17 am
Location: San Jose

Re: #2.

#5 Post by llondel » Fri Jan 13, 2023 4:35 pm

TheGreenAnger wrote:
Fri Jan 13, 2023 4:18 pm
We keep all our passwords encrypted in an online bit-locker that uses either password and/or and biometric credentials for access, so in principle we should not be able to lock ourselves out of that but, hey, who knows! ;)))

What say you Seagoon? :)
Until you need access and it's offline for some reason. Having several local copies of the database may be a better solution, including one on a USB stick so you can take it with you and also provide insurance against corruption/deletion of the on-line stuff.

TheGreenAnger
Chief Pilot
Chief Pilot
Posts: 3286
Joined: Fri Jul 15, 2022 11:40 pm
Location: Unfashionable end of the Western Spiral

Re: #2.

#6 Post by TheGreenAnger » Fri Jan 13, 2023 4:53 pm

llondel wrote:
Fri Jan 13, 2023 4:35 pm
TheGreenAnger wrote:
Fri Jan 13, 2023 4:18 pm
We keep all our passwords encrypted in an online bit-locker that uses either password and/or and biometric credentials for access, so in principle we should not be able to lock ourselves out of that but, hey, who knows! ;)))

What say you Seagoon? :)
Until you need access and it's offline for some reason. Having several local copies of the database may be a better solution, including one on a USB stick so you can take it with you and also provide insurance against corruption/deletion of the on-line stuff.
Sensible stuff and yes, I do keep a copy which is accessible offline on my phone and my laptop! :-bd

While I was contemplating all this I did remember the awful Friday back in 1987 when I managed to delete all the payroll slips intended for the weekly paid people on the assembly line at the automotive electronics company where I worked as an RPG programmer on the MAPICS system running on the IBM System 38. I had to admit my gaffe to the financial director, and then write a gash program to re-open the accounting period and to reprint the slips without reposting to the general ledger again. I had 70 minutes to get this done, and managed to do it and save my job. Talk about self-induced stress! I think, in the end, my gaffe, and subsequent honesty, endeared me to the director who decided that I might be useful after all, having not crumbled under the weight of the threat of dismissal, and he took a personal interest in my career thereafter and he helped me quite considerably up the career ladder later on!
My necessaries are embark'd: farewell. Adieu! I have too grieved a heart to take a tedious leave.

John Hill
Chief Pilot
Chief Pilot
Posts: 5695
Joined: Sun Aug 23, 2015 7:40 pm
Location: Aotearoa

Re: Human technical errors.... confessions.

#7 Post by John Hill » Fri Jan 13, 2023 6:18 pm

The phone was ringing as I walked in the door one morning, back in a previous century, the caller was in another country and he was fair ropeable!

The gist of his message was that our software installed at an airline in his country was 'crashing his message switching centre!'.

Apparently my validation and message reject software was better than his...
Been in data comm since we formed the bits individually with a Morse key.

TheGreenAnger
Chief Pilot
Chief Pilot
Posts: 3286
Joined: Fri Jul 15, 2022 11:40 pm
Location: Unfashionable end of the Western Spiral

Re: Human technical errors.... confessions.

#8 Post by TheGreenAnger » Fri Jan 13, 2023 6:36 pm

I see that apart from the FAA METAR issue noted yesterday there were issues in Canada as well although not attributable to the same root cause!
“We are still investigating the root cause of the failure. At this time, we do not believe the cause is related to the FAA outage experienced earlier today.”
https://globalnews.ca/news/9402058/u-s- ... er-outage/

Maybe the Canadians are still using your system John? =)) ;)))
My necessaries are embark'd: farewell. Adieu! I have too grieved a heart to take a tedious leave.

John Hill
Chief Pilot
Chief Pilot
Posts: 5695
Joined: Sun Aug 23, 2015 7:40 pm
Location: Aotearoa

Re: Human technical errors.... confessions.

#9 Post by John Hill » Fri Jan 13, 2023 6:46 pm

TheGreenAnger wrote:
Fri Jan 13, 2023 6:36 pm
I see that apart from the FAA METAR issue noted yesterday there were issues in Canada as well although not attributable to the same root cause!
Probably the same 'corrupted' NOTAM from Bongo-Bonkersland warning of people sleeping on the runways in the cooler weather.
Maybe the Canadians are still using your system John? =)) ;)))
If they are I would like to know the address of their payments officer.
Been in data comm since we formed the bits individually with a Morse key.

User avatar
OFSO
Chief Pilot
Chief Pilot
Posts: 18600
Joined: Sat Aug 22, 2015 6:39 pm
Location: Teddington UK and Roses Catalunia
Gender:
Age: 80

Re: Human technical errors.... confessions.

#10 Post by OFSO » Fri Jan 13, 2023 7:06 pm

As a witless T2A I was sitting up on a pair of wooden steps, Routining the ancient amplifiers in Free Lane Repeater station when I picked up the wrong front case of one multiple (valve) amplifier and tried to replace it on another. It blew the fuses for that rack and the surrounding racks, disconnecting Leicester from the world, and earning me a telling-off. In hindsight dismissal would have been fitting.

User avatar
4mastacker
Chief Pilot
Chief Pilot
Posts: 5141
Joined: Sun Aug 23, 2015 5:38 pm
Location: With the wife
Gender:
Age: 76

Re: Human technical errors.... confessions.

#11 Post by 4mastacker » Fri Jan 13, 2023 7:40 pm

OFSO wrote:
Fri Jan 13, 2023 7:06 pm
....... disconnecting Leicester from the world, .....
You make that sound like it is a bad thing. =))
It's always my fault - SWMBO

TheGreenAnger
Chief Pilot
Chief Pilot
Posts: 3286
Joined: Fri Jul 15, 2022 11:40 pm
Location: Unfashionable end of the Western Spiral

Re: Human technical errors.... confessions.

#12 Post by TheGreenAnger » Fri Jan 13, 2023 9:15 pm

4mastacker wrote:
Fri Jan 13, 2023 7:40 pm
OFSO wrote:
Fri Jan 13, 2023 7:06 pm
....... disconnecting Leicester from the world, .....
You make that sound like it is a bad thing. =))
I did think it too! =))

I shouldn't laugh, the English half sisters all come from Oadby! =))
My necessaries are embark'd: farewell. Adieu! I have too grieved a heart to take a tedious leave.

jimtherev
Station Padre
Station Padre
Posts: 1417
Joined: Sun Aug 23, 2015 4:40 pm
Location: A country mouse in Derbyshire
Gender:
Age: 85

Re: Human technical errors.... confessions.

#13 Post by jimtherev » Fri Jan 13, 2023 11:47 pm

I used to boast that if anything needed controlling, I could control it. (Mrs Jim might not agree, but still...) I guess my proudest moment was when a system I designed didn't fail to safety, which resulted in 43m galls of raw sewage being discharged into the Mersey back in the sixties.
Strange thing was, no-one noticed until Monday morning when the men with the clipboards came back to work.

User avatar
Woody
Chief Pilot
Chief Pilot
Posts: 10244
Joined: Tue Aug 25, 2015 6:33 pm
Location: Sir Kenny Dalglish Stand
Age: 59

Re: Human technical errors.... confessions.

#14 Post by Woody » Fri Jan 13, 2023 11:50 pm

When all else fails, read the instructions.

TheGreenAnger
Chief Pilot
Chief Pilot
Posts: 3286
Joined: Fri Jul 15, 2022 11:40 pm
Location: Unfashionable end of the Western Spiral

Re: Human technical errors.... confessions.

#15 Post by TheGreenAnger » Fri Jan 13, 2023 11:51 pm

jimtherev wrote:
Fri Jan 13, 2023 11:47 pm
I used to boast that if anything needed controlling, I could control it. (Mrs Jim might not agree, but still...) I guess my proudest moment was when a system I designed didn't fail to safety, which resulted in 43m galls of raw sewage being discharged into the Mersey back in the sixties.
Strange thing was, no-one noticed until Monday morning when the men with the clipboards came back to work.
So nobody raised a stink about it? ;)))

At first! :-o
My necessaries are embark'd: farewell. Adieu! I have too grieved a heart to take a tedious leave.

Post Reply