Human technical errors.... confessions.
-
- Chief Pilot
- Posts: 3286
- Joined: Fri Jul 15, 2022 11:40 pm
- Location: Unfashionable end of the Western Spiral
Human technical errors.... confessions.
I was reading Woody's entry to the FAA computer systems debacle yesterday viewtopic.php?p=357587#p357587 and was apt to grimace and think "there by the grace of God".
It is very easy to make this kind of blunder and sad to say, something of the same sort of category of error was made by me not two weeks ago while doing routine system security maintenance on the database infrastructure my company runs on behalf of a very big client, where I accidently disabled my own Administrator access to the core client database.
Fortunately this did not impact the client whose access was unimpeded but for an hour, before I had managed to write some server recovery scripts, the system was totally inaccessible to me, the guardian of the database.
The joke of the situation was that the master admin system administrator login had purposely been disabled after an external security company had recommend it be disabled in the name of security! Had it not been for the fact that there was a programmatic way for me to re-enable this master login I would have had to request client downtime to restore the database backup and reconfigure the whole database instance causing at least half a day's client outage. One lives and learns.
This could have been the biggest human technical error I have ever made, second only to the time I managed to cut the engine of the Piper Tomahawk I was flying while leaning the engine. It all went quiet until the engine sputtered back into life after the nose was pushed down to keep the prop turning and the mixture gently restored. Nobody with me to note my bumbling mistake until my mea culpa today.
What dreadful technical mistakes have you made and got away with?
It is very easy to make this kind of blunder and sad to say, something of the same sort of category of error was made by me not two weeks ago while doing routine system security maintenance on the database infrastructure my company runs on behalf of a very big client, where I accidently disabled my own Administrator access to the core client database.
Fortunately this did not impact the client whose access was unimpeded but for an hour, before I had managed to write some server recovery scripts, the system was totally inaccessible to me, the guardian of the database.
The joke of the situation was that the master admin system administrator login had purposely been disabled after an external security company had recommend it be disabled in the name of security! Had it not been for the fact that there was a programmatic way for me to re-enable this master login I would have had to request client downtime to restore the database backup and reconfigure the whole database instance causing at least half a day's client outage. One lives and learns.
This could have been the biggest human technical error I have ever made, second only to the time I managed to cut the engine of the Piper Tomahawk I was flying while leaning the engine. It all went quiet until the engine sputtered back into life after the nose was pushed down to keep the prop turning and the mixture gently restored. Nobody with me to note my bumbling mistake until my mea culpa today.
What dreadful technical mistakes have you made and got away with?
My necessaries are embark'd: farewell. Adieu! I have too grieved a heart to take a tedious leave.
Re: Human technical errors.... confessions.
While it does add a potential security weakness, creating a "spare" admin account that you don't normally use can go a long way to mitigating the lockout problem. It's the sort of thing where you write the account name and password on a piece of paper, put it in a sealed envelope and place the envelope in the safe. If you get run over by a bus, or there's some other good reason, the envelope can be extracted and the details used to gain system access. Then someone changes the password, writes the details on a new piece of paper and the whole thing is set up again.
-
- Capt
- Posts: 963
- Joined: Fri Sep 04, 2015 1:27 pm
- Location: Morayshire Scotland
- Gender:
- Age: 82
Re: #2.
...is that the same as Eccles* having the time written on a piece of paper so that he can always tell someone else the time, should they ask?
The Ancient Mariner
* for the under 70s age group, a character in a radio comedy programme "The Goon Show"
The Ancient Mariner
* for the under 70s age group, a character in a radio comedy programme "The Goon Show"
-
- Chief Pilot
- Posts: 3286
- Joined: Fri Jul 15, 2022 11:40 pm
- Location: Unfashionable end of the Western Spiral
Re: #2.
We keep all our passwords encrypted in an online bit-locker that uses either password and/or and biometric credentials for access, so in principle we should not be able to lock ourselves out of that but, hey, who knows!
What say you Seagoon?
My necessaries are embark'd: farewell. Adieu! I have too grieved a heart to take a tedious leave.
Re: #2.
Until you need access and it's offline for some reason. Having several local copies of the database may be a better solution, including one on a USB stick so you can take it with you and also provide insurance against corruption/deletion of the on-line stuff.TheGreenAnger wrote: ↑Fri Jan 13, 2023 4:18 pmWe keep all our passwords encrypted in an online bit-locker that uses either password and/or and biometric credentials for access, so in principle we should not be able to lock ourselves out of that but, hey, who knows!
What say you Seagoon?
-
- Chief Pilot
- Posts: 3286
- Joined: Fri Jul 15, 2022 11:40 pm
- Location: Unfashionable end of the Western Spiral
Re: #2.
Sensible stuff and yes, I do keep a copy which is accessible offline on my phone and my laptop!llondel wrote: ↑Fri Jan 13, 2023 4:35 pmUntil you need access and it's offline for some reason. Having several local copies of the database may be a better solution, including one on a USB stick so you can take it with you and also provide insurance against corruption/deletion of the on-line stuff.TheGreenAnger wrote: ↑Fri Jan 13, 2023 4:18 pmWe keep all our passwords encrypted in an online bit-locker that uses either password and/or and biometric credentials for access, so in principle we should not be able to lock ourselves out of that but, hey, who knows!
What say you Seagoon?
While I was contemplating all this I did remember the awful Friday back in 1987 when I managed to delete all the payroll slips intended for the weekly paid people on the assembly line at the automotive electronics company where I worked as an RPG programmer on the MAPICS system running on the IBM System 38. I had to admit my gaffe to the financial director, and then write a gash program to re-open the accounting period and to reprint the slips without reposting to the general ledger again. I had 70 minutes to get this done, and managed to do it and save my job. Talk about self-induced stress! I think, in the end, my gaffe, and subsequent honesty, endeared me to the director who decided that I might be useful after all, having not crumbled under the weight of the threat of dismissal, and he took a personal interest in my career thereafter and he helped me quite considerably up the career ladder later on!
My necessaries are embark'd: farewell. Adieu! I have too grieved a heart to take a tedious leave.
Re: Human technical errors.... confessions.
The phone was ringing as I walked in the door one morning, back in a previous century, the caller was in another country and he was fair ropeable!
The gist of his message was that our software installed at an airline in his country was 'crashing his message switching centre!'.
Apparently my validation and message reject software was better than his...
The gist of his message was that our software installed at an airline in his country was 'crashing his message switching centre!'.
Apparently my validation and message reject software was better than his...
Been in data comm since we formed the bits individually with a Morse key.
-
- Chief Pilot
- Posts: 3286
- Joined: Fri Jul 15, 2022 11:40 pm
- Location: Unfashionable end of the Western Spiral
Re: Human technical errors.... confessions.
I see that apart from the FAA METAR issue noted yesterday there were issues in Canada as well although not attributable to the same root cause!
Maybe the Canadians are still using your system John?
https://globalnews.ca/news/9402058/u-s- ... er-outage/“We are still investigating the root cause of the failure. At this time, we do not believe the cause is related to the FAA outage experienced earlier today.”
Maybe the Canadians are still using your system John?
My necessaries are embark'd: farewell. Adieu! I have too grieved a heart to take a tedious leave.
Re: Human technical errors.... confessions.
Probably the same 'corrupted' NOTAM from Bongo-Bonkersland warning of people sleeping on the runways in the cooler weather.TheGreenAnger wrote: ↑Fri Jan 13, 2023 6:36 pmI see that apart from the FAA METAR issue noted yesterday there were issues in Canada as well although not attributable to the same root cause!
If they are I would like to know the address of their payments officer.Maybe the Canadians are still using your system John?
Been in data comm since we formed the bits individually with a Morse key.
- OFSO
- Chief Pilot
- Posts: 18600
- Joined: Sat Aug 22, 2015 6:39 pm
- Location: Teddington UK and Roses Catalunia
- Gender:
- Age: 80
Re: Human technical errors.... confessions.
As a witless T2A I was sitting up on a pair of wooden steps, Routining the ancient amplifiers in Free Lane Repeater station when I picked up the wrong front case of one multiple (valve) amplifier and tried to replace it on another. It blew the fuses for that rack and the surrounding racks, disconnecting Leicester from the world, and earning me a telling-off. In hindsight dismissal would have been fitting.
- 4mastacker
- Chief Pilot
- Posts: 5141
- Joined: Sun Aug 23, 2015 5:38 pm
- Location: With the wife
- Gender:
- Age: 76
Re: Human technical errors.... confessions.
You make that sound like it is a bad thing.
It's always my fault - SWMBO
-
- Chief Pilot
- Posts: 3286
- Joined: Fri Jul 15, 2022 11:40 pm
- Location: Unfashionable end of the Western Spiral
Re: Human technical errors.... confessions.
I did think it too!
I shouldn't laugh, the English half sisters all come from Oadby!
My necessaries are embark'd: farewell. Adieu! I have too grieved a heart to take a tedious leave.
-
- Station Padre
- Posts: 1417
- Joined: Sun Aug 23, 2015 4:40 pm
- Location: A country mouse in Derbyshire
- Gender:
- Age: 85
Re: Human technical errors.... confessions.
I used to boast that if anything needed controlling, I could control it. (Mrs Jim might not agree, but still...) I guess my proudest moment was when a system I designed didn't fail to safety, which resulted in 43m galls of raw sewage being discharged into the Mersey back in the sixties.
Strange thing was, no-one noticed until Monday morning when the men with the clipboards came back to work.
Strange thing was, no-one noticed until Monday morning when the men with the clipboards came back to work.
- Woody
- Chief Pilot
- Posts: 10244
- Joined: Tue Aug 25, 2015 6:33 pm
- Location: Sir Kenny Dalglish Stand
- Age: 59
Re: Human technical errors.... confessions.
When all else fails, read the instructions.
-
- Chief Pilot
- Posts: 3286
- Joined: Fri Jul 15, 2022 11:40 pm
- Location: Unfashionable end of the Western Spiral
Re: Human technical errors.... confessions.
So nobody raised a stink about it?jimtherev wrote: ↑Fri Jan 13, 2023 11:47 pmI used to boast that if anything needed controlling, I could control it. (Mrs Jim might not agree, but still...) I guess my proudest moment was when a system I designed didn't fail to safety, which resulted in 43m galls of raw sewage being discharged into the Mersey back in the sixties.
Strange thing was, no-one noticed until Monday morning when the men with the clipboards came back to work.
At first!
My necessaries are embark'd: farewell. Adieu! I have too grieved a heart to take a tedious leave.