Blackout? Fire? Police? Absolutely crazy stories, that happened in undisclosed tech companies!

#programming #tech #softwaredevelopment

0:17 Blackout
4:00 Fire
7:26 Incident

Imagine this: you come to the office and sit at your desk, pop open your laptop ,boot up your terminal, only to find out that your screen is black. There is no text, no prompt nothing, just empty black terminal. You type in the keys but nothing happens. It's just not working. Pop open another tab and the same: thing it's just empty, blank. You're kind of stumped: what could be the reason for it?
You open up Google, you browse around, you search for an answer, but nothing really works.

All the settings are correct, everything seems to be in place, and on top of that: you didn't change anything that could affect it. So why why would you have a black terminal black console? It's actually key to start your working day, so you need to address this. And then you discover, that on the screen of your colleague there is a Google search opened: "why my terminal is black?"

We are onto something. This is not only your issue.
It's shared. You stand up and ask in the open space: who else has black's terminal?
What's going on? There are a few people: it's not only you and your colleague.
It's actually like a pretty major thing, but there are also some people who don't have black terminals because they use white backgrounds.

So you do a little bit of investigation and then you remember, that you have this new colleague that recently joined the company and likes to do things a little bit different.
You bring this question to him: why would I have black terminal on my laptop this morning?

He goes: did I do that?
Then you go: did you?
Turns out that your colleague decided to edit a shared config file that sets environment variables for everybody in the company.

And let me tell you: the company wasn't like 15 employees big - it was thousands and thousands of employees big. Right now everybody in the company uses lines that he added to the shared config. It was just something that was openly accessible in some remote machine that was just sourced as part of your setup script for some reason.

Whoever was responsible for managing those resources didn't block other people from editing those. Maybe he thought it's useful to have it shared. It’s never a good idea. Just don't do it.

He decided to spice things up a little bit because he didn't like the dull dark black background - instead he chose to have a different background every time he opens up his terminal. He popped open the setup and just added new lines that would set colours for him. He just failed to realize that everybody in the office who was using black background also had black text because it was part of the setup in it in its script.

Luckily we could just quickly reverse this and teach this colleague never ever to edit any config files on his own.

Another story: imagine that your company has specialised laboratories full of very very expensive equipment. Let's say that just one unit of this expensive equipment is worth like two or three yearly salaries of an average employee of that company. I don't want to reveal any details, not to put anybody in harm's way, but let's just say that this device was supposed to connect to other smaller devices and power them up for some internal testing.

You know how it is when you work in a huge corporation right? People who actually set up the equipment physically have to be in the same room, but team that actually configures the whole thing could be outsourced somewhere else as long as you can access them with an online interface.

Imagine that one day there is a fire alarm in the company and turns out that the issue is in the lab. The technician goes there only to discover that one of their units is actually on fire. Without thinking too much he just pulls the fire extinguisher of the wall, sprays the whole thing before the actual full-blown system kicks in and showers everything with a few meters of thick foam.

He manages to resolve the issue right there. The device gets disconnected - it's now safe to get in and like actually investigate what happened. It was very hard to actually discover any fault in the hardware itself because as it seems nothing was wrong, nothing was broken.

What they did is: they actually brought in an external consultancy firm to investigate the fire and figure out what was the reason. Why was the thing burning? These guys couldn't actually figure out what exactly was the issue, but what they did tell us is well the fire was caused by a power spike.