Viewing a single comment thread. View all comments

Spartanfred104 t1_jaa3kr3 wrote

Every story I read about these types of incidents just feeds into large corporations not spending money in IT, every damn time.

Record profits but skeleton tech support, they get what they paid for.

492

feuerwehrmann t1_jaa5xa3 wrote

Businesses look at IT as a line item like putting tp in the bathroom. Cheap as possible

213

Postnarcissim t1_jabdt2y wrote

We didn’t “produce revenue” in the NOC, but routinely solved multi million dollar outages before they happened, or after the lack of an IT department at the customer end caused an outage.

It was hard to get a raise, you only got yourself promoted out of it.

But you were the first person they called. I had three screens and 2 laptops and the all of a sudden I’d have 20+ IMs asking about this or that outage while I’m working to solve it.

Everyone wanted personal updates along with the actual updates and expected it right fucking now.

Meanwhile I’ve got Suzy on the line who I’ve asked repeatedly to check if her desktop is even plugged in while I trouble shoot a fiber break and a bad router or NAS who’s disk broke and is now filling up cloud storage.

I will never go back.

73

LaJolla86 t1_jabjq96 wrote

There was a time I was going to make myself the NOC manager (I wrote and managed all our NOC software and Splunk dashboards). Then I realized it would have been one of the most thankless jobs while still having people to manage; also being the first point of major business contacts for big outages.

I quit shortly after. I had never even had a vacation until that point in 10 years.

16

Postnarcissim t1_jabk278 wrote

I ran into an old coworker who was now manager of the NOC a couple years back and he offered me my old Tier 3 gig back ( I would’ve failed, been out for 5 years and all certs had lapsed) with a raise and I turned him down so fast it wasn’t funny.

I was the de facto on duty Incident Manager, they wouldn’t allow me to move into that role full time (with a 6 figure raise btw) so yeah. You have to really like abuse or the NOC to deal with it.

I occasionally think to myself maybe I’d like to go back to IT then I realize no matter how bad my life gets, it’s bette than working in IT.

13

sip487 t1_jachskr wrote

I’ve been working in NOC’s for years but only in telecom and although it’s stress full I fucking love the NOC 4 day work week and everyone leaves you alone if nothing is broken.

2

mightychobo t1_jacxcum wrote

Bro did we work for the same company? I feel like these stories come from the people who I worked right next too.

5

Postnarcissim t1_jad38g0 wrote

I think it’s just the same story in all NOCs. Sounds like there’s some unicorns but I’d bet those are few and far between.

2

HeavensCriedBlood t1_jaa8kbw wrote

If they could buy an entire IT department on craigslist, they would.

63

chaogomu t1_jaaq3za wrote

Pass IT off to that unpaid intern they conned into working during the summer between college semesters. The business administration major intern.

Or use the CEO's spoiled brat as the head of IT because the kid "knows computers".

52

MaximumLunchbox t1_jad35h5 wrote

Look, he installed Google Ultron for me, it's what the NSA uses!

5

panormda t1_jab17l1 wrote

They can! Now with IT As A Service (r) lol

7

LaJolla86 t1_jaec6hj wrote

Indian Craigslist. Do the needful. Kindly revert.

3

sonic_butthole_music t1_jabmt0d wrote

That’s part of the issue but it’s also a result of changing tactics. A whole ecosystem of hacker for hire companies have sprung up and among them are initial access brokers. They gain access to a network and often sit for months, slowly expanding access and collecting information to sell to other hackers for them to exploit. A few years ago the average time between an attacker gaining a foothold and exploitation was 5 hours. It’s grown to 9 months today.

7

BigBadBinky t1_jacpnac wrote

This is new info to me. Maybe we should do a few more backups

1

pleachchapel t1_jadiy7t wrote

I heard it described this way once: IT is not a value creator, it is a value multiplier. That works in both directions. Shit IT can eliminate the most productive employee's contributions, & the proper wizarding department can automate a ton of hair pulling to let your employees do what they do best.

1

feuerwehrmann t1_jadkghb wrote

It is a shame how some companies don't consider IT to be an asset. There are a number of places where off the shelf and consultants rule the land and they then wonder why the hell it is difficult to get a simple task done

2

blorpianblorp t1_jaamv9z wrote

Quite a few businesses treat IT as a thankless entity. Instead of a thank you it's "hey the environment is stable and everyone can work, why do we need to spend money on IT?"

Followed by "why the fuck is there an outage again? What do we even pay IT for?"

47

Cakeking7878 t1_jab0iyf wrote

“The building is fine, it will never catch on Fire, we don’t need a sprinkler system and we don’t need to pay taxes for a fire department”

“Why the fuck is no one coming to put out the fire??”

22

Leege13 t1_jab6qv5 wrote

Literally the plot of The Towering Inferno except that the fire department did show up that time.

9

PromiscuousMNcpl t1_jab0rhj wrote

“Everything is always broken, why do we even pay you guys?”

“Everything is running fine, why do we even pay all you guys?”

26

9-11GaveMe5G t1_jaayk6q wrote

"tech debt" every company has it, some have so much it could sink them

15

gramsaran t1_jabirb1 wrote

As an IT employee, it's actually highly common and hilarious at this point.

8

AltCtrlShifty t1_jacmbse wrote

I found out my company, years ago, had a public facing web page that executed a MySQL query, on a primary server with no backup, sent in the url string. (?query=xxx)

4

Crizbibble t1_jabs4az wrote

IT is so low priority it’s sickening. I left corporate IT a couple years ago to start my own shop but I failed because nobody will pay me enough for the work I do. They try to become your friend and then expect you to do free work for endless hours while paying half or a quarter of your invoice. If you walk in the door you get hit by 20 crying minions begging you to fix everything under the sun and by the time you get to the work you are there for you are burnt out already cause you know it was all for free. I love IT and been building systems since 1977 but it’s one of the worst jobs there is in corporate America. Thankless and you are blamed for everything under the sun plus all the fake friends you make along the way. People suck.

3

pzerr t1_jact5jd wrote

Agree on the value they assign it. Is too low.

Knowing a couple of incidents first hand, this is going to be a very difficult problem to fix. The biggest issue is that IT is IT's worst enemy. Unlike most departments, management has to fully rely on IT's word that they are following and implementing security policies that are effective. These policies suck for IT personnel though as they make their job much harder. Thus they love to take shortcuts. Attacks come in a few vectors but predominantly they like to hold data for ransom or in this case, may love the Intel they can get. Virus scanners work mainly on known viruses and new viruses can get past them.

So here is an actual true attack I was personally involved in. Large company with very good virus scanning has employee install, unwittingly, a remote access application. Some new virus but it took the employee's approval. Employees need internet access and draconian restrictions result in IT being chastised by said employees. So IT hates being hated and tries to accommodate for multitude of reasons that results in less secure networks. Once remote access is running, bad guys install apps to make virus scanners appear functional but do nothing. Then they install keystroke recorders while scanning the network and just getting a lay off the land. At some point an IT technician is officially at this computer because for 'some reason' it lost access to a shared resource. Oh it just needs elevation. Instead of pulling out their laptop and logging into their secure desk computer few stories up, they decide to use said employees desktop instead to access their computer and update the infected computers credentials. This alone is not dangerous because the infected computers does not have access to backups. But the Keylogger on it has now transmitted the IT personnel access credentials to the bad guys. Later that night when business closed, they use the infected computers to log into the IT technician's computer. From there they install additional keyloggers and review access and any other software they want. Then they they watch this guy as he does upper level maintenance across their network for weeks/months. Maybe they get into a few more computers until bingo, someone maintains a backup that gets keylogged. Ransomware attack encrypts all databases and the backup and demands for two million dollars shows up.

In this particular case, luckily an off-site backup is found but it is a month old. Lots of employees trying to recover by memory some of the lost data. Management angry because IT been telling them they are following best practices. IT angry because they truely could use more money. Regardless, more money won't fix a guy that is too lazy or too overworked or doesn't care or... to start up his secure laptop or real two floors up to insure he is not using elevated services on a compromised system.

2

ottawawebguy t1_jad6xmf wrote

There is a bit of delusion of knowing about technology which translates into "I'm an expert in technology" while really ignoring the person in the room going "we should pen test"

2