Klythe
160724
1660
40
Stolen.
Nov 24, 2021 5:26 PM
Klythe
160724
1660
40
Stolen.
GoodJuJuuu
I spent hours getting a SAN connected to vCenter. Turns out my dude was zoning the wrong freaking switch the entire time. Took 3 days :/
Hellcowboi
The question is irrelevant, the answer is DNS.
Maevellous
Don't get me started on it. I respect 90% of you know what your doing but the ones I deal with are the 10% who think they do.
ComicSansHumor
If I'm blaming CI, it's my fault. If I'm blaming myself, it's my fault and someone else noticed. If I'm blaming networking, I have no idea.
bippityboppitybuttsex
Some of us have CCIEs....
sgaduuw
I always provide a pcap with proof, and threaten to fix it for them
NottSure0
Story of my life. I get called in for unnecessary shit all the time. No one takes the time to troubleshoot.
Thisisnotreallymyname
(An hour later) The networking team: "so, funny thing. One of our guys did an update to an ACL, and..."
vissago
i feel this in my bones.
WhyEvenHaveAName
The problem I have with networks is no matter how many times they are proven wrong they never give me the benefit of the doubt and check.
WhyEvenHaveAName
I'm convinced they all have imposter syndrome so bad they get scared and go into denial mode when real issues arise.
jennym123
Reminds me of a ticket that said something like "User declined instructions to resolve issue and requested alternatives."
RowanUnderwood
I have replaced So Many Motherboards because of this nonsense. Sometimes more than one even!
MichaelCBennett
It's the DNS...
skwint
It's hardwares fault. I've proved it. But we're making it into a software bug so that hardware can meet their deadline. Then we're going to
skwint
have a ridiculously expensive meeting where I have to prove it's a hardware fault again to get it off my plate because PROCESS.
skwint
wow. I'm way more bitter about that than I thought I was. God we pissed a lot of money up the wall in the name of paperwork in that company.
byebuddyhopeyoufindyourdad
When other IT groups do not communicate w/ Client support that an “update” is being pushed the night before
Masonrig
As a former network engineer, the number of times I've seen coworkers who would ping router to router and do NOTHING else was staggering.
MireLurkKing
This, so much this.
Stanistani
Former IT person. Don't *claim* it's a network issue, *prove* it. Then go complain.
WhyEvenHaveAName
First of all, you only need evidence networks needs to investigate. Second, I've found proof isn't even good enough. /1
WhyEvenHaveAName
If I have to nail them to a wall with evidence every time they get to ashamed to admit fault. /2
jamsplodge
If it can be a network issue, it can be a network issue AGAIN
quzar
Device is connected now therefore it never has been and never will be a network issue. - My network team
jamsplodge
I used to work in customer service at a computer store. "Yeah we need to replace part X", response "Well it was working yesterday"
ChuckKarma
It's DNS
Decstorose
It's not DNS. There's no way it's DNS. It was DNS. (Outage Haiku)
Yibber74
It's always DNS
cross3dg
As a net eng myself, the number of times I've been told the network was down, only to find the user wasn't even physically connected...(1/2)
cross3dg
...is a significant non-zero number. (2/2)
UnluckyLunkhead
Time-expired RAS passwords. If you've no idea, you too could join our network section.
thesavagery
So if the network connection speed is fine but the network drive only has 2.2gb/100tb free, who's fault is it?
boondoggle2025
Networking!
muchLongerUsername
Server team, specifically the guy who set the alerting thresholds. Followed by the users who insist they need 15 years of unused data.
easylikerain
So many users want years, GB and GB of old email. They aren't lawyers, so why? And never young users either.
b0Bm3h
For the user who thinks they need their families photos on a work computer. Also the help desk agents who "fix" an .ost by renaming ...
Khaosus
Surprise! Its actually SecOps borking a firewall rule for some random vlan.
kgnzpfwws
Good network admins understand layer 3,4. But what makes an admin really worth their salary is when they really really understand layer 2.
AndNowForSomethingCompletelyDifferentdk
I would say that you have to understand L2, in order to fully understand L3 og L4. After that comes overlays and underlays. 1/2
HeraldofOmega
Mexican Mouse: "Eeba eeba overlay underlay"
AndNowForSomethingCompletelyDifferentdk
2/2 Then comes routed L2, mostly using EVPN. Good old routed L3 is still the most stable, most proven solution and the easiest to troublesh.
browsererror
I once had to write a utility and then spend a week of my time installing it on user machines to prove it was the network.
crazyspelling
... so you wrote a virus?
browsererror
No, I added a logger to our internal software so it gave me realtime data for network connectivity to the DB. I added no significant load.
crazyspelling
browsererror
We discovered that the load wasn't being balanced across the WAPs. Everyone was connecting to one, but none of the others.
crazyspelling
*Cardi B has entered the chat*
CyborgScribe
That poor pussy.
AndNowForSomethingCompletelyDifferentdk
There are test-utils you can use for free. Like iperf for performance testing or nmap for test for responding ports.
browsererror
This was a while ago, and the IT Director denied the use of unapproved 3rd party apps. So I write the util to be part of our internal suite.
browsererror
He was also the guy who configured the access points wrong.
AndNowForSomethingCompletelyDifferentdk
Sounds like a swell guy..
lDoNotThinkltMeansWhatYouThinkltMeans
The majority of the problem is folks not knowing their career very deeply. I don’t assume it’s not a group, I work to understand the 1/
CatWithHands
I dont get to "know my career very deeply." I have to be part DB admin, part HW/networking, part web admin, part release engineer, part QA..
lDoNotThinkltMeansWhatYouThinkltMeans
The problem and then I collect data to reinforce the “most likely causes” of any incident. Incompetence runs rampant in IT… maybe even 2/
lDoNotThinkltMeansWhatYouThinkltMeans
More than other industries. Solving IT “incidents” is different than solutioning for development/progress, and troubleshooting is a 3/
lDoNotThinkltMeansWhatYouThinkltMeans
Developed trait/skill that isn’t taught, but put together by either gift and/or experience. Networking is a mess with the inundation 4/
lDoNotThinkltMeansWhatYouThinkltMeans
Of appliances and applications across other IT units. Security is the worst mess and usually the most incompetent. I’ve solved incidents 5/
lDoNotThinkltMeansWhatYouThinkltMeans
For fortune 50 orgs, and orgs with over 150k employees. I’ve led engineering teams. The problem has been in places where they “proved” it 6/
clide005
So here in lies the issue. Networking is like Social Security. They get so many false claims. They just disbelieve as step one. 1/2
ahl1
I am not convinced that social security actually gets that many false claims.
clide005
2/2 so always come correct with logs and diagrams showing what's happening with the traffic. Ps it's always layer 3 :3
blackozone
JeRMP
Layer 8!!!
ShiftingPattern
I swear at least half the time it's layer 1...
velicitia
The other 90% of the things it's layer8
cross3dg
Either Layer 1 or Layer 8.
CumsInPies
Never has it ever been layer 2
finabar
Clearly you've never seen some block ARP because "I heard hackers can use it to infect the network"
ShiftingPattern
What would that even be? Pure hardware failure? Botched firmware update on a router?
WhiteTexan
Layer two is always framed
caedescorvidus
Ayyyyyy lmao
BishlamekGurpgork
Thank you for that insight, CumsInPies.
CumsInPies
Yes. Computers. Right click. Toggle. New Folder?
Kerro1981
It’s always the network teams fault. ALWAYS!!
boblives
I blame the firewall /s
ChuckKarma
20 years ago, it was almost never networking, and it was easy as pie for every programmer and tech to verify it wasn't networking. Today 1/2
ChuckKarma
2/2 noone can even ping their own ass.
crazyspelling
To be fair, a lot of us gained weight during the pandemic...
nclu
Except for that time we found a bug in Kubernetes interacting with a bug in Alpine Linux that caused 5 second timeout delays. Fuckin' nuts
baltus
This. I work for a very well known tech company and the 4 last major outages we have had were all network fault even though they denied it
aIohasnackbar
At a small startup, I was the network 'team'. It was never my fault. We were acquired, I became DevOps, now its 100% the network teams fault
FajitaJane
Note to self. Don't go into network IT
SheepySleepySmuggler
It depends - if you thrive on the bitterness of eventually proving that it was not a network error, then it could be the job for you...
Kerro1981
Ah no it’s great craic. I have worked across the infrastructure and operations board and networks is as good as anywhere just be humble.
wuttwuttinthebuttbutt
Hi, part of the other teams in IT. It's fucking network's fault. If they say it's not, then you can go ahead and consider that proof it is.
MithridateEupator
Guarantee the issue is just some user trying to connect over hotel wifi
AgnesMcGillicuddy
Hi. Cybersecurity here. My scanners definately didn't cause the outage so i won't be answering my phone tonight
loupgarou21
As a network admin, way too much of my time is spent proving its not the network.
machinelogic
IT guy here. It's true, it's always networking's fault.
bl0rpen
Here's my fucking pcap log of why it's network's goddamn fault.
Szj2
Cloud engineer here. Yes it's networking or security's fault. It's either routing or firewall. Those are my biggest daily obstacles.
JarmFace
"Here's the packet capture proving that it is network. I did your job for you, now fix the network." I've had to do that far too many times.
quzar
At my company it's never IT/Network's fault. It just always just resolves itself ~15min after they receive the issue.
WillowFox
Hey, I'm not allowing any ports if App and QA didn't provide the documentation. Fuck your app, it's not passing by my firewalls!
ProcrastinatingWork
As QA I support this attitude. Make us do it right the first time.
WhatisaYute
This is why my IT thinks I’m too much of a go-getter. I don’t care it’s not your fault, you’re impacted so help me fucking figure it out.
AreYouAwareThatYouAreACat
I work IT and am the main link between firm and ISP. 2 hours on the phone with them, saying that our router was faulty 1/2
AreYouAwareThatYouAreACat
After countless "off & on" troubleshooting with no progress, they sent a team the next morning. It was a faulty 2/3
AreYouAwareThatYouAreACat
Cable & switch from their end. The 2 dudes on the field resolved the issue within 20 min from their arrival. 3/3
skylardarkfox
I feel like sometimes I have an advantage in wearing all the hats. Can't just point the finger at someone else when I'm the only one.
skylardarkfox
If there's a problem, I just trace the wires (literally or figuratively) until I find it, and then I fix it.
HereDueToYourOnePeskyDownVote
Wow this place comes alive when IT hits the front page. FWIW I've seen waay too many teams blame networks and Splunk proving it disk (SAN)
HereDueToYourOnePeskyDownVote
Or server related. When you look after all the infra in a bank and you see how overworked the network teams are it's depressing how easily
HereDueToYourOnePeskyDownVote
People just blame the network. Oh and AppD was great from showing wanky code up too.
JessieArr
My packets went from the IP the network assigned me to the default gateway the network assigned me. After that it's networking's fault!
Anju42
If it's today it's AWS's fault. I love when it turns out to be an external service going down.
Simusar
But I can ping your pc from 6 states away. Call your local IT.
PTMdragon
Popcorn.gif
beeftornado
Spoken like a real other team. If you can’t figure it out, must be all those fancy internets and clouds.
ChromeBadger
I work IT security, and the amount of times I've had Network teams tell me "Not their fault" without even looking at the issue is staggering
blessedarethecheesemakers
Helpdesk here. this is correct. Networking proved this themselves Monday. Upgrading to Big Sur isn’t going to fix an AP problem you walnuts!
Dvicemuse
100% this
ShiftingPattern
I've had the pleasure of watching someone claim that two Docker containers on the same host failing to communicate was networking's fault...
vegivamp
Well, it's certainly a networking issue, it's just that Intrahost networks aren't really something networking team can help you with.
WatcherNotCreeper
Used to have a guy that would "clicky clack" the keyboard for a min when you reported an issue then "try again".
WatcherNotCreeper
Dude was always testing shit in prod but I was the only one smart enough to catch him.
Lalatino
If I can ping it; not the network
Allthegoodsnamesweretaken
Amen!
jeffois
So, how long HAVE you had your CCNA?
Kerro1981
LeftRightThere
Uh, ping does not prove SSH, HTTP/S, or any other ports are not blocked by firewalls
AndNowForSomethingCompletelyDifferentdk
Plus ping doesn't care is there is asynchronous routing in the network, it will still work, where other application will fail.
lwbell851337
Fuckin a. As someone that has to battle network security admins, this hits home. "The software won't work cause you still have SSH 1/2
lwbell851337
2/2 packet decryption enabled on the URLs it utilizes you fuckwit. Now exempt that shit!"
CorrectMostOfTheTime
Firewalls should be a security issue not a networking issue
ShiftingPattern
Unless your network access control is allowing ICMP traffic but denying TCP...
NikitaTokmakov
That would be stupid
ShiftingPattern
Not really. With least priv policies you aren't opening up TCP on any ports that don't need it, but ICMP traffic is almost always allowed.
NikitaTokmakov
Yeah I guess , but at that point you would allow only specific Sources IP like monitoring server or SLA , and block everything else.
AndNowForSomethingCompletelyDifferentdk
True, even though most don't recommend it, its opens because server admins are like "I can't ping my gateway!". You are not suppose to!!
ComcastLover
Was in a situation where it was networking's fault just last week. Turns out their monitoring software had froze and they didn't see alarms.
[deleted]
[deleted]
ComcastLover
Ask networking but I'm pretty sure they'll say no
cleverusernamealreadytaken
I worked at a university where the IT groups did not report to the same VP. I was a sysAdmin for the student registration system and ->
cleverusernamealreadytaken
the network team was “performing upgrades” that created a 45 second login delay for everyone across campus. My boss and I spent months ->
cleverusernamealreadytaken
showing that this was not an issue with the application but rather with the network. The director for the network group played the ->
cleverusernamealreadytaken
that we were creating these delays because we did not want to consolidate our group under a new CIO. This went on for months until ->
cleverusernamealreadytaken
finally showing the new CIO logs showing network latency, confronted the network director during an open meeting and during ->
SheepySleepySmuggler
As part of a network team, 99% of the major calls that come in get classified as a network fault. Have an application that....
viridianasovari
I spent 3 weeks fixing a printer issue just to prove it wasn't the networks fault. HD lied about what they tested, the symptoms didn't 1/
SheepySleepySmuggler
...can't access a database on the same server? "It says connection error, it must be a network fault". Or on a seperate VM...
SheepySleepySmuggler
...but same physical host? Blame networks... We currently have a finance application that makes calls out to 3rd...
SheepySleepySmuggler
...parties via TLS1.2 only the application sort of fudges TLS1.2 support and the finance and application team blame networks....
SheepySleepySmuggler
...inspite of significant evidence showing the fault is with the application including the software provider saying they need to...