Phaze357
38518
568
6
This was from two different occasions where the UPS (Uninterruptible Power Supply) for a hospital server room began emitting hydrogen sulfide. I honestly can't tell who is being sarcastic with the UPS/Fedex comments, and who genuinely believes they were delivered in this condition. Anyway, I was blamed for farting and stinking up the place. For once, it wasn't me. The rep servicing the batteries had to use a pry bar to remove them. I kept my distance during that procedure.
I was not the one responsible for maintaining and monitoring this battery bank. I was the one that discovered it due to the stink though, as it was in a room attached to my office.
This might seem pretty bad, but bear in mind this was at a for profit hospital that wouldn't spend money on anything that was truly needed; the server room had water sprinklers for fire control instead of a xenon gas system. It was like that for four years that I worked there, only replacing the system the last month I was there. Note that this item was at the top of the list for the "we need this to be fixed ASAP" and was constantly ignored by the C suite, despite only costing about $10k. If that sounds like a lot, they spent $30k replacing furniture in the doctor's lounge.
Eaton 93e, 20-60 kVA capacity; note that both sets of photos were different occasions. Quote from one of my comments on this matter:
"Hmm, looking at the dates on the straps, the units in the UPS were installed July 14, 2016. My date on the photo is January 26, 2018. So 561 days. Yikes.
The closeup was on May 9, 2017. So those must have come out of the other side of the UPS; there is another cabinet on the left. If we assume the same install date for this set, 299 days."
tombeithemist
For balance, you really should show the new furniture in the doctor's lounge tho'. Two sides to every story and all that...
FuckmotheringxVampire
"Gently used"
combatkillaz
Uinterrupted Power Supply
76000BatteryLlamas
Seems your power supply was not as uninterruptable as you thought, Mr. Bond
Rodltwo
KumquatKarl
They are more like ... guidelines.
KickStartMyShart
Those batteries look like they have eaton a lot
Phaze357
DarwinsWeasel
I was wondering what UPS did to deliver batteries in such a poor condion.
Phaze357
These were pregnant. The delivery comes next week. Oh, and this was at a hospital; the labor and delivery department was one floor directly below this.
twodinners2
Soon to be Interrupted power supply batteries
ME2BNS12
Looks like heat damage
Bubblewrapiscool
They will also have been very hot when powered as they plates inside have buckled and shorting out against each other.
Bubblewrapiscool
Had to double check I wasn't talking shit https://electricalbaba.com/sulphation-and-buckling-in-lead-acid-battery/
benderfreak
"ups" is the german equivalent to say "whoops/oops"
GravitationalFluctuations
I once ordered an original box for a rare 1980's cassette deck on ebay. The seller shipped it as it was, slapped a label on top of it with no extra box to protect it (Yeah, shipping a box within a box sounds silly but it needed it). When UPS delivered it, the thing was trashed. Visible boot marks, heavy scuffs all around from them kicking it and sliding on the floor, and blown out sides. It wasn't even worth keeping. I had to throw it away.
BendadickCumabatch
Absorbed glass matt sealed lead acid batteries cannot take continuous high-current charging, they have a difference charge profile and "full" voltage than your regular deep cycle marine battery, and also can't vent gas products from being overcharged.
Also China, Inc.
unusuallyAverageJoe
Pb stands for portable battery /s
mayhemonroads
š Brilliant.
Hexidimentional
heehoo, but seriously Pb is the shorthand for lead, they're lead acid batteries, hence why they're giving off hydrogen sulfide
Ariltongadottir
Pergnant battery
Hammertulski
How batty get pegant
MisterPrimeMinister
Oops batteries
inthepines
Are these made by Apple?
seedye
Exploding UPS batteries was a plot point in Mr Robot.
7YearsALurker
This is why we have a inspection and replacement schedule.
CrapsMcgee
*squish* so cute
dougswammie
Too cheap for regular maintenance? UPS is too important to skimp. Sounds like the Finance Dept is in charge of IT like back in the day.
Phaze357
Tenet Health, they don't care about the well being of the IT services that are the foundation of all the modern technologies that allow them to operate on such a level. This will put it in perspective: /gallery/7lLbDEV/comment/2410720877
ProfFurryPaws
Of course they are, it's an expensive, not a revenue generating department. Gotta cut costs and pinch all those pennies, or IT might just spend money all willy-nilly on things like "maintenance" and "reducing downtime".
dougswammie
Running around all nimbly-bimbly buying the latest tech for the C Suite.
Phaze357
No shit. This was at a hospital, and they spent $30,000 on furniture for the doctor's lounge instead of spending about $10,000 for a xenon (I think it was xenon) extinguisher system for the server room. The server room that had regular water sprinkler heads. It was at the top of the list of the stoplight report for the entire 4 years I was there, they only replaced it a month before I left.
Same hospital that re-hired a doctor that was abusing amphetamines; his original offense was harassing-
Phaze357
an employee. They re-fired him, but only once his behavior got so bad that he threatened to shoot up the C suite. Typically execs; the rest of the hospital can burn and they won't do anything until it affects them. The cowards had us set restrictive badge access to the C suite doors for a while after that, the rest of us be damned if he showed up ready to blast.
CaliforniaCornucopiaVegetableJubilee
*backs away slowly*
craque
Oh no you don't. Not until you finished all of your aerosolized lead.
ElbowDeepInAJedi
*turns and runs*
peregrinus19
OlessanThings
I have a personal UPS for my desk (PC, monitors, etc). It's just a little 4 years old, seems fine, maybe I should inspect it...
Phaze357
Might be wise lol
marthafarquar
I initially thought they had been delivered by UPS, because I have had a couple of packages arrive in that condition.
Mithi
both.gif
Phaze357
I had a phone delivered from AT&T for one of our sales team members the other day; came in via UPS. Looked like it had been used in that first scene in Ace Ventura.
TheSilverSeraph
The gentle bulging just tells you they are all full of electricity.
Peak readiness, nothing wrong here
Exordos
"gentle bulging"
itsallaboutthecones
It's to indicate they are ripe. The design is very human.
Phaze357
MapleSyrupMafia
Not ideal for pole vaulting tho.
finallythisisnottaken10
Lol, potentially very uninterruptable
GubmentCheez
Pretty soon they will be giving birth to new baby batteries!
Buffoonery
Yep. They're about to eject the magic smoke. Once that happens...it's time for a new job.
Klaaism
SacrificialClam
Time for the harvest
HughGRecti0n
I know thatās right
DGCA01
Hah
Goldmarble
I feel like it would be a good time to start running...
OneOfTheEnemiesWithin
TheMayo16
These are lead batteries, they dont fail spectacularly like lithium batteries do
nemocares
What would make a lead-acid battery bulge like that without it being time to run? Only thing I can think of is hydrogen gas build-up and, well, significant explosions are very much on the menu there.
TheMayo16
Yea, they swell from a buildup of gasses for various reasons and yes those gasses can be flamable, but the case on lead batteries usually cracks and bleeds out the gas, so it never has the chance to act like a bomb (in a ventilated area).
TheMayo16
It probably doesnt take much gas to warp the plastic when they are hot
Filanwizard
huh I did not think lead acid batteries puffed up, Thought that was a lithium thing. I don't think I have ever seen a car battery do that,
Ghlargh
They shouldn't, i would guess these have been severely overheated for the plastic to soften enough to bulge like this.
Daraanel
Lead-acid batteries bulge over time from heat generated by overcharging. There are some types more prone to this than others.
Hexidimentional
overcharging plus bad vents
lurkercat
A tech looked at those and decided to rack them anyway. Thanks Eaton!
Phaze357
No... that was after being removed. No one put swollen batteries into a UPS lol
You can see them covered in pry bar marks from being removed. That wasn't from using a battering ram to put them in.
Phaze357
At one point the battery tech grabbed a crowbar to pry them out. They were releasing a powerful funk. I'd been blamed for crop dusting the office, but this time it wasn't me. Don't forget to inspect your batteries folks, or they get pregnant.
Groovechild
...this time...
Hexidimentional
spicy pillows
bound4doom
Does your USP not alert you when a battery goes bad? Might want to look into that. Mine have had that tech for 20 years where they can at least send you an email. If not set up, might want to do that. Then pull it out immediately in my experience when that alarm goes off one will be very hot, you might have less capacity, but better than alternative.
Hypothesist
I've seen a few of those go up in a large fireball, Im no ups tech but i wouldnt be in the room while they were doing that lol.
PipWhipple
Knew someone with a camper who spent a week trying to clean it and figure out the terrible smell. Yep, bad battery!
makeSX
What kind of batteries Ar those
anubis132
Sealed lead acid. They shouldn't bulge like that, even when aged. Their charger is overcharging them, probably because of bad cells.
makeSX
Yeah I really need to concentrate more, its says Pb in one of the pics. Never seen a battery bulge like that before
Vengeraxe
Last IT job I had, as usual, once i pointed out ALL the UPS batteries in the Noc were far far overdue for replacement (also cracked, bulging etc) I had to fight the Head of the department for the funds to replace them. I kept telling them they likely were not going to work in a emergency, even offered to cut the power to the building to test. Small buisness's dont like to spend money, and once a battery backup is installed, they expect it to work forever.....
Phaze357
This was Tenet Health, which is by no means a small company. Infected with parasitic shareholders that suck every penny out to the detriment of the company, employees, contractors that were unfortunate enough to service the department the company sold or laid off, and patients.
Vengeraxe
Yeah the Job after was for a much much bigger company, and I found they were even more stingy with preventative maint "We've never had a issue before..." So when the power went out, and the backup power did not kick in, who's to blame? (not me or my manager, we documented the shit out of every proposed method of ensuring our power stayed on, as well as the denials......) Keep copies of all yer Emails IT lads, can save yer bucket later on.
jinbee
I've seen the safety video where they explode a bigger rack than this, be careful as all fuck around that stuff even when it's not ripe & juicy.
xaviar3000
As a battery tech, these should be replaced every 2-3 years. 5 at the āweāre broke and canāt afford regular serviceā level. 8 years is too many years.
Phaze357
I don't know where my original comment went and apparently I can't pin a comment, but this was back in 2017 and 2018 (both photo sets were from two different occasions, same result.)
bohjeenie
If those are lead acid batteries and sulfurous smelling, itās H2S (hydrogen sulphide). Flammable and toxic. Itās generating by overheating the acid, usually from dead cells.
SDS: https://www.airgas.com/msds/001029.pdf
bohjeenie
From OSHA:-
permissible exposure limit (PEL) for hydrogen sulfide (H2S) at 20 parts per million (ppm) as a ceiling limit and 50 ppm as a peak limit. The 20 ppm limit cannot be exceeded at any time during an 8-hour shift, but the 50 ppm limit can be exceeded for up to 10 minutes within that shift if there is no other measurable exposure.
Fatal if inhaled.
bohjeenie
ā0.47 ppb (Parts pr Billion) is the odour threshold, the point at which the human nose can detect and recognise H2S.ā
If the whole room was smelling (vs. Just lightly detectable by putting your nose next to the battery), likely youāre getting a very high concentration, this event should be logged with your EHS and those impact checked out by a doctor.
Different batteries should be used in future or ventilation & detection added to that enclosed space.
bohjeenie
Context: Iāve had 3 coworkers hospitalized from H2S exposure (different industry however, chemical lab during a sulphuric acid spill that was mismanaged in the response]
Kuiper
That funk is extremely hazardous. Let it vent and gtfo
neptix
It will also burn out your ability to smell it, GTFO, and don't go back in until the gas meter says it's safe. After a decent dose, you can't tell anymore
BrdCdn
If it's H2S and you can smell it you are safe. Once you stop smelling it is when it's dangerous.
Phaze357
Hydrogen sulfide, which is one of the components of flatulence. Which does indeed explain why it was mistaken for a fart.
minipancho94
Also IIRC my Chem correctly, will basically melt your brain and leave you with immediate brain damage if it doesn't outright kill you within seconds if you're exposed to more than your body can buffer. Once your body uses up all of the chemical it makes to protect you against the HSĀ² itself produces, you're pretty much dead. So many horror stories of workers running the moment they smell it and still suffering horrible brain damage if they live.
itsallaboutthecones
Also very flammable and explosive...
paulbloface
The LEL (Lower Explosive Limit) for H2S is above the PPM (parts per million) that will rapidly knock you unconscious then start killing your brain. If you're exposed to enough to be a fire hazard, you're already dead.
MapleSyrupMafia
*Laughs in former mechanic! Science is nice and all, but there's been multiple incidents with batteries in garages and even people wrenching on their cars outside, not exactly an enclosed room. It just needs to be concentrated enough in one place when a sparky sparky happens. That smell in a shop is never a good sign, so yeah... I wouldn't just brush it off as it will kill you before it's remotely close to hindenburgering you.
cyno01
Like farts!
Dimension09
Phaze357
Apparently my added descriptions only show up if you open the individual pictures, so I'm going to do this the janky way and paste the info below:
This was from two different occasions where the UPS (Uninterruptible Power Supply) for a hospital server room began emitting hydrogen sulfide. I honestly can't tell who is being sarcastic with the UPS/Fedex comments, and who genuinely believes they were delivered in this condition. Anyway, I was blamed for farting and stinking up the place. 1/4
Phaze357
For once, it wasn't me. The rep servicing the batteries had to use a pry bar to remove them. I kept my distance during that procedure.
I was not the one responsible for maintaining and monitoring this battery bank. I was the one that discovered it due to the stink though, as it was in a room attached to my office.
This might seem pretty bad, but bear in mind this was at a for profit hospital that wouldn't spend money on anything that was truly needed; the server room had water 2/4
Phaze357
sprinklers for fire control instead of a xenon gas system. It was like that for four years that I worked there, only replacing the system the last month I was there. Note that this item was at the top of the list for the "we need this to be fixed ASAP" and was constantly ignored by the C suite, despite only costing about $10k. If that sounds like a lot, they spent $30k replacing furniture in the doctor's lounge.
Eaton 93e, 20-60 kVA capacity; note that both sets of photos were different 3/4
Phaze357
occasions. Quote from one of my comments on this matter:
"Hmm, looking at the dates on the straps, the units in the UPS were installed July 14, 2016. My date on the photo is January 26, 2018. So 561 days. Yikes.
The closeup was on May 9, 2017. So those must have come out of the other side of the UPS; there is another cabinet on the left. If we assume the same install date for this set, 299 days." 4/4
nik282000
Uhh, unless it is special that 3 phase breaker is not meant for DC.
Phaze357
A UPS has to convert power back to AC to supply to the devices receiving power in AC. So you have AC in converted to DC > batteries < DC out converted to AC to the server room.
nik282000
Those are some pretty heavy cables for the AC side. How many VA was this thing?
Phaze357
It was an Eaton 93E, which supports 20-60kVA. Looking at the full image, only the right side housed the batteries.
nik282000
Bruh, that's one hell of a ups.
CookieDave
How long were they sitting for? Gotta check those things annually at the least.
Phaze357
Hmm, looking at the dates on the straps, the units in the UPS were installed July 14, 2016. My date on the photo is January 26, 2018. So 561 days. Yikes.
The closeup was on May 9, 2017. So those must have come out of the other side of the UPS; there is another cabinet on the left. If we assume the same install date for this set, 299 days.
This was at a hospital I worked IT at, I will add that I wasn't the one responsible for this maintenance, but they did fart up my office.
Bystandr
Lead acid car batteries are commonly good for 3-5 years, and lithium batteries can be good for 5,7 and 8 years depending on application at
Phaze357
One should note that these were the UPS batteries for a server room at a hospital. They were critical and should have been inspected and tested regularly. They weren't lol
Bystandr
various sizes. The key is to note when the manufacturer for the UPS batteries rates that the lifetime has expired, in addition to checking
Bystandr
the stack when the manufacturer recommends (Usually less than yearly).
Phaze357
Without me pointing out the stink and figuring out the UPS was shitting itself who knows how long it would have been before the 'network admin' got around to checking them.
My lack of confidence in this individual is exemplified by the time I asked him for a flash drive, so he reaches in a cabinet and pulls out a container of drives. Probably at least a dozen, all mixed. I'm thinking it's just drives he's used over the years. Nope. These are found drives. One of which had porn, but the scary-
Phaze357
part is that the was an admin for the site and overrode his machine to not have an automatic lockout so it would stay unlocked indefinitely, and he was plugging in USB drives that could easily contain malware. I stood there in slack jawed amazement that someone in that position would do something so foolish. It tracked for him though.
CookieDave
This the type of fool that signs into all domain systems with domain admin. Oh, you saying you need admin? Sure, letās add you to the domain admin group.