Well, that escalated quickly

We’re replacing our 20-plus-year-old NEC PABX with a VOIP phone system. While it’s a big learning curve for me it’s coming together and we are on track for go-live next week.

$offsider and I went to roll out the remaining IP phones this morning, the comms room for which also serves the room we will be using for training on the new phone system. So while $offsider was putting phones on desks I quickly powered up the spare Power-over-Ethernet (PoE) switch (that the training phones would connect to) and plugged it into the network.

Within a couple of minutes $offisider is in the room: “Did you take down his connection already?”

“No…” I said, as someone from the call centre came barreling in to tell me that the whole team had just lost all network connectivity.

“Well, it must have been this switch…” I said, as I was interrupted by a call from $boss. “What did you do?” he thundered. “The network is down EVERYWHERE.”

“Well, I just plugged in this switch… and I’ve unplugged it now,” I said.

“OK. Still isn’t up,” he replied.

Oh bugger.

I ran back to the server room and noticed that the two switches in there were behaving strangely, so I power-cycled them… to no avail. Realising that the problem was most likely where I’d just been, I legged it back to the other building and called $offsider, who had gone back to the office to man the phones.

I got him to set up a continuous ping to a DC, and pulled the fibre uplink on one of the switches. Immediately $offsider reports a response. Resetting the switch elicits the same response but the ping is dropped once the switch comes back up.

Hmmm, methinks this switch is hosed. Plenty of spare ports on the other switches in the rack, so I re-patch them all. Suddenly $offsider tells me the ping has gone down again; some more troubleshooting reveals that the offending port is actually an uplink to a third switch. So I start repatching cables from that switch into the one previously thought dodgy. One patch lead in particular seemed to be causing the problem, so I set about tracing it back to… the same room that I’d plugged in the training switch 45 minutes before.

Sitting innocently right next to the now-standalone Cisco PoE training switch was a D-link ADSL router that had been connected to a network port under the meeting table and had been serving as a hub for a couple of years. Unplugging this from the network restored order to a chaotic morning.

The D-link has now been filed under B.

TL; DR: A “rogue” hub causes pandemonium after a new Cisco switch is connected.

This sounds like a job for BPDU Guard!

(Unless I’m reading wrong. But friends don’t let friends create broadcast storms.)

1 Like

My Computer Scouts* think I’m joking when I say that the only purpose of a D-link is to insert it into the enemy’s network and watch it all go up in flames. IT. IS. NOT. A. JOKE!

*I seem to have attracted some youths to the world of geek/nerd. They have at least figured out that I know a lot and that they can get hardware and knowledge from me. My mom think I should call them computer scouts. The server-happy one got a HP DL180 G6 from me that was only taking up space.

1 Like

Oh no, it’s not a joke. I’ve done the stupid once or twice in my career and plugged one of those in without turning off DHCP. :blush:

$engineering_manager (who is an electronic engineer by profession) thinks DHCP or DNS was enabled, but there are two main flaws with this theory:

  • we would have been seeing different IP configs
  • it would have taken much longer than 3 minutes to disable the entire network (over 3 buildings)

I think the act of plugging in the training PoE switch caused the D-link to implode and start spewing out packets.

Could well be a case for BPDU Guard… but being a Cisco n00b I don’t know whether it would have helped in this scenario. Certainly there was no mention of it by the two pro-spec Cisco ninjas that looked over the switch(es) previously.

Happened to $site - technician was bored, and he was looking at the switch cabinet.

Found a loose connection, and proceeded to plug it into the switch. No network.

Recently I was wiring up an UPS in a small cabinet. Plugged the UPS into the power (but silly me didn’t check WHERE the power came from)… noticed a strip extension (multiplug extension) wasn’t connected to the UPS, so I plugged it into the UPS - which promptly shut itself down as it was feeding itself. Luckily the servers wasn’t booted yet, only switches and ancilliary equipment.

1 Like

I vote for technominions. :smile: You could even get them yellow shirts or something.

2 Likes

Next generation scum. :wink: Ops wore red, just like security. :stuck_out_tongue: Yellow’s for command.
(Not serious.)

Filed under “B”? What does “B” stand for?

Basura? (Spanish for trash)
Bitch?
Bastard?
Bonehead?
Blow It Up?

Inquiring minds want to know. :smile:

Actually I was thinking yellow like the minions in Despicable Me. :smile:

Tempting.

Red shits… those are the expendables in Star Trek right? More tempting.

1 Like

LOL - don’t think you meant it that way, but it is accurate…:slight_smile:

2 Likes

Although all of the above are solid candidates, your first guess was the closest - “B” in this case stands for “bin”. :wink:

1 Like

And here i was thinking that “Computer Scouts” was a reference to PGSM…