Outage since last night was my fault, derp

#1

If you encountered a 502 error getting to the forums over the past 12-ish hours, that was me.

I had the hosting company doing a BIOS upgrade on the web server last night, which required a reboot, and it appears that someone hit the CoG forum URL at exactly the right instance at the tail end of the reboot, just before the Discourse docker container was fully back online. The 502 bad gateway response was cached, and Varnish has been serving that since last night. It would have eventually fixed itself as the cache expired, but I’ve killed & restarted Varnish and that appears to have fixed the problem.

Thanks to @nabiki for the heads-up!

tl;dr - Caching. It’s always caching.

#2

Oops, that probably was me

Muhuhahahaha

/runs away

#3

Who do I contact for a refund?

#4

Wait you pay for BigDino??

#5

I’m on the premium plan.

#6

Premium plan eh?

You have Lee’s personal mobile number? :rofl:

#7

Well, shicks, that may have been me… I was uploading pics and it was giving me weird error messages but I figured I was just doing it wrong so continued on my way. Then the forums were gone. Sorry!

#8

It’s no problem! Didn’t mean to make it sound like someone was to blame or anything like that—it’s just a consequence of upgrading in the middle of production rather than doing things the “right” way.

The “right” way would be to pick an off-hours outage window, alert everybody about the outage at intervals starting 48 hrs prior to the picked time, switch the site over to a maintenance/outage “we’ll be right back” page, do the upgrade, and then cut back live.

1-8xraf6eyaXh-myNXOXkqLA

#9

I am Spartacus

#10

:joy: lol hehe

But I’m also guilty of doing such things like rebooting a DC during the day…