BOYS
SITE WAS DOWN FROM SUNDAY MORNING TO WEDNESDAY ARVO. THOUGHT THE WORLD HAD ENDED..
WAS SCROLLING THE BACKUP TAB I HAVE OPEN BUT THE SEARCH FUNCTION ISNT THE SAME WHEN ITS NOT LIVE
LEFTERIS WHAT HAPPENED MATE
Site was down from Sunday 11/08 about 6am to Wednesday 14/08 about 4pm. Sorry to everyone trying to use it..
Short version - power board at the host room got unplugged during a clean up. Then no one noticed. Server went down. UPS battery held for 6 hours then died.
I found out at lunchtime Sunday when Boonie texted me asking if i was OK. Couldnt get access until Monday morning.
Power restored Monday morning. But the UPS had drained completely so the server wouldnt boot - the BIOS battery had also died sometime during the outage and the boot config was lost. Tuesday i drove down with a new BIOS battery, reconfigured, ran the database recovery (4 hours), tested everything. Site back up Wednesday afternoon.
No data loss. The recovery script worked. Some thanks-counts may have been lost from the few posts made between 5am and 6am Sunday morning but no posts themselves missing.
Cause of outage: human error at the host site. Mitigation: the power strip is now labelled "DO NOT UNPLUG - INTERNET SERVER" in Greek and English.
Lefteris is being polite. wrong plug got pulled. The modem is BLUE tape. The server is GREEN tape..
Labels are going up Friday. Big ones.
Im sorry boys. Lefteris was polite about it.
I thought it was my phone at first.
Was in the garage waxing the 380 on Sunday when the site went down.. pulled out the phone to check a thread, nothing. Refreshed for an hour. Then assumed my data was cooked. Mrs B asked me why i was sighing every 5 minutes.
By Tuesday I was getting twitchy. By Wednesday afternoon I was on the phone to Lefteris asking if it was permanent.
Lefteris im sorry mate. Didnt mean to add to the stress. Cons old man bless him.
Thanks for the recovery work.. three days for a residential-grade server setup with no on-call cover is exceptional response time.
If the forum eventually requires a more robust hosting arrangement i would be willing to contribute to a co-hosting fund. I will not push the discussion. But the offer stands if Lefteris ever needs it.
Hugo - cheers mate but the local box stays.. steve set it up in 2008 and the server hasnt moved since. Wont move while im running it..
If someone touches the wrong plug again ill add a SECOND UPS and im looking at a low-cost off-site backup that runs nightly. Wont change the local setup.
Boys, the forum is back. Lets get on with it.
Three days outage with a single-person admin team and a local server setup. Cheers mate. The recovery work matters and most of the boys here have no idea what goes into it..
For my part if you ever need 4 hours of someone watching the server while you sleep before a big migration or whatever, im in Campbelltown, ill come down.
No hard feelings from any of us. Couple of days offline is nothing.
BROOOS.. lost 3 days of pipeline. Two clients went elsewhere thinking i was unavailable..
No complaints Lefteris, just commenting. forum is back, thats the main thing.
If you ever want a secondary phone number forwarded for outages you can use my line, no charge. Just say its for the forum.
LEFTERIS
3 DAYS WITHOUT THE BOYS WAS HARD MATE
THE GREEN TAPE LABEL IS THE NEW SOUL OF THIS PLACE
GANGSTA WORK ON THE RECOVERY
Groovy
[no content posted]
Wally - your second post above (post #11) was a known bug. Empty message body was bypassing the Send button validator on the new-reply form. Specifically when the user clicked Send too quickly after focus moved away from the textarea, the validator returned true on an empty string.
Issue identified, patched this morning. SMP §11.6 noted - no further empty posts will be accepted by the form. Cheers for catching it the boring way.
- Lefteris