WebMail Issues

Tue, 03 Aug 2010 23:59:27 GMT

Are now all resolved following discussion with the software provider. Please open a ticket if you are still experiencing these issues.

WebMail Issues

Mon, 02 Aug 2010 23:29:04 GMT

Some customers may be experiencing issues viewing folders other than their Inbox. The Software provider has identified an issue with folder caching. With it turned off things work properly, so we're proceeding that way until a fix can be provided.

ASSP Disabled!

Sat, 31 Jul 2010 15:59:00 GMT

ASSP has now been disabled. We are now using the grey-listing capabilities built into the mail server software.

You can also enable Bayesian SPAM filtering by logging into your mailbox via the WebMail and going to Options > Spam Rules. You can select how harsh the filtering is for your mailbox there.

Mail Services Upgrade - Complete

Sat, 31 Jul 2010 01:14:00 GMT

Updates to our Mail Services are complete for tonight.

We will be enabling the built in grey listing and stopping ASSP tomorrow night once further tweaks have been applied.

Mail Services Upgrade

Fri, 30 Jul 2010 22:05:00 GMT

Customers may experience problems connecting to POP3/SMTP while we conduct a Mail Services Upgrade. We're currently working on this task, and will post an update once complete.

Firewall Upgrade Complete

Thu, 01 Apr 2010 01:17:00 GMT

Tonight we successfully swapped in our pair of new PIX515e firewalls, configured for High Availability with separate feeds to separate switches on our upstream providers network. Hopefully, nobody noticed a thing :)

Slow Access

Thu, 24 Dec 2009 15:46:55 GMT

Customers may have noticed slow page load times since www2 was replaced, this has now been resolved.
Whilst removing the old www2 from the rack on Tuesday (and 2 more machines which were not required any more) we moved the private LAN switch up the rack and reconnected it. This was due to it being removed to give space for the new R200 servers we installed a few months ago, but now there is space we relocated it.
Unfortunately when we did this, one of the cables was plugged back into the switch, causing a storm on the switch. We also missed the cable for www3's private lan. This caused the firewall's CPU to overload, causing the slow access (and also for some customers to require database IP's on www3 changing temporarily.
We apologise for this service interruption, and will be replacing the firewall in the new year and at the same time re-labelling and replacing cables in the rack. This will require a few hours of downtime, which we will notify you of dates after the Xmas break.
The new firewalls are a redundant pair also, to avoid the problem we had a in October.

WWW2 Unavailability Monday 21/12/2009

Mon, 21 Dec 2009 23:41:35 GMT

WWW2 has suffered a RAID controller failure.

Update 19:33 21/12/2009 - DNS and FTP Services have been restored. Website data is now being restored from a backup taken in the early hours of Sunday Morning, sadly the disk controller failed before the incremental backup was taken on Monday Morning. Once this data is restored, we will restore access via IIS which we are working on setting up whilst the data restores. MySQL databases will also be restored before IIS so your websites should work once restored.

Update 20:28 21/12/2009 - DNS, FTP, and MySQL Services have been restored. Website data has been restored. IIS is now being restored and this message will dissapear in time as each web is rebuilt.

Update 22:54 21/12/2009 - All Services now Restored. Please contact support if you are still experiencing errors.

Unavailability of services 17/10/2009, 18/10/2009, 19/10/2009

Mon, 19 Oct 2009 19:52:40 GMT

We apologise to our Customers for the recent period of unscheduled downtime.

At 21:30 on the 17th of October 2009, we lost our primary firewall. Our engineers attempted to troubleshoot the issue remotely, before dispatching an engineer with a spare unit to site just after 10pm. After a heroic battle with the roadworks on the M4, engineers arrived on site to find a unresponsive (though seemingly alive from the lights blinking away) firewall. A reset was attempted, but failed, and the unit was then removed from the rack and replaced with another unit. Service was resumed at approx 02:00 on the 18th of October.
At approx 09:00 on the 18th of October, the backup firewall also failed. This took us a while to source a replacement unit, and partial service was resumed after a lengthly session in the datacentre at approx 14:00hrs. Full service was restored by 17:00hrs
At approx 07:00 on the 19th of October, we lost connectivity again, however this was due to a fibre break somewhere in Redbus Sovereign House in London, on our up stream's network. They were onto the issue immediately and service was restored by 09:00hrs.

From examination of the two firewall units, our original firewall appears to have suffered a PSU malfunction, which has damaged the main logic board, causing it to appear to power up but not do anything. After being left off for 12 hours, the PSU would no longer turn on, and powering the logic board from a normal ATX PSU failed to bring it back to life. The logic board was a hybrid pc design, incorporating some standard PC components (VIA CPU for example, and standard PC133 memory) with custom security ASIC's and compact flash OS card, which sadly isn't replaceable without just buying a new unit.
The backup firewall appears to have faulty network interfaces, and died with fatal exceptions output to the console port. We were able to wipe it's OS, and reinstall, but the same error re-appeared.

Our post incident analysis came to the conclusion that while having a single firewall was clearly a vulnerability, we could not have predicted that both the primary and backup firewalls would fail within hours of one another. When we initially purchased the primary firewall, high-availability was an expensive option on what was a fairly expensive firewall, so the decision was taken to purchase a new unit, and then have a second hand unit stored back at base. It's possible that storing it for 4 years in a non-temperature controlled environment was a contributory factor to the failure, due to expansion of components as they change temperature.

We will shortly be purchasing a high-availability pair of firewalls, as well as a third identical unit to keep at base.

Colocation/Dedicated Server Customers: If you had a custom firewall configuration or VPN into your private LAN, then this will require re-configuring. Unfortunately there appears to be no hard copy of the rules we had setup prior to this failure, and as the unit appears to be BER, we have no way of retrieving them. If you need a rule created, please contact us and we will endeavour to set it up as soon as possible.

Regards,

Alex Threlfall
Network Operations Manager
Cyberprog New Media

Unavailability of services from 17/10/2009@21:30 till 18/10/2009@02:00

Sun, 18 Oct 2009 02:50:00 GMT

We apologise to our Customers for the recent period of unscheduled downtime. Our services were unavailable from 17/10/2009@21:30 till 18/10/2009@02:00, due to a firewall malfunction. Our engineers attempted to troubleshoot the issue remotely, before dispatching an engineer with a spare unit to site just after 10pm. After a heroic battle with the roadworks on the M4, engineers arrived on site to find a unresponsive (though seemingly alive from the lights blinking away) firewall. A reset was attempted, but failed, and the unit was then removed from the rack and replaced with another unit.
Colocation/Dedicated Server Customers: If you had a custom firewall configuration or VPN into your private LAN, then this will require re-configuring. Unfortunately there appears to be no hard copy of the rules we had setup prior to this failure, and as the unit appears to be BER, we have no way of retrieving them. If you need a rule created, please contact us and we will endeavour to set it up as soon as possible.

Regards,

Alex Threlfall
Network Operations Manager
Cyberprog New Media

Recent News

WebMail Issues

WebMail Issues

ASSP Disabled!

Mail Services Upgrade - Complete

Mail Services Upgrade

Firewall Upgrade Complete

Slow Access

WWW2 Unavailability Monday 21/12/2009

Unavailability of services 17/10/2009, 18/10/2009, 19/10/2009

Unavailability of services from 17/10/2009@21:30 till 18/10/2009@02:00