Blog
Forum
StatusIncident: panelbox server (s046 s047 s105 s106) [Resolved]
Update: 2011/12/20 16:00 EST – Issue is now resolved. We will continue monitoring these servers to make sure everything runs normally. If you have any questions or encounter any problem, please contact Funio’s support team (http://www.funio.com/contacts/).
–
Update: 2011/12/13 10:38 EST – During the last couple of weeks, we worked on 3 major issues. An important increase in SPAM and Hacks created serious server resource problems, which lead us to work on overall server performance as well. For a detailed explanation please follow this link: http://blog.iweb.com/en/2011/12/incident-panelbox-server-s46-s47-s105-s106/9406.html/comment-page-1#comment-590129
–
Start: 2011-12-08 12:00 PM EST
Estimated time to resolution: 48 hours
Impact: Mail and http instability
We are currently investigating performance issues affecting some panelbox servers (s46 s47 s105 s106) that could cause delays in web page resolving during peak hours. This means that your website could be slower to open and that mails could give “Timeout” errors when trying to fetch them.
Recently, we have put into place many temporary measures to intervene quickly in this matter but it seems those measures were not sufficient. We are currently working on permanent solutions that will re-establish adequate performances on our shared infrastructure. These measures takes time to put into place properly but we are confident that service will come back to normal in 48 hours.
We will keep you informed about the next steps in this incident.
Comments
Promotion
Highest Rated
- Securing Wordpress against hackers
- An overview of RAID technology
- The Control Center, Our Next Generation Customer Hub
- iWeb launches its new website
- Top 9 Reasons why Web Hosting is Awesome
- A new logo for iWeb
- Smart Group: Optimizing Local Security
- Who has the coolest costume?
- 4 Tools to Create A Website Quickly & Easily
- Season’s Greetings from iWeb!
December 10th, 2011 3:04 pm
AGAIN!!!! I ‘M THE ONLY ONE TIRED OF THIS ????,
I’M SUFFERING INSTABILITY SINCE THE LAST 3 WEEKS ON ( S105 ) CPANEL AND/OR WEBMAIL
ANY COMPENSATION ???!!! ANYTHING, AT LEAST GIVE US A PROPER SERVICE
December 11th, 2011 9:39 am
This is not acceptable, were not receiving any e-mails since the last 3 days . This is an urgent matter !!!!!!
December 12th, 2011 2:49 am
So I had 30 newspaper websites down and lost 3 staff for half a day going through logs to figure out why. Overall financial cost to business – $220. Overall reputation cost – ????
You should have sent an email to ALL your customers!!!!!! So that we could then email all of ours with an answer.
Couldn’t do that, but I see my credit card number changes and you notify me immediately a payment is missed.
How about 1-month credit for all your dedicated server customers?
John
December 12th, 2011 8:11 am
Work without the capacity to send and receive e-mails is problematic.
Please resolve this issue as soon as possible.
December 12th, 2011 8:14 am
I have received a reply to our complaint but I still don’t know what to do in order to receive our e-mails. We need assistance ASAP. Please call at 514-527-1391 ext. 221. Thank You.
December 12th, 2011 9:31 am
I reported problems with S105 on November 30 2011 at about 3pm (our mail server was completely down for about 9 hours). Until today iWeb has been unable to resolve these problems (service is very unstable). We are fed-up with this level of service. Rebuilding the whole server shouldn’t take more than 24 hours (iWeb – what on earth are you doing? Isn’t your reputation important to you?).
December 12th, 2011 1:01 pm
@Martin, @Daniel, @John, @RFA, @lolanda, @Sam,
I do understand and share your frustrations. The last couple of weeks haven’t been easy on all of us but many things have been done. Other interventions have been completed over the week-end and at the moment that I’m writing this message, the targeted servers (s046-s047-s105-s106) are stable. We will continue monitoring these servers closely and I’ll get back to you this afternoon with more details on what has been done over the last couple of weeks.
December 12th, 2011 2:17 pm
Monitoring is not a solution for this issue.
Immediate action is called for.
I will be leaving iWeb if there isn’t a immediate fix!
Weeks of problems are not acceptable!
December 13th, 2011 10:38 am
Martin, Daniel, John, RFA, lolanda, Sam, Lou and all others who are reading this,
As I mentioned earlier, let me share with you more details about what has been done so far in order to stabilize S046, S047, S105 and S106, and how the situation is getting better. Note that many of our interventions also go beyond these four servers.
First, anyone in the IT field will tell you that the hardest issues to diagnose are the ones which are intermittent. This has been the case for the recent server issues we have encountered within the last couple of weeks, therefore it took us more time to identify the specific problematic sources. Also, since we are working with live/production servers, we constantly have to work around certain elements, such avoid shutting down the servers for maintenance, or making sure that any software/configuration update does not affect our customers’ services. Consequently, this adds additional delays to the resolution of the server stability issues.
During the last couple of weeks, we worked on 3 major issues. An important increase in SPAM and Hacks created serious server resource problems, which lead us to work on overall server performance as well. All these are linked together, but let me separate them for the purpose of getting a clearer explanation.
1- SPAM:
.: Effect:
.:: An abnormal increase in SPAM was going out of some SMTP servers;
.:: This lead to our IPs getting blacklisted by anti-spam organizations (RBL);
.:: This uses a lot of server resources which decreases performances.
.: Cause:
.:: We found that the cause of spamming was sometimes intentional (customers who opened an account for spamming purposes). Those customers often use external SMTP servers to spam, which is not allowed and more difficult to flag. Note that these flagged customers are no longer doing business with us;
.:: Other customers were unintentionally spamming (customers who have been hacked – see “hack” explanations below).
.: Interventions:
.:: Asking the anti-spam organizations to un-blacklist our IPs; we cannot ask them to remove us from a blacklist our IPs before we are sure the problem is under control. If we do so, our IPs’ reputations are going to be low and we don’t want to take the risk of having many other IPs blacklisted for a longer period of time;
.:: We re-enforced our SMTP usage policy by blocking the use of external SMTP server connections. We gain more control over the servers’s outbound SPAM if all emails are sent by our SMTP servers;
.:: We have invested in an outbound SPAM filtering solution. This solution works like a regular anti-spam application, but it filters out emails that are going out instead of the spam that is coming in. We are currently in the process of rolling out these changes to all of our servers.
.: Consequences
.:: We have had our IPs un-blacklisted;
.:: The re-enforcement of our SMTP usage policy helped us control the spamming problem from malicious customers;
.:: The use of the outbound spam filter works great for spam filtering (we didn’t have any other IPs blacklisted since our last configuration), but it has slightly affected our servers’s performances (see “performances” explanation below).
2- Hack:
.: Effect:
.:: Website homepages replaced by an hacker message;
.:: Lower server and email performances, since spam is often sent after exploitation of the vulnerability;
.:: Code injection, creating phishing sites, or relaying traffic can cause an account to use additional server resources.
.: Cause:
.:: Many customers do not maintain their CMS (like Wordpress, Joomla and others) up-to-date;
.:: Hackers/spammers are then exploiting vulnerabilities in previous versions and use these accounts to send spam.
.:: or these same hackers can insert malicious code for relaying or phishing.
.: Interventions:
.:: Modifications in server security;
.:: Collaborating with costumers to update their CMS;
.:: Collaborating with external providers to add security measures even with CMS that are not up to date.
.: Consequences
.:: No major hacking has been reported since.
.:: This is “work in progress” and we will continue to improve it continuously. In fact, we are in the process of deploying a global web application firewall based on mod_security that will help prevent most known security vulnerabilities in web applications (Code injection, SQL Injection, information disclosure, etc..)
3- Performances:
.: Effect:
.:: Some servers tend to be slower than usual;
.:: Page loading is longer;
.:: Database requests are taking more time;
.:: Email servers are often the first service to stop when there is a server overload (connection timeouts).
.: Cause:
.:: Since we installed new software to prevent spamming and hacking on the servers, these took more resources than expected;
.:: It causes a larger load of some servers in peak times during the day;
.:: If we stop the softwares, there is a risk that spam might lead to server blacklisting again, or that hacks re-occur more often.
.: Interventions:
.:: We updated CloudLinux on all our servers (see our maintenance blog post here: http://blog.iweb.com/en/2011/11/maintenance-funiopanelbox-shared-hosting-servers/9163.html);
.:: We deployed new servers to unload others, per example S046, S047, S105, S106.
.:: We have migrated some larger accounts from these servers to rebalance disk usage and lessen the load caused server maintenance or synchronizations.
.:: We continued account migrations overnight and during the weekend, and we will continue with other migrations during the next days/weeks when necessary.
.:: Newer servers are built based on the latest Xeon processors and newest Intel Nehalem micro-architecture. Most server components were upgraded to increase overall server stability. Current servers might receive some of these upgrades in the future.
.: Consequences
.:: Migrations of some larger accounts stabilize the performance of the affected servers;
.:: Installation of the most recent version of CloudLinux helps us increase server stability and limits the impact of resource abuse from single clients. This new version also enables us to provide an overall increased security.
.:: The intermittent email problems should be resolved as soon as the load of the servers will be stabilized, although we already see an amelioration on that side.
As you can see, we have done a lot, but there are still many things to get done. We are always working on the servers’ stability and we will continue to do so. We understand how this affects your daily activities and business, and we centre our interventions based on these repercussions. All our team works hard, days, nights and weekends to give you a more stable infrastructure, and we are confidant we will achieve our goals within the next few weeks. Every day, you should see improvements in our servers’ stability.
We invite you to continue sending us your comments, even if they are hard to hear/read some times. We understand how important it is to host your websites on a strong and reliable infrastructure, so we see your comments as improvement opportunities.
For those who are looking for compensation, please fill in the SLA request form from your customer hub and we will be happy to help.
Thank you for your time and understanding,
The Funio Team
December 15th, 2011 1:11 pm
Well it’s the 15th and e-mail hasn’t been to bad until today … Now everyone is getting a message to check their e-mail address and password … no e-mail coming through at all!
We are on s046, but s107 seems to be involved. One of our employees had setup an autoresponder around the time that this mess began. Even though we turned it off, s107 was still sending autoresponders on their behalf…even though our account is on s046! Tried to log in to s107 with the master account with no success. Then someone tried logging into s107 with the e-mail specific login and was successful…and there was the auto responder on s107???? Well we fixed that.
Now we want to update our web-site and logging into s046 and making changes there are unsuccessful. Maybe because our hosting site is now pointing to s107 instead of the right one – s046. Again, tried to login to s107 and can’t login…
Geesh, what a mess! and very, very frustrating.
December 16th, 2011 2:10 pm
According to SORBS, the spam activities continued recently on IP address 184.107.100.65
The server was listed on the 11th and subsequently delisted the same day, a few hours later. I hope you are closely monitoring traffic in order to shut down the spammer(s) using this IP address.
December 16th, 2011 4:34 pm
Hi M. Morin. As mentionned in the update above Funio and iWeb are closely monitoring the situation. I also made sure to forward your comment to our abuse department (abuse@iweb.com).
December 20th, 2011 3:07 pm
Mr Pearce, your comment was not deleted, just caught by our comment-spam filter (false positive, sorry). I found it in the quarantine and approved it.
I’ve asked Funio’s support team to give me a status update and the issue is now resolved. Have you had any related issue with your website lately?
January 3rd, 2012 3:15 pm
We are currently again unable to send emails …
We are on s105
“Server error: ‘421 Unexpected failure, please try later”
January 6th, 2012 9:08 am
Hello Sam H.: I may be wrong, but his error message looks like a problem with an application that runs in your account on s105, not necessarily a problem with the mail service (which I was confirmed runs correctly). If the problem persists, please open a ticket about this in your customer-hub so Funio support team can make the necessary verifications. Also send me the number (to community@iweb.com) so I can track the request in our system.