Incident: Network infrastructure (DDoS) [Resolved]

Published on October 31, 2011 at 3:10 pm by chefsc in: Status

Latest and final update available here: http://blog.iweb.com/en/2011/11/incident-network-infrastructure-ddos-resolved/8953.html

–

Update: 2011-11-01 21:30 EDT – We are still monitoring the situation while our network team is working on resolving this incident for good.

At this current time, a small amount of users may still be experiencing issue seeing website hosted through privatedns.com DNS server for the most part, most users will not experience any issue. Our network team is working on getting this service available for the users experiencing connectivity issue.

We thank you for your patience, if you have any question or issue with the service, please contact our technical support team at http://iweb.com/contacts.

iWeb’s latest official statement that documents the DDoS attack that we are dealing with is available here: http://blog.iweb.com/en/2011/11/incident-network-infrastructure-ddos-update/8916.html

–

Update: 2011-10-31 15:10 EDT – We are aware that several of our customers still encounter connectivity problems. Network team is actively working on a permanent solution. We thank you for your patience.

–

Update: 2011-10-31 10:00 EDT – The situation is still stable. However, if you encounter packet loss and low network performance please contact support directly via livechat, telephone, or customer-hub.

–

Update: 2011-10-30 20:00PM EDT – The situation has been stabilized and network performance has improved. The destination of the attack seems to have been isolated. Our DNS are now responding fast and access to our sites, including the customer-hub, is stable.

If you still encounter packet loss and low network performance please contact support directly via livechat, telephone, or customer-hub.

–

Update: 2011-10-30 2:15 PM EDT – The DDoS attack has started again and our network team is doing everything in their power to minimize the impacts. The attacks have been quite massive and we are splitting the traffic on different network links in order to avoid congestion.

–

Update: 2011-10-29 3:40 PM EDT – The situation is under control and only a minority of clients might still be experiencing network latency or packet loss. Our network team is actively monitoring the situation in order to minimize the impacts for our customers.

–

Update: 2011-10-29 2:40 PM EDT – The attack is now ongoing and is currently affecting domain names using the privatedns.com name servers. A minority of clients might also be experiencing some network latency issues. We will communicate with you on a regular base to provide update throughout the process.

–

Update: 2011-10-28 11:30 PM EDT – The attack is being mitigated and the situation is under control. We will keep a close eye on the situation in order to minimize the impact in case the attack would increase. If you have any questions, please do not hesitate to contact our support team (http://www.iweb.com/contacts/).

–

Update: 2011-10-28 10:30 PM EDT – The attack is currently being mitigated and we are still monitoring the situation to ensure that our DNS services are working properly.

–

Start: 2011-10-28 9:30 PM EDT
Estimated time of resolution: 60 minutes
Impact: An important DDoS attack is currently affecting clients using privatedns.com DNS servers.

We are currently investigating a situation that might affect a segment of our network infrastructure.

Our technical team is actively working on getting the services available. We will communicate with you on a regular base to provide update throughout the process.

We apologize for any inconvenience this situation may cause. If you have any questions, please do not hesitate to contact our support team (http://www.iweb.com/contacts/).

Thank you for your patience and understanding.

Comments

  1. alex
    October 29th, 2011 1:13 pm

    i hate how people think performing ddos attacks are fun. Wish they would get a life. Also for those that get mad at iweb please grow up. DDOS attacks happen daily to all hosts. Its part of being on the internet. Hope you guys can put a stop to the attackers.

    Thanks
    Alex

  2. George
    October 29th, 2011 1:17 pm

    by saying that those who use the DNS privatedns.com are affected?
    I do not use that service, I have only 2 dedicated servers with iwebs and are affected, loss of ping and connectivity, intermittent lag …

    are more sincere, instead of saying that only the DNS is affected, must say that the attack affects datacenter connectivity

  3. Rat
    October 29th, 2011 1:42 pm

    At first i thought i was just getting DDoS’d (i run two minecraft servers that get griefed from time to time) but why iWeb as a whole?

    i hope you guy’s can get this fixed (ironic that you sell DDoS protection)

  4. George
    October 29th, 2011 1:46 pm

    iweb took 2 months and only bad experience has given me a several occasions in just 2 months.
    I thought that this network would be stable but it is not.
    is vulnerable and unstable.

    — 70.38.38.213 ping statistics —
    Packets Transmitted 74, 43 received, 41% packet loss, time 72961ms
    rtt min / avg / max / mdev = 66.911/68.900/70.059/0.466 ms

  5. George
    October 29th, 2011 1:50 pm

    PING 70.38.38.213 (70.38.38.213) 56(84) bytes of data.
    64 bytes from 70.38.38.213: icmp_seq=13 ttl=51 time=68.6 ms
    64 bytes from 70.38.38.213: icmp_seq=23 ttl=51 time=68.8 ms
    64 bytes from 70.38.38.213: icmp_seq=39 ttl=51 time=68.4 ms

    — 70.38.38.213 ping statistics —
    43 packets transmitted, 3 received, 93% packet loss, time 41999ms
    rtt min/avg/max/mdev = 68.470/68.644/68.812/0.255 ms

  6. Steve
    October 29th, 2011 2:16 pm

    And it’s happening yet again. This is getting beyond ridiculous. I’m losing clients over this… sigh.

  7. Steve
    October 29th, 2011 2:24 pm

    While I’m sure their tech support is highly experienced, they’ve not been able to deal with these attacks over the past few weeks. I hope iWeb is looking to bring in some specialized help to solve this problem for them.

  8. Stéphane Jose
    October 29th, 2011 3:17 pm

    Thanks for your patience everybody. We know that the situation is highly unpleasant.

    @George: as stated in the 14:40 update a minority of clients are also experiencing some network latency issues.

    The network team is working on it.

  9. Catherine
    October 29th, 2011 5:17 pm

    Like others are saying this has been happening more frequently recently, I’ve been with iweb for a few years, and this month this is the 4th attack, well 3 since this afternoon’s attack apparently counts with yesterday’s. Like the poster Steve I’m losing clients over this and also getting a lot of complaints. I’m sure you’re working on resolving it, I just hope this is the last of these, iweb has given me good service for a long time, I’d hate to have to find a new host.

  10. Michel
    October 29th, 2011 7:21 pm

    Our server is again inaccessible, 5th attack in 3 days, please solve the problem we lose money and visitors – a minority of customer but each time the same…. the situation is bad…

  11. Evans
    October 29th, 2011 7:28 pm

    I still can’t access my site. and it’s playoff time. This is not good. can someone cut off the cogent attack and use your BGP network to reroute your traffice over another network?

    Isn’t Iweb a multicarrier provider ?

  12. r4
    October 29th, 2011 7:34 pm

    Please help me cancel this iWeb Shared Hosting r4ds-r4i.com, because I have another iWeb hosting r4i3dsgold.fr, I very want to my domain r4ds-r4i.com host in my r4i3dsgold.fr hosting.

  13. Michel
    October 29th, 2011 7:39 pm

    The situation become totally bad… how i update my websites now – we agree with you Evans

  14. Mike
    October 29th, 2011 7:48 pm

    i’m running an asterix server in the rack of iweb and my clients are being affected for the past 3 days.
    now i wonder, when iweb is going to fix the problem and just not patch it.

    i’m affraid to go back in the office monday, just to know how many clients i loss.

    please guys do something fast.

    thx

    Mike

  15. Ryan
    October 29th, 2011 7:57 pm

    This problem is still ongoing. When I run MTR I get a bunch of dead connections starting at te8-4.cl-core06.cogent.mtl.iweb.com. The loss is almost 90% now, it has been hovering at around 80% for the past half hour.

  16. John
    October 29th, 2011 8:13 pm

    How is an attack on their DNS service affecting dedicated users? Are we all behind the same choke point and if so wouldn’t segregating operations allow one to at least throttle attacks more effectively? BGP portability won’t do much good if all of the subnets are bundled together.

    Anyway a great case is being made here for datacentre diversification.

  17. Michel
    October 29th, 2011 8:22 pm

    work now hope IWEB has denitively solved the problem

  18. Steve
    October 30th, 2011 12:08 am

    And yet again, for the 3rd time today… it’s happening! This really needs to get fixed.

  19. Matt
    October 30th, 2011 12:25 am

    and we’re down again… i was planning to migrate some other servers over to iweb next week. i’ll be holding off on that decision now.

  20. Tamouh
    October 30th, 2011 1:27 am

    We appear to be experiencing intermittent network issues right now. Packets are dropping alot

  21. Pierre-Luc
    October 30th, 2011 3:08 pm

    Again…. This becomes completly ridiculous !

    I know a DDos attack is not easy to control, especially when it’s massive. But do you think you’ll get over it one day ? It’s been 4 days my sites go up and down and I am afraid because tomorrow is the beginning of the week and ~100 employees here use emails/online billing system/websites, and I don’t even want to think about lost visitors/customers.

    I am seriously thinking about moving my 2 servers to another hosting company which knows how to manage ddos attacks.

  22. Christopher
    October 30th, 2011 3:54 pm

    I’m the CEO of a game publisher and recently we migrated to iWeb for hosting solution on one of our MMORPG. We have intented to migrate the remaining of our games, client and browser, to iWeb but this issue of DDOS is getting too frustrating.

    It has been for few days now and being a game publisher, we have more than few thousand online players complaining to us thereby affecting our business and revenue stream. It is even more absurd that what we have signed under the SLA isn’t fulfilled with obligation. Seriously if this doesn’t stop by today, our company may consider seeking alternative or necessary action for our losses!

  23. Ricky
    October 30th, 2011 4:44 pm

    i`m customer of iWEB for many years, but what is going on in the last 6 month make me and my customers more and more unhappy, the outage today has involved full network and is still going on…I have ~1000 users dipending my and iWeb services, a bad situation…I hope all this comments will turn back iWeb like befor, an EXECELNT Company!!!

  24. Christopher
    October 30th, 2011 4:53 pm

    This is way too outrageous. Been for an hour now and our game is still very laggy where most of our players couldn’t login. Please quickly rectify the problem.

  25. Tom
    October 30th, 2011 7:34 pm

    This became really funny. We migrated our server to iWeb few days ago thinking how they are professional men, working hard with good service and even better support… And what we got? Well i will tell you what we got. We got thousands unhappy customers, our server is down without any clear date or IDEA when it could go up and stable, we got a dude from their online live chat technical team, Jean-Pierre, being very rude and not interested to help, let me quote what he said: “That is correct, the best i can advise would be to contact our customer satisfaction during their business hours for compensation, if you still choose to go for a different host, then they will be able to assist with the account closure” and logged out afterwards from the chat without further notice?!?!?! Okay, i mean i know he is under pressure, but aren’t we all in these moments? We are losing MONEY here, and don’t even bother to answer properly? Go to their home page and see the reasons “Why to chose iWeb?” and tell me… really, why to chose iWeb?
    I am sorry if being rude but please do understand me, my money and time is on table here..

    Regards,

    Tom

  26. Stéphane Jose
    October 30th, 2011 7:48 pm

    @Catherine @Evans @Michel @John @Matt @ Steve @Pierre-Luc: The DDoS attack was exceptionnally vigourous (around 20Gbps). Traffic was rerouted through various carriers and our network team used various mitigation solutions to lower the impact on our clients as much as possible. At one point rerouting this traffic was necessary to protect the whole network, keeping it functionnal although degraded.

    The situation has been stabilized and the destination of the attack seems to have been isolated. Our DNS are now responding fast and access to our sites, including the customer-hub, is stable.

    If you still encounter packet loss and low network performance please contact support directly via livechat, telephone, or customer-hub (choose the “server-down” option).

  27. Stéphane Jose
    October 30th, 2011 7:51 pm

    @tamouh @Mike Do you still experience packet loss?

  28. Stéphane Jose
    October 30th, 2011 8:13 pm

    Tom, I am very sorry you feel that way. I did not find that the phrase you quoted was particularly rude or impolite but I can understand your lack of satisfaction. In any case if you still wish to ask for compensation there are two ways: you can ask iWeb to honour its 30 day satisfaction garantee (if the order date of your server falls within the 30 days period) or you can ask for SLA credits (according to our power and uptime garantee). In both case you have to contact customer service department or open a ticket via the customer-hub (send me the ticket number at community@iweb.com so I can follow up with CS).

    I also invite you to leave a comment to management (also via the customer-hub).

    If you still wish to stay with us, which I sincerely hope, note that the information about the complete resolution of this incident, will be posted on this very blog as soon as the network team informs me.

    A post mortem is also expected in the coming days.

  29. Stéphane Jose
    October 30th, 2011 11:25 pm

    Hi @r4, your question is not in the right thread. Actually requests about shared hosting and domain name configuration should be directed to Funio.com. I asked Funio support team to look into it for you.

  30. Tamouh
    October 30th, 2011 11:43 pm

    @Stephane, we seem to be good now. Thanks! Glad to hear you got handle on it. DDoS is a b*tch in this day and age. Wonder who was the target?

  31. Steve
    October 31st, 2011 8:26 am

    It’s happening again…

  32. Paul
    October 31st, 2011 8:46 am

    I’m guessing it has started again?
    As of 8am (ET) my dedicated server has become up unreachable at times with major packet loss between
    te8-2.v0701.co-core09.mtl.iweb.com and new-1.hd-core01.torix.mtl.iweb.com

  33. Stéphane Jose
    October 31st, 2011 8:57 am

    @Tamouh thanks for the feedback!

  34. Stéphane Jose
    October 31st, 2011 9:00 am

    @Steve @Paul Can you open a ticket and provide support with a traceroute from your side? Please send me the RT number, so I can followup with support (write community@iweb.com)

  35. Tom
    October 31st, 2011 9:16 am

    It just happened again… Any perm solution for this issue?

  36. Paul
    October 31st, 2011 9:17 am

    @Stéphane seems to be OK at the moment I will monitor and contact you if something comes up.

  37. Steve
    October 31st, 2011 9:20 am

    @Stéphane – It appears to have improved but next time I’ll do the traceroutes and send you (and support) the results. Thanks.

  38. Christopher
    October 31st, 2011 9:38 am

    @Stéphane – Our players are getting disconnected from the server again. The server that is last traced before the first timeout in the tracert is te8-2.v0708.cl-car08.mtl.iweb.com, if this information helps. It has been for 4 days without a solution from iWeb. Being a game publisher, this is affecting our business revenue and causing great damages to the company’s goodwill.

    Please get a permanent fix and not a case of fire-fighting letting your customers suffer from the DDOS attack daily!

  39. Stéphane Jose
    October 31st, 2011 9:59 am

    @Steve @Paul thanks for the feedback!

    @Tom As I mentionned previously please provide support with a traceroute from your location (open a ticket via the customer-hub (our sites are available now) and send me the ticket number (community@iweb.com)

  40. Stéphane Jose
    October 31st, 2011 10:37 am

    @Christopher: thanks I mentionned this to our support team.

  41. Juan
    October 31st, 2011 10:56 am

    another attack this morning? my users called very upset because their websites and other applications were extremely very slow. I have 2 private servers with IWEB and they both were affected. I wonder how their network configuration is. I know that IWeb has the money back guarantee and all that, but what about me loosing customers because of these issues, that is way more expensive.
    This is becoming a big problem. I can tell my customers 1 or 2 times that there was a problem because of a DDOS attack and they will understand, but after the 3rd time, they wonder how professional I am, the worst is that they question me, not IWEB about the services….
    Today I will start considering moving to another hosting provider.

  42. Stéphane Jose
    October 31st, 2011 11:40 am

    @Juan: goblincreative.com is not hosted at iWeb. What is the ip and domain of the affected servers? please send details to community@iweb.com.

  43. Christopher
    October 31st, 2011 1:35 pm

    Hi, I have just sent a ticket #4858665. The DDOS is back. About 300 of our players in the game suffered a disconnection and currently having problem to login to our game server. Those who managed to login are having a severe lag. We did several tracert and found ping as high as 300 to 600 ms.

    This is really getting very frustrating and we seriously hope that iWeb inform us if there is a permanent solution.

  44. rob
    October 31st, 2011 1:45 pm

    Same problem here. Shut down a webcast we were doing, ground to a halt 12:50pm-1:45pm Eastern.

  45. rob
    October 31st, 2011 2:12 pm

    all good now!

  46. Pierre-Luc
    October 31st, 2011 2:13 pm

    I am even not able to stay connected enough time on my FTP to make a simple website/mails backup and I’ve been trying for few hours now. I start to feel anxious since we’re having several server disconnections and I already received 4 calls from customers telling me they are not able to reach our system to get necessary informations for stuff we sale.

    My boss told me this was absolutely unacceptable and I’m taking the bullet for Iweb. In 8 years of dedicated server, I NEVER experienced any Ddos attacks with RackSpace/ThePlanet/OVH and when I bought with Iweb via a seller named Mohammed K., this one told me I can get assured that Iweb’s network never falls down.

    Seriously, I don’t care that issues are critical from your side, as other companies like RackSpace or Google are really more quick to solve that kind of problem. This is like if you were taking random actions and you don’t even know how to completly solve the problem.

    If you want to offer profesionnal stuff for profesionnal companies, please develop a security/network department with administrators that clearly know how to manage that kind of attacks and PLEASE, FOR GOD’S SAKE, LET US KNOW WHAT’S HAPPENING, IF YOU EXPECT OTHERS ATTACKS AND IF YOU FINALLY FOUND A DEFITIVE SOLUTIONS AND NOT ONLY PATCHES.

    I’m getting exhausted and I am really stressed now. I don’t even know what to say to the employees because I don’t even know what is happening exactly, cause Iweb gives us no details or almost. I don’t want to live a complete week with that kind of random issues.

  47. Christopher
    October 31st, 2011 3:00 pm

    What is really going on?

    When we think that the server is stable for our users, 5 mins ago all our players were having latency issue and getting disconnection problem. DO YOU READ ME AND COPY THAT WE ARE LOOSING OUR BUSINESS AS A GAME PUBLISHER BECAUSE OF YOUR POOR SERVICES IN HANDLING THIS DDOS ISSUE? Don’t you have a good CISCO router, maybe a 3000 series or 5000 series to clean traffic, or a blackhole solution on those attack. Least you could do is to build a honeypot to divert those attack away from your client’s servers.

    I give myself and iWeb the very last straw for this issue to be resolved within the day. Else we will consult our legal department for necessary action and migrate away from iWeb.

  48. Gabriel Latour
    October 31st, 2011 3:25 pm

    We will be loosing VoIP business here.

    Gabriel

  49. Gabriel Latour
    October 31st, 2011 3:26 pm

    Any way to route our network trough another *quality* provider?

    Gabriel

  50. alex
    October 31st, 2011 3:28 pm

    dont know why they think its fun to do ddos attacks. hope you guys stop those losers. Also all of your threating to leave just leave. omg how old are you. crap happens. welcome to the world of networking.

  51. Anthony Alberto
    October 31st, 2011 3:43 pm

    Hello,

    Intermittent connectivity issues on all my Smart Servers (I have 6) from 2:30 pm to 3:15 pm

    Did you fix the problem or will it happen again ??

  52. Stéphane Jose
    October 31st, 2011 3:52 pm

    @Alex: Thanks for your understanding!

  53. Stéphane Jose
    October 31st, 2011 3:53 pm

    @Anthony Alberto: issue affecting Smart Servers seems under control now. Please open a ticket via customer hub if it reoccurs (use “Server Down” option).

  54. Stéphane Jose
    October 31st, 2011 3:56 pm

    @Gabriel Latour: We understand you situation. I can only tell you that our network team is actively working on a permanent solution but I do not have any additional information at the moment.

  55. Marco
    October 31st, 2011 4:13 pm

    honestly not impressed… most of our business is “live” feeds… RDP, VoIP, Streaming, etc… How can I explain to a customer that our Datacenter connection experience more problems than a Home Cable modem???. My Quality of service has been ugly for the last 2 days, I’ve been apologizing all day and handing out credits. To my opinion, you’re FAR FAR FAR off the SLA I signed for in my contracts for my 2 cabinets.

  56. alex
    October 31st, 2011 4:13 pm

    thanks for your hard work!

  57. Pierre-Luc
    October 31st, 2011 4:50 pm

    Alex, this is not a question of beeing mad or whatever, but you must understand that if we, as developpers, understand how Ddos attacks work, this not means visitors/customers/employees and bosses will do the same. I would also like to say “thanks for your hard work!” myself, but I don’t even know the actions Iweb took and how many techs/admins work on this critical issue, as they don’t keep us up to date clearly, except for saying “we’re on the case, some server might experience latency” for the 5th or 6th time in four days.

    I am personally responsible of an online system that ~100 employees here use everyday. When the server falls down for “only” 1 hour, can you imagine the money my boss and its entreprises loose ? This is why he’s so mad and this angryness is, of course, redirected on me and my credibility.

    If I was running websites for fun, I wouldn’t take these downs so badly. This is so simple to say “If you don’t like what Iweb did till now, just leave”. This is really more complexe than only moving a lot of data and switch the DNS. But if you are a profesionnal in the business, you already know it. Actually, even if I wanted to leave, I couldn’t because I’m always loosing connection because the end of any transfer (many packet loss).

    When we chose Iweb, we’ve been told by the seller that there’s no down garanteed. You can also read this here (http://iweb.com/about-us/networks/) : “We understand that network uptime is crucial for your valuable online presence and this is why iWeb’s entire network is fully redundant. From the Cisco routing equipments to the upstream bandwidth providers, there is no single point of failure. At iWeb, your servers will be accessible and online 24 hours a day, 7 days a week and 365 days per year.”

    We can afford one down from time to time. But not 5 times in the same week. I already assisted a Ddos attack defense years ago when I was a student and I had the chance to work with administrators that know how to deal with. You talk like ddos attacks were almost a norm in the industry, which is not the case when you got the good people in place. Otherwise, big enterprises like Google/Apple/RackSpace/etc. would always be down, if you only knew how many ennemies they collected by last years !

    You must also understand that when you pay the big bill to get something profesionnal to avoid this kind of situation, you deserve what you pay for.

    By the way, why the hell Iweb is showing its bandwidth monitors (http://iweb.com/about-us/networks/usage-graphs/) ? I don’t know the structure of their network, but I hope the ddos attackers can’t use theses graphs to target specified provider to make things go worst.

    Hope this is gonna be fixed really soon, because I got the order to switch for another hosting company if we experience another down… this sucks.

    Pierre-Luc

  58. George
    October 31st, 2011 4:58 pm

    this seems to be endless, my servers experience a 17 to 20% packet loss

  59. Coaster
    October 31st, 2011 5:44 pm

    @IWEB Thanks for the attempts to mitigate the attacks.

    @Marco 20+ Gbps DDos is nothing to scoff at, be patient and a solution will come.

  60. Pavel
    October 31st, 2011 8:05 pm

    Finally I can open iweb.com (I do not even say about our web site and emails – up to 80% lost packets since last Wednesday)! Online support keeps saying, this is not their problem but rather our ISP and the same time refusing to conference call with our ISPs (BELL, ROGERS, TELUS) to resolve this issue, because they know they cheat by purpose – this sounds technically very unprofessional for the company stating 99.9% on. And my “Server down” ticket is never responded or processed since Friday! I do not know what to say to my bosses and co-workers as the DDoS for the third time during the short period does not sound true to them. The worst thing is, after sitting on hold for an hour, even the call support provides no ETA to resolve this nightmare (I do not even say about the online chat)!

  61. Nicolas
    October 31st, 2011 10:19 pm

    I am very disappointed by the service and by the supposedly “99.9% reliability”. We have a shared hosting and we have been hacked 4 times in the past 3 months. Furthermore, the network is oftentime very slow and unstable.

    They are far from being reliable and secure.

    I DO NOT recommend iWeb.

  62. Christopher
    November 1st, 2011 5:48 am

    @ alex – You can’t be saying the same for others who have suffered business loss due to the consistent DDOS attack on iWeb. I put to you that the attack started on 28 Oct 2011. Since that day, the attack has been massive as pointed out by iWeb.

    DDOS attacker who launches massive attack on an organization are either out for vengence or extortion. They will not stop the attack until they have achieved their objectives or realised that the victim has secured a mitigation solution. iWeb, being a veteran in this industry, should know better what is a DDOS, a botnet attack, etc. The disappointment here can be understood because it has took more than 5 days where iWeb did not satisfy their customer with a permanent solution. Rather, they adopt the most unreliable approach by trying to rerout traffic to different carrier and minimise the impact.

    Under the contractual terms as stated in iWeb’s SLA, they have failed in their up-time obligation. As a customer, we are able to accept failure on the part of iWeb if the DDOS has been permanently fixed with a mitigation solution within 1 or 2 days but don’t you think 5 days is way too ridiculous? Guess what? I just spoke to their Live chat staff and you can see what was the reply:

    Christopher: So can we be assured that there will be no further DDOS attack again as it has been on-going since 28 Oct.

    Nick P.: You can post a comment in the blog to have more information. Ddos attack is hard to predict or to control 100% in any case

    So basically, iWeb is sending a message to their customer that they may still encounter packet loss due to DDOS and there is nothing that can be done?

    Yesterday, when we first experience the packet loss in our game server, we quickly send an urgent ticket to iWeb and inform them that the DDOS seems to have emerged again. Instantly, the reply came back to inform us there is no DDOS. We sent some tracert test result and surprisingly they asked for more tracert test. 5 hours passed and finally a reply came back to us that iWeb is suffering a massive DDOS attack.

    This is absolutely unprofessional. Without rubbing more salt to the wound, I await an explanation here since the ticket I have sent yesterday has been closed without a reply and as advised by Nick P from Live Chat!

  63. Alain Voukirakis
    November 1st, 2011 8:12 am

    Just to let you know guys that it’s not just IWEB that got attacked by this DDOS attacks, many universities and others ISP’s too as far as I can see. Don’t forget these F*C*ing guys who make these attacks exploits lacks of routers infratructures and softwares. their goals are to slow down traffic and make ppl like us to complain and get headaches. I am in same position as you having my sites responds slowlier, have files uploaded aborted…
    Please understand that NO ONE have full control on the traffic on the Internet network.
    Better to send to your customers notice about the attacks. Hope these guys will be sue if we can trace the origin.

  64. Marco
    November 1st, 2011 8:32 am

    How come we can’t consult your traffic graphs to mesure the impact of the atttack?

    http://iweb.com/about-us/networks/usage-graphs/

    please don’t blame the MRTG server… I can have my ports graphs

  65. Pierre-Luc
    November 1st, 2011 9:18 am

    @Marco : I think this is really more secure to remove those graphs from their website, because it could be used by attackers to target the best networks to make things crash again, depending, of course, on the infrastructure of Iweb’s network. However, it’s been useful for us to see the attacks and I already made printscreens of those graphs, especially for Cogentco, as our classic server is using this provider.

    We experienced 6 downs till now, “graph approved”.

    But one of the worst thing is Iweb keeps on posting blog posts (http://blog.iweb.com/en/2011/10/thailand-a-shortage-of-hard-drives-announced/8881.html) while we’re still not getting informed of any development here. Big priority issue here.

  66. Stéphane Jose
    November 1st, 2011 10:22 am

    @Pierre-Luc @Marco We have removed the MRTG graph, the rest of the page will be removed later on.

    @Christopher @Nico @Pavel @George: Really sorry for the unpleasant situation and we understand your lack of satisfaction. It’s really painful for all of us. Rest assured the network team is hard at work to find a permanent solution, but unfortunately still no ETA.

    @Alain @Coaster: thanks for your support!

  67. Stéphane Jose
    November 1st, 2011 11:24 am

    @Pierre-Luc: Sorry you felt that way. I am the one who wrote the article, it was already scheduled for publication before the DDoS issue, and as the communications guy, I am not the one doing the actual network work!!! Still I can manage writing articles AND answering comments on the blog ;-)

    I can understand your frustration, but publishing one unrelated article on the blog does not mean that the network team is not working hard on the issue that affects you and several other of our clients.

    We simply published the story about Thailand because it is newsworthy and that this situation could ultimately affect many companies. We are monitoring this too.

  68. Pierre-Luc
    November 1st, 2011 11:29 am

    Thanks for the specification Stephane. I just hope everything’s gonna get fixed ASAP, because if I am so mad, this is because I have to deal with employees and bosses really more harsh then me.

    I just realized that we can’t ping iweb.com anymore even tought we can still access the website; is this part of the new solution to avoid “Ad” without “hoc” response by overpinging iweb’s network ?

    Thanks,
    Pierre-Luc

  69. Emil
    November 1st, 2011 1:34 pm

    My servers are very slow to connect, and I can’t even login to the iweb online interface … is this being resolved?

  70. Denis
    November 1st, 2011 3:43 pm

    Looks like DDoS is back on.

  71. George
    November 1st, 2011 3:57 pm

    again problem??

    my servers is slow, and ping lost

  72. Matt
    November 1st, 2011 4:05 pm

    losing connections again :(

  73. Andre
    November 1st, 2011 4:09 pm

    Same thing here, but it’s seem back to normal.

    Pinging my.server [my.server] with 32 bytes of data:
    Request timed out.
    Reply from my.server: bytes=32 time=92ms TTL=52
    Reply from my.server: bytes=32 time=92ms TTL=54
    Reply from my.server: bytes=32 time=127ms TTL=54
    Reply from my.server: bytes=32 time=92ms TTL=52
    Request timed out.
    Request timed out.
    Reply from my.server: bytes=32 time=92ms TTL=52
    Request timed out.
    Request timed out.

    Ping statistics for my.server:
    Packets: Sent = 10, Received = 5, Lost = 5 (50% loss),
    Approximate round trip times in milli-seconds:
    Minimum = 92ms, Maximum = 127ms, Average = 99ms

    ping my.server -n 10

    Pinging my.server [my.server] with 32 bytes of data:
    Reply from my.server: bytes=32 time=124ms TTL=54
    Reply from my.server: bytes=32 time=125ms TTL=52
    Reply from my.server: bytes=32 time=154ms TTL=54
    Reply from my.server: bytes=32 time=238ms TTL=52
    Reply from my.server: bytes=32 time=175ms TTL=54
    Reply from my.server: bytes=32 time=113ms TTL=54
    Reply from my.server: bytes=32 time=110ms TTL=52
    Reply from my.server: bytes=32 time=150ms TTL=54
    Reply from my.server: bytes=32 time=110ms TTL=52
    Reply from my.server: bytes=32 time=114ms TTL=52

    Ping statistics for my.server:
    Packets: Sent = 10, Received = 10, Lost = 0 (0% loss),
    Approximate round trip times in milli-seconds:
    Minimum = 110ms, Maximum = 238ms, Average = 141ms

    ping my.server -n 10

    Pinging my.server [my.server] with 32 bytes of data:
    Reply from my.server: bytes=32 time=118ms TTL=54
    Reply from my.server: bytes=32 time=130ms TTL=52
    Reply from my.server: bytes=32 time=121ms TTL=54
    Reply from my.server: bytes=32 time=157ms TTL=54
    Reply from my.server: bytes=32 time=161ms TTL=52
    Reply from my.server: bytes=32 time=111ms TTL=54
    Reply from my.server: bytes=32 time=121ms TTL=52
    Reply from my.server: bytes=32 time=141ms TTL=52
    Reply from my.server: bytes=32 time=113ms TTL=54
    Reply from my.server: bytes=32 time=112ms TTL=52

    Ping statistics for my.server:
    Packets: Sent = 10, Received = 10, Lost = 0 (0% loss),
    Approximate round trip times in milli-seconds:
    Minimum = 111ms, Maximum = 161ms, Average = 128ms

    ping my.server -n 10

    Pinging my.server [my.server] with 32 bytes of data:
    Reply from my.server: bytes=32 time=27ms TTL=54
    Reply from my.server: bytes=32 time=27ms TTL=54
    Reply from my.server: bytes=32 time=27ms TTL=52
    Reply from my.server: bytes=32 time=27ms TTL=54
    Reply from my.server: bytes=32 time=27ms TTL=52
    Reply from my.server: bytes=32 time=27ms TTL=52
    Reply from my.server: bytes=32 time=27ms TTL=54
    Reply from my.server: bytes=32 time=27ms TTL=52
    Reply from my.server: bytes=32 time=27ms TTL=54
    Reply from my.server: bytes=32 time=27ms TTL=54

    Ping statistics for my.server:
    Packets: Sent = 10, Received = 10, Lost = 0 (0% loss),
    Approximate round trip times in milli-seconds:
    Minimum = 27ms, Maximum = 27ms, Average = 27ms

  74. Pierre-Luc
    November 1st, 2011 5:23 pm

    Just got a flash : did Iweb think about using a Blackhole DNS server ? This might help with those ddos attack.

    PL

  75. Stéphane Jose
    November 1st, 2011 5:49 pm

    Unfortunately no, according to our network admins, a Blackhole DNS server is not useful for the type of DDoS we are dealing with.

  76. Adrien
    November 1st, 2011 8:13 pm

    Tuesday 20:12 problem still happening:

    [root ~]# ping http://www.iweb.com
    PING iweb.com (174.142.1.18) 56(84) bytes of data.
    From te8-1.v0699.cl-core06.mtl.iweb.com (67.205.127.86) icmp_seq=7 Packet filtered

    — iweb.com ping statistics —
    8 packets transmitted, 0 received, +1 errors, 100% packet loss, time 7009ms

  77. Marco
    November 1st, 2011 10:26 pm

    @Stéphane Jose, @ Pierre-Luc

    I do understand why you would want to remove the graphs
    Still the graphs can be handy at times… why not just put them in the client hub behind security and not at the sight of anyone on the web but only for your actual clients or maybe COLO clients like us who need more than a “simple” website. And while at it… why not show us the latency on each link.

    Another thought like that… user focus group anyone??? especially for colo users like us… our needs go FAR beyond what your “in house” services provide most of the time and good quality hosting is a major concern for a lot of other colo clients like me, espicially for live services… cloud, VoIP, RDP… etc

    ohhh…. also… HAVE YOUR DAMN BLOG HOSTED SOMEWHERE ELSE WE NEED TO GET INFORMED WHEN SOMETHING GOES BAD!!! YOUR BLOG BEING HOSTED BEHIND THE PROBLEMS, IS FRUSTRATING WHEN PANIC KICKS IN… place yourself in our seats…

    - Bandwidth problems
    - no one giving straight anwsers on the live chat
    - a blog entry hours late
    - no one answering tickets
    - no/flaky access to the website or blog to at least get a glimps of info….

    you never actually know if the datacenter is on fire, vanished in another dimension or someone just created a “network loop”

  78. Marco
    November 1st, 2011 10:28 pm

    @Alain Voukirakis: have any links on this???

    I know lotsof my clients who would like to read on this including me.

  79. Henry
    November 1st, 2011 10:33 pm

    Please fix it. My server is still down……

  80. Stéphane Jose
    November 1st, 2011 11:52 pm

    Latest update available here: http://blog.iweb.com/en/2011/11/incident-network-infrastructure-ddos-update/8916.html

  81. Mathieu
    November 2nd, 2011 12:07 am

    Our customers are mad..

    Do you have news about the progress of the situation ?
    Our websites are still down and our server is down.

    We really need a solution as soon as possible!!!!

  82. Marc O. Chouinard
    November 2nd, 2011 1:28 am

    Thinking that iWeb (or any other provider) can be 100% functional is ridiculous. Though claiming to be 100% up time guaranteed when they have several power, and network issue within the last 8 month might be considered false claim, and if it were my business, I would remove those out of respect for customer who are down or affected.

    I have half a rack with iWeb, and the actual facility isn’t bad (My primary data center have no bathroom, so it less stressful to go to iweb for long stretch of time). But I did also get lot of problem with them since the 8 month I’m setup. They try to offer a balance price/quality that I believe is hard to beat. I got 3 site, all with different ISP and all interconnected via a private layer2 network. So DOSS or not, that internal network wont be affected(Unless hardware failure or fiber optic get cut… and power failure ;). AND I can get IP assigned across all my 3 data center.

    So right now, even if I can’t trust iWeb internet connection (I’m on the HQ bandwidth), I still have my private network with ip from my second ISP that I can use on my server at iweb.

    What bottered me most is the lack of update and information communicated in the past 4 days. Every time I tried to chat with someone at iweb, I get response that everything should be ok, look fine, but it definitely doesn’t look like it, and when checking the blog, it either not functional ( You should REALLY move to a different provider your status page…), or it just a same 1 day old update saying the same thing than the previous day. I find this surprising, since i found that normally iweb were pretty good at updating their blog about issues. But I believe that this time, even the guys on the chat didn’t had good communication or tool to see what currently going on. No need to open 10 000 ticket, if I see that blog and iweb.com are also giving packet the same time my server are… Waste of tech resource to look at them. I submitted an issue via the chat Tuesday at 9h am , and I got a reply at about 10h pm.

    I got bit more worried when I get alarm that my rack lost power for a minute today… I hope it was just a tech flipping the wrong switch.

    Even all that said, I will probably renew my service next February… iWeb is only a backup solution if my other 2 site fail. They do need to improve on stability (including keeping power to server), but I believe they can and will do it. And I also did have failure at my other site, had a failed Tripplite powerbar !!! (Yea, power switch is defective after being installed for 4 years and never being touched). Another time, a tech deleted all vlan on their main switch ! Sh!t happen… (ok that one should have never happen). What important, is how good you can inform your client, and what solution you can provide them. Sadly this time iweb failed at it.

    Note, I offer VoIP service to SMB, so I feel the pain for iweb customer who depend only on them for voip… Packet lost are not forgiving on voip… But you should not have trusted that 1 provider will never fail in the first place.

    And to conclude… Some people are thinking of Cloud right now… This is just a small reminder to tell them that the implementation of a cloud in today business term is nothing more than hosted service… So it mostly a marketing ployed to get your money. See amazon cloud service crashing it in entirely this year, and issue lasted a few days for some. And if iweb had a cloud service, it would also be fully affected. A real cloud if something that is actually shared between multiple provider. So in theory, it would be a broker that give you a portal that let you put server easily on different colo like iweb and other on the planet… So in an event like this, you could just select another provider to start an instance and your dns would get updated to route to that host. Then you would be on the cloud… But today cloud is just a synonym for hosted service by a single provider.

  83. Stéphane Jose
    November 3rd, 2011 9:44 am

    @Marc O. Chouinard: Thanks for this testimony and we appreciate your fidelity. The garantees (http://iweb.com/about-us/legal/) are quite clear. We do not assume that there will never ever be any problems. No responsible technology provider would ever claim this. The power and uptimes guaranties stipulates that “iWeb guarantees 100% network uptime for all subscribers of a solution which includes a network connection. During one month, a complete network outages shall never occur. If a network outage should happen, iWeb will compensate its clients with a credit [...]” (complete text available here: http://iweb.com/about-us/legal/uptime/).

    You mention the Cloud as an alternative. As you certainly know, any hosting solution run on an interconnected network and whether you use a “dedicated” or a Cloud infrastructure no one is unfortunately completely safe from network problems, see what happened to Amazon (http://venturebeat.com/2011/04/23/amazons-outage-in-third-day-debate-over-cloud-computings-future-begins/), or Blackeberry (http://crave.cnet.co.uk/mobiles/bbm-down-in-blackberry-outage-50005561/) recently.

    As per communications, at the moment I can only redirect you to iWeb’s last official statement that documents the DDoS attack that we are dealing with: http://blog.iweb.com/en/2011/11/incident-network-infrastructure-ddos-update/8916.html

    We definitely know that this situation is very unpleasant to our affected clients and we appreciate all the constructive comments we receive.

    As soon as new information is available it will be published on that page.

About the iWeb blog

The iWeb Blog covers web hosting, web development, web design, Internet marketing and just about every announcement concerning iWeb.

System Status

All Systems Running without Issues.
Details