I'd like to apologize to those who have been negatively impacted by my decision to pull support for Pastie (especially Josh). To understand why I made the decision to pull our support after 9 hours of multiple DDOS attacks, I'd like to share some background and our ops philosophy.
It is important to understand that I put our existing customers that pay us to manage and scale their high growth revenue-generating web applications before all else. This is the core of our business and what they trust us to do. As we are seeing now, this means that I will protect them at the expense of making some non-customers and "risky" customers upset. Let me explain further...
Rails Machine at this time is 6 people. Through a lot of tools like Moonshine, experience, and process we manage 100+ web applications. Please note that I did not say "host". Hosting is only part of the package. We commit to do whatever it takes to keep our customers' applications available and growing their business.
Everyone in the organization is a developer on a varying scale of dev to ops including myself who started with Rails in 2005 and have been a professional dev for 20 years.
Although not as quiet as we would like, in general the workload of responding to outages, bugs, scaling problems, and traffic bursts are managed by the team. We've been doing this for 6 years focused specifically on Rails and have seen most problems with Rails applications in production. This makes us fairly efficient in identifying and resolving issues.
We've been hosting Pastie pretty much since the beginning and free of charge for several years. In the past two years, Pastie began to attract a lot of users intending to use it to do illegal things. This includes sharing stolen credit cards, stolen passwords, phishing schemes, copyrighted content, virus/trojan horses, hacker scripts, confidential corporate info, etc, etc.
Please know that the overwhelming majority of Pastie's user base are well meaning folks who kindly follow Josh's basic rule of "using Pastie for good". A tiny minority however attract a lot of attention through their public pastes that ruin the experience for everyone else.
Aside from the obvious problems, the public existence of this stuff makes a lot of people upset who in turn threaten us. This includes but not limited to criminals, giant corporations, angry individuals, trolls, and more importantly our data center/upstream provider. These upset people then send us nastygrams requiring us to take action or else. "Or else" includes suing us, arresting us, DDoSing us, and more importantly terminating our service.
To avoid bad things happening to us and all of our other customers, we have to take action immediately. Every now then other customers get a spam notice or a DMCA notice but in general it happens once and not a huge deal. Pastie on the other hand generates 100s of abuse complaints. Abuse complaints that we can not ignore and require us to investigate and follow up on. While we are doing this, we are not helping the 100+ other customers that generate zero abuse complaints and most likely never will.
Enter the DDos. Over the past 6 years, we've handled multiple DDoS attacks on different applications. Given that 95% of our customers are running revenue-generating business applications, we deal with a DDoS about once or maybe twice per year. They are really annoying and consume a lot of time and a lot of concurrent team members. Although we wish they never happened, as some have pointed out DDoS attacks are a fact of life. Many high profile sites with much larger teams and budgets have struggled for multiple days fighting waves of attacks. We accept this as part of our job.
The problem in this particular instance is that these DDoS attacks on Pastie were a continuation of the stream of operational disruptions already being generated by the site. After handling the first attack with four of us covering all of the angles around 10pm and with the help of Internap's network team, we halted the attack.
Within a few hours, it began again in the wee hours of the morning. At the same time, alerts for another customer who had entrusted us with their business came in. So a decision was made to halt the second attack as quickly as possible and focus on doing our job as we promised to the rest of our customers. We could have chosen to ride out multiple other attacks and engage in a lot of time consuming and expensive behavior to preserve a site that was already a source of ops disruption. Making that choice would have been inconsistent with our values and commitments.
This decision was purely ops motivated to protect our team members ability to serve our core customers.
Some of you are upset about this decision and I am sorry for that. I know 100+ customers that would approve. I put my customers and team first.
Please feel free to reach out to me directly (email or twitter) if you would like to discuss this further.
Keep on keeping on. You made the right choice and in the long run dealing with the take down notices and legal wrangling would have been a full time job.
What most people probably wont understand is that no sane business is going to go to bat for a non-customer who is costing them signifigant time and money as well as putting their whole business at risk.
I think this was a hard decision and I really believe that you're trying to protect your business, but I also believe this is the wrong decision.
- Not being able to handle 100s abusive requests is an indication that you can't handle rapid growth. You're going to (want to) host other websites that can grow big and will catch a lot of wind. Although there is no direct revenue for you Pastie.Org was good practice for this it seems also this wasn't a sudden problem (you yourself indicate here that the problem was slowly growing) so this should be no factor in the decision to terminate the hosting so suddenly. Of course if it's more hassle than it's worth you and Pastie can come to an agreement (like stopping next month) but it now sounds like that decision should've been made months earlier or not yet at all.
- Although it's extremely hard to protect yourself from DDOS attacks you've now openly indicated that you're vulnerable and that you will drop things you are committed to do if there is some pressure in order to save the rest of your customers. This will make other customers nervous since they can now be easily threatened and might even go down for the fun of it (I personally don't think it's fun but some people apparently do)
- Even though pastie.org was not generating any revenue and was not a 'customer' you still had a commitment to them and tbh it looks pretty bad that you've one-sidedly decided in an instant to break that commitment, this is really costing you cred.
Of course what's done is done now, but I just wanted to voice my opinion in this debate. Say I'm a devil's advocate here because by the positive comments here mostly you guys seem to be doing mostly good!
You made the absolute right decision, and anybody that is saying otherwise hasn't been in your position. The good of the many over the good of the one; that's hosting.
The operational awareness in this comment makes me wish everybody would print it out as an example of a tough call to make in defense of your brand.
You ignore the fact the this customer wasn't paying for the hosting. He only costs them money, a lot of time (through dealing with DMCAs, DDoSs and more).
At this point they had to make a choice and they chose for their paying customers and drop pastie.org
This was obviously a difficult decision for Rails Machine. I put food on the table from an application we've hosted at Rails Machine for several years: they are simply a fantastic team to work with. I can't imagine using a different Rails hosting provider.
You should edit your post to remove the 6th paragraph. Not your place to be calling pastie a increasing malware/virii distribution site. In fact, Not only is it libelous, but its not even appropriate to share with us. There is a reason companies dont get to comment on every bad thing that gets pointed their way.
IMO, your response does more damage to you than trying to explain it away. Doesnt matter if the guy is not paying you or not. In fact, your privacy policy says you wont do this and you just did.
I am not sure whats worse when you get hosted with you:
Is it when they disconnect you for having a incoming attack, or the public post afterwards where they air your dirty laundry?
The pastes that we receive complaints about are publicly viewable and searchable by anyone. The increase in these kinds of pastes is also publicly observable.
If a YouTube employee said that some naughty YouTube users post copyrighted videos, it wouldn't be private information or libellous as such videos are publicly viewable.
I'll edit to clarify that the overwhelming number of Pastie users are "using it for good" as Josh politely requests on the site.
It is important to understand that I put our existing customers that pay us to manage and scale their high growth revenue-generating web applications before all else. This is the core of our business and what they trust us to do. As we are seeing now, this means that I will protect them at the expense of making some non-customers and "risky" customers upset. Let me explain further...
Rails Machine at this time is 6 people. Through a lot of tools like Moonshine, experience, and process we manage 100+ web applications. Please note that I did not say "host". Hosting is only part of the package. We commit to do whatever it takes to keep our customers' applications available and growing their business.
Everyone in the organization is a developer on a varying scale of dev to ops including myself who started with Rails in 2005 and have been a professional dev for 20 years.
Although not as quiet as we would like, in general the workload of responding to outages, bugs, scaling problems, and traffic bursts are managed by the team. We've been doing this for 6 years focused specifically on Rails and have seen most problems with Rails applications in production. This makes us fairly efficient in identifying and resolving issues.
We've been hosting Pastie pretty much since the beginning and free of charge for several years. In the past two years, Pastie began to attract a lot of users intending to use it to do illegal things. This includes sharing stolen credit cards, stolen passwords, phishing schemes, copyrighted content, virus/trojan horses, hacker scripts, confidential corporate info, etc, etc.
Please know that the overwhelming majority of Pastie's user base are well meaning folks who kindly follow Josh's basic rule of "using Pastie for good". A tiny minority however attract a lot of attention through their public pastes that ruin the experience for everyone else.
Aside from the obvious problems, the public existence of this stuff makes a lot of people upset who in turn threaten us. This includes but not limited to criminals, giant corporations, angry individuals, trolls, and more importantly our data center/upstream provider. These upset people then send us nastygrams requiring us to take action or else. "Or else" includes suing us, arresting us, DDoSing us, and more importantly terminating our service.
To avoid bad things happening to us and all of our other customers, we have to take action immediately. Every now then other customers get a spam notice or a DMCA notice but in general it happens once and not a huge deal. Pastie on the other hand generates 100s of abuse complaints. Abuse complaints that we can not ignore and require us to investigate and follow up on. While we are doing this, we are not helping the 100+ other customers that generate zero abuse complaints and most likely never will.
Enter the DDos. Over the past 6 years, we've handled multiple DDoS attacks on different applications. Given that 95% of our customers are running revenue-generating business applications, we deal with a DDoS about once or maybe twice per year. They are really annoying and consume a lot of time and a lot of concurrent team members. Although we wish they never happened, as some have pointed out DDoS attacks are a fact of life. Many high profile sites with much larger teams and budgets have struggled for multiple days fighting waves of attacks. We accept this as part of our job.
The problem in this particular instance is that these DDoS attacks on Pastie were a continuation of the stream of operational disruptions already being generated by the site. After handling the first attack with four of us covering all of the angles around 10pm and with the help of Internap's network team, we halted the attack.
Within a few hours, it began again in the wee hours of the morning. At the same time, alerts for another customer who had entrusted us with their business came in. So a decision was made to halt the second attack as quickly as possible and focus on doing our job as we promised to the rest of our customers. We could have chosen to ride out multiple other attacks and engage in a lot of time consuming and expensive behavior to preserve a site that was already a source of ops disruption. Making that choice would have been inconsistent with our values and commitments.
This decision was purely ops motivated to protect our team members ability to serve our core customers.
Some of you are upset about this decision and I am sorry for that. I know 100+ customers that would approve. I put my customers and team first.
Please feel free to reach out to me directly (email or twitter) if you would like to discuss this further.
@bradleyktaylor, Founder, Rails Machine