[gnso-ff-pdp-may08] How are Internet users affected by fastflux hosting?

[Joe St Sauver <joe@xxxxxxxxxxxxxxxxxx>]

Dave did an excellent job of handling this question already as 
part of his writeup, but since I promised to send along some text
for this topic during today's call, here's a first cut at explicitly 
answering the question, 


"How are Internet users affected by fastflux hosting?"
------------------------------------------------------

While most Internet users have never heard of fastflux hosting,
a growing number of them are nonetheless directly affected by it. 

Internet users provide both the raw material that fastflux hosting 
runs on (malware-compromised broadband-connected consumer PCs), 
while also serving as the target audience for the spamvertised web 
sites which fastflux enables. 

Internet users are thus central to the entire fastflux problem, and
unless it is handled appropriately, they are also the ones who will
be subject to yet more breakage and loss of Internet transparency.


To understand how consumer PCs came to be converted into fastflux nodes,
we need to step back for a moment and consider the related problems
of malware and spam. 

Internet miscreants use malware -- viruses, worms, trojan horses, 
etc. -- to efficiently gain control over large numbers of vulnerable 
networked consumer PCs. Those compromised systems, subject to remote
manipulation by shadowy masters, are commonly known as "bots" or 
"zombies."

Having obtained control over those compromised PCs, the miscreants
can than use those bots as a base from which to search for additional 
vulnerable systems, as a platform for sniffing network traffic, as a 
source of network attack ("DDoS") traffic, or most commonly, to 
deliver spam directly to remote mail servers (so-called "direct-to-MX 
spamming").

The Messaging Anti-Abuse Working Group, a consortium of leading 
international ISPs, has issued recommendations for managing port 25
traffic to defeat direct-to-MX spamming, see http://www.maawg.org/port25
If traffic on port 25 is blocked through following those recommendations,
as it now is at many ISPs worldwide, spam can no longer be sent directly 
to remote mail servers from those compromised PCs (although non-spamming 
normal mail users can still send regular mail). 

When the ISPs control port 25, that leaves the shadowy "bot herders" with 
millions of compromised systems which are now incapable of directly 
spamming remote mail servers.

At the same time, spammers (and other miscreants) find themselves 
confronting a second orthogonal problem: it has become hard if not 
impossible for them to obtain and retain mainstream web hosting for 
illegal content.

While what's illegal will vary from jurisdiction to jurisdiction, 
there are some categories of content which are illegal virtually 
everywhere, including, among other things:

-- narcotics, anabolic steroids and other dangerous drugs distributed
   without a valid prescription

-- child pornography

-- viruses, trojan horses and other malware

-- stolen credit card information

-- phishing web sites

-- pirated intellectual property, including pirated software ("warez"),
   copyrighted music and movies, and trademarked consumer goods (most
   notably things such as premium watches, shoes, handbags, etc.)

In fact, many hosting companies specifically exclude hosting of any 
product or service (whether legal or not) which has been "spamvertised" 
(advertised via spam), because they recognize that to permit spamvertised 
products or services on their hosting service will commonly result in 
their address space getting listed on one or more anti-spam DNS block 
lists, such as those operated by Spamhaus [http://www.spamhaus.org/]. 

Listings on Spamhaus or similar lists result in significant complaints 
regular customers who may be incidentally impacted by such a listing. 

With that for background, you can now guess what happened next: spammers
repurposed some of their "surplus inventory" of compromised-but-unspamable
systems to provide "web hosting" for illegal or spamvertised content 
which they couldn't host elsewhere.

By this do we mean that spammers actually replicated all the hundreds or
thousands of html files, images, databases and other bits and pieces
of content and software making up a sophisticated web site on each of
dozens or hundreds of fastflux hosts? No, that would be too complex, too
time consuming, and too easily detected.

Instead, spammers found that they could simply use "reverse proxy" software 
to accept web connections on the compromised consumer host, tunnelling that 
traffic back to their actual (hidden) backend master host. nginx is one
product often used for that purpose, although it is also routinely used
by regular web sites as well.

The compromised consumer PC then acts as if it were delivering web pages, 
but in reality it is just acting as a pipeline to a hidden master web 
server (or farm of servers) located elsewhere.

[insert suitable illustration here]

Naturally, you might wonder, "Does the owner of the compromised PC
know that all this is going on via his or her computer and network 
connection?"

"No." 

No one asks the owner of the compromised PC, "Do you have any objection
if we use your computer to distribute stolen credit card numbers?" 

No warning light goes off on the compromised PC saying hey, "Someone's
serving stolen software from your system!"

Typically the owner of the PC *only* becomes aware that they have 
unwittingly become a participant in illegal online activity when:

-- antivirus software, or other security software, eventually detects 
   the presence of malicious software on the system

-- someone complains to their ISP, and their ISP contacts the customer
   with the bad news that they're infected

-- the ISP disconnects the customer, blocks traffic to/from them, or
   plops the customer into a quarantine zone where all they have 
   access to are clean up-related sites and tools

-- the user finds their system has become slow or unstable, and 
   takes steps to figure out why, 

-- the user find that they can no longer access some remote network 
   resources because they've been blocked at those remote sites as a
   result of their infection, or

-- the user is visited by law enforcement officials investigating
   the illegal activity that has been seen in conjunction with "the
   user's" connection.

The user is then left with the unenviable chore of trying to get their
compromised system cleaned up. Because of the complexity of cleaning many
infections, and the substantial possibility that at least some lingering
badness may be missed during efforts at cleanup, most experts recommend 
formatting compromised systems and reinstalling it from scratch, however 
that can be a time consuming and laborious process, and one that may be 
practically impossible if the user lacks trustworthy backups or cannot 
find original media for some of the products they had been using.

What a mess. That mess is the first impact of fastflux hosting, but one
which only some unlucky users experience. 


The next effect of fastflux hosting is one which virtually all Internet
users experience, and that's spam. Remember, fastflux hosting exists to
host illegal content or spamvertised products or services. All of us
receive spam, whether that's an occaisional message that slips through
otherwise efficient filters, or a steady deluge that may have caused 
some of us to abandon email altogether.

Without the ability to obtain reliable web hosting services, spammers
are left with only a few categories of potential spam, such as stock
pump-and-dump spam, where users don't need to visit the spammer's web
site to purchase a product or service. Clearly spammers are powerfully
motivated to find an alternative, and that's what fastflux has given
them. With fast flux, they've got it. 

With fastflux, if one compromised machnie is discovered and taken off
line, another system will be ready to take over. It thus becomes very
difficult to "completely take down" the spammer's "web hosting" unless 
you can:

-- identify and take down the back-end hidden master web server

-- take down the domain name that's being spamvertising, or 

-- take down the name servers that the spamvertised domain relies on.

Spammers quickly recognized that the name servers were a weak point in 
their scheme, so they adapted. How did they adapt?

Well, they began not just using compromised systems for web hosting, 
but they also began to use those systems to do DNS for their domains.  
A domain that does both its web hosting and which gets its DNS service
via compromised systems is normally referred to as a "double fastflux"
or "doubleflux" domain. 

All of this malicious activity, taking place on systems that are not
professionally administered, resulted in ISPs endeavoring to control
these phenomena via the network. It is understandable why they were
inclined to do so: blocking port 25 controlled the spewage of spam, 
even if it did nothing to fix the underlying condition, so maybe
something similar could be done to address fastflux and doubleflux
abuse. 

Unfortunately, unlike email where controlling port 25 is sufficient
to control the emission of spam, when it comes to fastflux web pages,
web pages can be served on *any* arbitrary port (e.g., to access a web
page on port 8088 instead of the default port 80, one might use a URL
such http://www.example.com:8088/sample.html ). 

Blocking http traffic from consumer web pages thus often results in 
ISPs deploying more draconian solutions, such as banning all web servers 
from dynamic customer address space, or deploying potentially expensive
deep packet inspection (DPI) appliances to identify fastflux or double
flux traffic (at least until the spammers begin using SSL/TLS to defeat
DPI.

The problem gets even more complex when double flux is involved. When
name servers are routinely hosted on consumer systems, controlling 
that DNS traffic requires managing port 53 traffic, blocking external
DNS queries coming in to the name server running on the compromised 
customer host, and typically also managing blocking or redirecting any 
DNS traffic coming from the local customer base, permitting it only to
access the provider's own DNS recursive resolvers. This loss of Internet
transparency can keep customers from readily (and intentionally!) using
third party DNS servers (such as those offered to the Internet 
community by OpenDNS), and may also complicate or preclude things such 
as accessing access-limited information products delivered via DNS, such 
as some subscription DNS block lists. 


In conclusion, Internet users see their systems used without their
permission by abusers who've set up fastflux nodes on them; they face
the daunting task of cleaning up those compromised systems once they
discover what's happened; they are the target of endless spam, spam that
would be materially harder if fastflux hosting didn't exist; and they
experience a loss of Internet transparency as ISPs strugle to control
the fastflux and doubleflux problems on the network. The combination
of those effects can result in Internet users having a pretty bad
experience, all thanks to the choice by some to use fastflux and double 
flux techniques.