[gnso-ff-pdp-may08] Chunked 19

[Joe St Sauver <joe@xxxxxxxxxxxxxxxxxx>]

When it comes to question 5.6, "How are Internet users affected by fast
flux hosting?" I propose answering that question with the following
(broken into discrete chunks for ease of discussion, with subsection
titles added)

19.1 -------------------------------------------------------------------

_Introduction_

While most Internet users have never heard of fastflux hosting,
a growing number of them are nonetheless directly affected by it. 

Internet users provide both the raw material that fastflux hosting 
runs on (malware-compromised broadband-connected consumer PCs), 
while also serving as the target audience for the spamvertised web 
sites which fastflux enables. 

Internet users are thus central to the entire fastflux problem, and
unless it is handled appropriately, they are also the ones who may
be subject to further restrictions and loss of Internet transparency.

19.2 ------------------------------------------------------------------

_Malware,_Spam,_and_Bots_

To understand how consumer PCs came to be converted into fastflux nodes,
we need to step back for a moment and consider the related problems
of malware and spam. 

Internet miscreants use malware -- viruses, worms, trojan horses, 
etc. -- to efficiently gain control over large numbers of vulnerable 
networked consumer PCs. Those compromised systems, subject to remote
manipulation by shadowy masters, are commonly known as "bots" or 
"zombies."

Having obtained control over those compromised PCs, the miscreants
can than use those bots as a base from which to search for additional 
vulnerable systems, as a platform for sniffing network traffic, as a 
source of network attack ("DDoS") traffic, or most commonly, to 
deliver spam directly to remote mail servers (so-called "direct-to-MX 
spamming").

19.3 ----------------------------------------------------------------------

_What_Are_Miscreants_to_Do_With_Compromised_Hosts_That_Can't_Be_Used_for_Spam?_

The Messaging Anti-Abuse Working Group, a consortium of leading 
international ISPs, has issued recommendations for managing port 25
traffic to defeat direct-to-MX spamming, see http://www.maawg.org/port25
If traffic on port 25 is blocked through following those recommendations,
as it now is at many ISPs worldwide, spam can no longer be sent directly 
to remote mail servers from those compromised PCs (although non-spamming 
normal mail users can still send regular mail). 

When the ISPs control port 25, that leaves the shadowy "bot herders" with 
millions of compromised systems which are now incapable of directly 
spamming remote mail servers.

19.4 -----------------------------------------------------------------------

_Spammers_and_Other_Internet_Miscreants_Have_a_Hard_Time_Getting_Web_Hosting_

At the same time, spammers (and other miscreants) find themselves 
confronting a second orthogonal problem: it has become hard if not 
impossible for them to obtain and retain mainstream web hosting for 
illegal content.

While what's illegal will vary from jurisdiction to jurisdiction, 
there are some categories of content which are illegal virtually 
everywhere, including, among other things:

-- narcotics, anabolic steroids and other dangerous drugs distributed
   without a valid prescription

-- child pornography

-- viruses, trojan horses and other malware

-- stolen credit card information

-- phishing web sites

-- pirated intellectual property, including pirated software ("warez"),
   copyrighted music and movies, and trademarked consumer goods (most
   notably things such as premium watches, shoes, handbags, etc.)

In fact, many hosting companies specifically exclude hosting of any 
product or service (whether legal or not) which has been "spamvertised" 
(advertised via spam), because they recognize that to permit spamvertised 
products or services on their hosting service will commonly result in 
their address space getting listed on one or more anti-spam DNS block 
lists, such as those operated by Spamhaus [http://www.spamhaus.org/]. 

19.5 ---------------------------------------------------------------------

_Miscreants_Discover_One_Thing_They_CAN_Do_With_Non-Spamable_Compromised_Hosts_

With that for background, it is easy to imagine what happened next: spammers
repurposed some of their "surplus inventory" of compromised-but-unspamable
systems to provide "web hosting" for illegal or spamvertised content 
which they couldn't host elsewhere.

19.6 ---------------------------------------------------------------------

_Reverse_Proxies_Are_Used_to_Actually_Deploy_Fast_Flux_Hosting_Networks_

Spammers actually replicated all the hundreds or thousands of html files, 
images, databases and other bits and pieces of content and software making 
up a sophisticated web site on each of dozens or hundreds of fastflux 
hosts. That would be too complex, too error prone, too time consuming, and 
too easily detected.

Instead, spammers found that they could use "reverse proxy" software 
to accept web connections on the compromised consumer host, tunnelling that 
traffic back to their actual (hidden) backend master host. "nginx" is one
product often used for that purpose, although it is also routinely used
by regular web sites as well.

The compromised consumer PC then acts as if it were delivering web pages, 
but in reality it is just acting as a pipeline to a hidden master web 
server (or farm of servers) located elsewhere.

[insert suitable illustration here showing reverse proxy setup here]

19.7 ---------------------------------------------------------------------

_Use_of_Botted_PCs_Is_Non-Consensual_and_Surreptitious_

The owner/user of a compromised PC doesn't know that his or her PC is being
used as part of a fast flux hosting network.

No one asks the owner of the compromised PC, "Do you have any objection
if we use your computer to distribute stolen credit card numbers?" and
no warning light goes off on the compromised PC saying "Hey, someone's
serving stolen software from your system!"

Typically the owner of the PC *only* becomes aware that they have 
unwittingly become a participant in illegal online activity when:

-- antivirus software, or other security software, eventually detects 
   the presence of malicious software on the system

-- someone complains to their ISP, and their ISP contacts the customer
   with the bad news that they're infected

-- the ISP disconnects the customer, blocks traffic to/from them, or
   plops the customer into a quarantine zone where all they have 
   access to are clean up-related sites and tools

-- the user finds their system has become slow or unstable, and 
   takes steps to figure out why, 

-- the user find that they can no longer access some remote network 
   resources because they've been blocked at those remote sites as a
   result of their infection, or

-- the user is visited by law enforcement officials investigating
   the illegal activity that has been seen in conjunction with "the
   user's" connection.

19.8 ------------------------------------------------------------------------

_Post_Fast_Flux_Infection_Cleanup_

Once the user discovers that they've been botted and used for fastflux
purposes, they are then left with the unenviable chore of trying to get their
compromised system disinfected.

Because of the complexity of cleaning many malware infections, and the 
substantial possibility that at least some lingering malware components
may be missed during efforts at cleanup, most experts recommend formatting 
compromised systems and reinstalling them from scratch, however 
that can be a time consuming and laborious process, and one that may be 
practically impossible if the user lacks trustworthy backups or cannot 
find original media for some of the products they had been using.

The need to deal with this mess is the first tangible user impact of 
fastflux hosting, but one which only some unlucky Internet users
experience. 

19.9 -------------------------------------------------------------------------

_One_Universal_Impact_of_Fast_Flux:_Spam_

The next effect of fastflux hosting is one which virtually all Internet
users experience, and that's spam. Remember, fastflux hosting exists to
host illegal content or spamvertised products or services. All of us
receive spam, whether that's an occaisional message that slips through
otherwise efficient filters, or a steady deluge that may have caused 
some of us to abandon email altogether.

Without the ability to obtain reliable web hosting services, spammers
are left with only a few categories of potential spam, such as stock
pump-and-dump spam, where users don't need to visit a spamvertised web
site to purchase a product or service. Clearly spammers are powerfully
motivated to find a takedown-resistent way to host their web sites, and 
that's what fastflux has given them.

With fastflux, if one compromised machnie is discovered and taken off
line, another system will be ready to take over. It thus becomes very
difficult to "completely take down" the spammer's "web hosting" unless 
you can:

-- identify and take down the back-end hidden master web server

-- take down the domain name that's being spamvertising, or 

-- take down the name servers that the spamvertised domain relies on.

19.10 -----------------------------------------------------------------------

_Fluxing_*Name_Servers*_As_Well_As_Web_Sites:_The_Rise_of_"Double_Flux"_

Spammers quickly recognized that the name servers were a weak point in 
their scheme, so they adapted by beginning to not just use compromised 
systems for web hosting, they also began to use those systems to do DNS 
for their domains.  

A domain that does both its web hosting and which gets its DNS service
via compromised systems is normally referred to as a "double fastflux"
or "doubleflux" domain. 

19.11 -----------------------------------------------------------------------

_Port_Blocks_Won't_Work_to_Curtail_Fast_Flux_Web_Hosting_

All of this malicious activity, taking place on systems that are not
professionally administered, resulted in ISPs endeavoring to control
these phenomena via the network. It is understandable why they were
inclined to do so: blocking port 25 controlled the spewage of spam, 
even if it did nothing to fix the underlying condition of the infected
host, so maybe something similar could be done to address fastflux and 
doubleflux abuse?

Unfortunately, unlike email where controlling port 25 is sufficient
to control the emission of spam, when it comes to fastflux web pages,
web pages can be served on *any* arbitrary port (e.g., to access a web
server running on port 8088 instead of the default port 80, one might 
use a URL such http://www.example.com:8088/sample.html ). 

19.12 -----------------------------------------------------------------------

_ISP_Efforts_to_Control_Fast_Flux_and_Double_Flux_Result_in_Collateral_Damage_

Blocking http traffic from consumer web pages thus often results in 
ISPs deploying more draconian solutions, such as banning all web servers 
from dynamic customer address space, or deploying potentially expensive
deep packet inspection (DPI) appliances to identify fastflux or double
flux traffic (at least until the spammers begin using SSL/TLS to defeat
DPI.

The problem gets even more complex when double flux is involved. When
name servers are routinely hosted on consumer systems, controlling 
that DNS traffic requires managing port 53 traffic, blocking external
DNS queries coming in to the name server running on the compromised 
customer host, and typically also managing blocking or redirecting any 
DNS traffic coming from the local customer base, permitting it only to
access the provider's own DNS recursive resolvers. 

This loss of Internet transparency can keep customers from readily (and 
intentionally) using third party DNS servers (such as those offered to 
the Internet community by OpenDNS), and may also complicate or preclude 
things such as accessing access-limited information products delivered 
via DNS, such as some subscription DNS block lists. 

19.13 -----------------------------------------------------------------------

_Conclusion_

In conclusion, Internet users see their systems used without their
permission by abusers who've set up fastflux nodes on them; they face
the daunting task of cleaning up those compromised systems once they
discover what's happened; they are the target of endless spam, spam that
would be materially harder if fastflux hosting didn't exist; and they
experience a loss of Internet transparency as ISPs strugle to control
the fastflux and doubleflux problems on the network. The combination
of those effects can result in Internet users having a pretty bad
experience, all thanks to the choice by some Internet miscreants to use 
fast flux and double flux techniques.