From DevRandom

High Level Overview

What Is a Denial of Service Attack?

The primary goal of a DoS attack is to deny legitimate users access to a service. Unlike most other attacks which are intrusive in nature, a DoS does not require the attacker to gain access to the server and is commonly performed remotely. Attackers achieve their DoS objective by flooding the target until it crashes, becomes unreachable from the outside world, or its ability to server legitimate traffic is severely impaired. The actual volume of the attack traffic involved depends on the type of attack. DoS attacks can vary in impact from consuming the bandwidth of an entire network, to preventing service use of a single targeted host

What Is a Distributed Denial of Service Attack?

A Distributed Denial of Service (DDoS) attack is a DoS attack that is coordinated(distributed) across many systems all controlled by a single attacker. Prior to the attack, the master may have compromised a large number of hosts. These compromised hosts, called zombies, are then used to perform the actual attack. Typically, the master is not involved in the attack; it just coordinates it. DDoS attacks are harder to prevent as they can be made to look like real legitimate requests - and/or can be made from enough compromised hosts to make effective manual blocking impossible.

Types of Dos

If you get into the technical details, there are several types of DoS and DDoS attacks. See http://en.wikipedia.org/wiki/Denial-of-service_attack if you want to read further. Broadly speaking we have :-

Network Flooding attacks

In this case the attacker floods the target with network traffic, eg. ICMP or TCP SYN packets. The TCP SYN flood attack is a protocol violation attack that is used in several variations. In the simplest case, an attacker sends the first packet (with the SYN bit set) of the well known TCP three way handshake. The victim responds with the second packet back to the source address with SYN-ACK bit set. The attacker never responds to the reply packet, either on purpose or because the source address of the packet is forged. A variation to this attack uses public servers as a reflective media to flood the victim with TCP SYN ACK packets. In this case, the attacker spoofs the source address of the TCP SYN packet with the victim’s address. The packet is sent to a public server that provides a public TCP service (such as HTTP). The server sends a TCP SYN ACK packet to the victim’s host. The victim, having not sent the original packet either ignores the packets or sends a TCP RST packet.

Attack to dependant services

Sometimes attacking critical services like DNS or authentication servers (active directory) or other such services will cripple the actual service. Such attacks need a better knowledge of the actual application and architecture.

Attack to an application

These are more directed attacks to a specific application like web server or ftp server. In this case, the attacker need not flood the network (which usually has a higher flooding threshold), but they can target a vulnerable part of the service. For example: If a particular section/URL of a website is database intensive, an attacker can hammer that URL with several requests, thus impacting the underlying database and in turn the whole website. Exploiting flaws or bugs in the application software is another example.

Dos & the law

Denial-of-service attacks are violations of the laws in several nations. They almost always violate the acceptable use policies of virtually all Internet service providers.

In UK, the 2006 computer misuse act specifically makes this illegal. (http://www.out-law.com/page-7462)

In US, there are similar laws and the FBI or the secret service can be asked to investigate if needed.

However, in practice bringing legal action following a DoS attack is often next to impossible. Not only is finding the original perpetrator(s) difficult, many times they are operating from countries without any strict laws governing cybercrime. International law is also unclear in this matter - if a DoS attack is launched from country A; involves compromised zombies in countries X, Y, Z; and targets a site in country B, there is no clear precedent for which laws a perpetrator should be prosecuted under.

DoS and Linux

Detection

Counting connections

At the basic level you can start by counting connections

This example does tcp only but can be adapted to UDP and other protocols

netstat -tn | awk '{print $5}' | cut -d: -f1 | sort | uniq -c |sort -nr | head -10

-t is for tcp; -a for all connections -n for numeric output and -p for prog

GO a step further and look for SYN_RECV. If you see lots of connections in this state, & if they stay for a long time, then something is not right.

watch 'netstat -n -t |grep SYN_RECV'

Iptables

Iptables can be used to help log information.

Eg. Log over 100 connections in 10 seconds.

iptables -A INPUT -j LOG --log-level 4

iptables -I INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --set

iptables -I INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --update --seconds 10 --hitcount 100 -j LOG --log-prefix 'DOS IP:' --log-level 4

Log messages

Look in dmesg and /var/log/messages for any obvious issues/errors. Some worthy mentions are

1. "TCP: Treason uncloaked! Peer ..." This is not of concern and is a common occurence on Linux kernels. This happens because of TCP window scaling issues. This message indicate that the window size unexpectedly changed during TCP communication and is normally caused by some intermediate router preventing window scaling. So while this looks disconcerting, it is only an informative message.

2. "TCP: drop open request from ..." This is a little more concerning. This usually means that the tcp_max_syn_backlog buffer was full. The default value for this buffer is 1024. The value determines the maximal number of remembered connection requests, which still did not receive an acknowledgment from a connecting client. This message is accompanied by a corresponding IP address and that specific IP is main culprit. This could mean a DoS but it can also happen with slow connections or in case of a web service, a centralized proxy that is not behaving right. tcp_max_syn_backlog can be raised to help here.

Prevention

SYN cookies

In normal situation, when a server receives a SYN packet it starts building the connection in its stack. A SYN flood takes advantage of this behaviour by sending enough SYN packets and overflows the buffer thus preventing any new connections.

When syn cookies are enabled, the behaviour of the TCP stack changes. With this enabled, the server never updates its buffer when it receives a SYN. It replies to the sender with a SYN + ACK but before it does it injects a cookie into the packet header. This cookie is a calculated initial sequence number (ISN). This is done using several connection attributes like source IP, source port, destination IP, destination port plus an internally generated secret value (Ref: http://cr.yp.to/syncookies.html ). When the server receives an ACK back from the client it verifies the cookie & now creates a connection in its buffers. This process eliminates the problem of the buffers getting filled with half open connections.

It is important to understand that SYN cookies help in a SYN flood attack or a SYN spoofing attack only. They also add additional burden on the servers' CPU which is used to calculate the cookie. While this can be left as default on a server, it is typically only enabled when the server is under a DoS attack.

Enabling this on 2.6 kernel is fairly straightforward

  echo 1 > /proc/sys/net/ipv4/tcp_syncookies

or

  sysctl -w net.ipv4.tcp_syncookies=1

To make this permanent add the following to /etc/sysctl.conf

net.ipv4.tcp_syncookies = 1

TCP parameter tuning

There few other kernel variables that can help in a SYN style attack.

net.ipv4.tcp_max_syn_backlog variable is the maximal number of remembered connection requests, which still did not receive an ACK from connecting client. Default value on a 2.6 kernel is 1024. Note, this applies to all connections , IP addresses and ports and is a global value This will result in extra memory being used for the connection buffers. This won't solve the problem as a.) the attacker can just increase the number of packets he is sending and b.) the server still needs to handle all these connections.

The change is again a kernel value set by sysctl

  sysctl -w net.ipv4.tcp_max_syn_backlog=4096

net.ipv4.tcp_syn_retries & net.ipv4.tcp_synack_retries determine the number of SYN or SYN+ACK packets sent before the kernel gives up on the connection.v operating system. The default on a 2.6 kernel is 5. Lowering it will help clear up half open connections faster but can also lead to some legitimate connections being dropped.

net.ipv4.ipfrag_time determines the number of seconds, an IP fragment will stay in memory

IP tables rate limiting

This will block an IP if there are more than 10 hits in a 60 second period. Be careful with these as they might block legitimate traffic. You can play it safe by first logging it (see above):

iptables -I INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent  --set 
iptables -I INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --update --seconds 60 --hitcount 10 -j DROP

Apache

Detection

Review connections to apache

You can use Apache's status if it is available. The same can be done using netstat and counting connections to port 80/443.

curl http://localhost/server-status?auto

Review access logs

Show top 10 IPs:

cat access_log |  awk '{print $1}' | sort -n | uniq -c | sort -rn | head -10

Count hits to specific URL

cat access_log | awk -F'[ "]+' '$7 == "/index.php" { ip[$1]++ } END { for (i in ip) 
{printf "%d - %s\n",  ip[i],i } }'

Count hits by a specific IP:

cat access_log | awk -F'[ "]+' '$1 == "89.234.23.50" { pcnt[$7]++ } END { for (i in pcnt) 
{ printf "%d - %s \n", pcnt[i],i } }'

You can sort any of these by piping output to sort:

cat access_log | awk -F'[ "]+' '$7 == "/index.php" { ip[$1]++ }     END { for (i in ip) 
{ printf "%d - %s\n",  ip[i],i } }'  | sort -nr | head -10

NOTE: The above assumes a log format like this but you can adapt as needed:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %T/%D" combined

Prevention

Most of the Apache tuning techniques will help. Apache has some modules that help. mod_evasive is one of them that is available in EPEL. Apache mod_security is probably the most stable and versatile one. Note: These are not supported by Rackspace and it is a reasonable endeavour.

Note also that in most cases, Apache based limiting will not be ideal and can usually be achieved by iptables at a lower level. In any case, Apache modules will not be able to handle a reasonably large DoS.