Main menu

I'm the System Administrator, I Defeat HULK!

Http Unbearable Load King (HULK) is a web server denial of service tool designed to generate volumes of unique and obfuscated traffic at a webserver, bypassing caching engines and therefore hitting the server's direct resource pool. It is written in Python by Barry Shteiman.

HULK, released May 17, uses randomized header and parameter values to generate a threaded GET flood attack; the randomized requests make it more difficult to distinguish attack threads from legitimate traffic, particularly for automated mitigation solutions. HULK takes advantage of out-of-the-box web server configuration vulnerabilities and spawns 500 threads that collectively stream random GET requests at its website target upon launch, bypassing caching engines to exhaust server resources. - PROLEXIC

 

The Attack

Last May 11 2013, before the national election day (May 12) in The Philippines. One of the Drupal site I'm handling suddenly throws 503 or/and 502 errors consistenly. I SSH'ed to the webserver a checked if the FastCGI is still up and responding. Yeah, spawn-fcgi is still up. The webservers CPU and RAM load also looks good. Checked the Nginx logs and something unusual is flooding the access log. There are lot of weird GET request that's not even legitimate pages on the site.

125.134.224.98 - - [11/May/2013:22:19:52 +0800] "GET //?3294ddae=19758401392027736 HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31"
1.162.142.170 - - [11/May/2013:22:19:52 +0800] "GET //?469cc01a934f5=2758293213731211 HTTP/1.1" 301 178 "-" "Opera/9.80 (Windows NT 6.1; WOW64) Presto/2.12.388 Version/12.14"
111.242.23.9 - - [11/May/2013:22:19:52 +0800] "GET //?14eb0ce4=08171158377081156 HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.7 (KHTML, like Gecko) RockMelt/0.16.91.483 Chrome/16.0.912.77 Safari/535.7"
61.31.89.140 - - [11/May/2013:22:19:52 +0800] "GET //?d1e8a6d67cc21=8199562333727572 HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 5.1; rv:20.0) Gecko/20100101 Firefox/20.0"
220.134.244.117 - - [11/May/2013:22:19:53 +0800] "GET //?39215f21=2231654601637274 HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31"
114.38.154.213 - - [11/May/2013:22:19:53 +0800] "GET //?bbbdc208=7333642262965441 HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.57 Safari/537.17"
1.162.142.170 - - [11/May/2013:22:19:53 +0800] "GET //?55f58b3=3357779495418072 HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11"
115.165.220.74 - - [11/May/2013:22:19:53 +0800] "GET //?e2a3031c30039=8852998679940482 HTTP/1.1" 301 178 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"
61.31.89.140 - - [11/May/2013:22:19:53 +0800] "GET //?641998427a258=3910155451104824 HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 5.1; rv:20.0) Gecko/20100101 Firefox/20.0"
111.242.23.9 - - [11/May/2013:22:19:53 +0800] "GET //?38c7389a=22178987273946404 HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.7 (KHTML, like Gecko) RockMelt/0.16.91.483 Chrome/16.0.912.77 Safari/535.7"
218.166.47.84 - - [11/May/2013:22:19:53 +0800] "GET //?aa9d4929=6664624905679375 HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31"
1.162.142.170 - - [11/May/2013:22:19:53 +0800] "GET //?7e95a61559d5f8=49447095891249393 HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0"

IPs are coming from Taiwan. I immediately know that the site is being DDoS attacked. Read the Philippines-Taiwan conflict here.

The GET request was hitting the database very hard. All of the CPU cores is running at 800% and it never drops down.

 

The Defense

The first thing I can think of is to automatically blocked those IPs. I installed fail2ban and implement it in 5 minutes. It starts blocking IPs but it seems that it didn't make any improvement on the site accessibility. I waited for about 30 minutes and it already blocked more than 2k IPs but the site is still unstable. I decided to put the site into maintenance mode and examine the pattern of the attack. I'm thinking to redirect the GET pattern with if and rewrite rule. After an hour of trial and error, I ended up using the below rewrite rule.

if ($args ~* "(.{1,})=(.{1,})" ){
        rewrite ^/$ /444_rewrite?;
}
location  /444_rewrite {
        return 444;
}

What it does?

Since the site is using friendly URL and none of the site URL starts with ? and using =, I can redirect all those weird GET requests to 444. The argument (.{1,})=(.{1,}) tells Nginx to redirect all GET requests that have any characters with = (equals) between them.

 

Conclusion

The effect of the rewrite rule is effective. The site immediately went back online and the CPU load of the database became normal. Although the site is still under DDoS attack for a few days, it doesn't have any impact on the site after implementing the rewrite rule.

The pattern was not exactly the same as the GET request coming from HULK, I think the DDoS tool was somewhat a rewritten kind of HULK.

 

Have you experienced the same type of DDoS attack? How did you mitigate the attack? Let me know on the comment section below!

FacebookG+TwitterRSS