Nginx: UserAgent based transfer rate limit

In this post we’ll show an example on how to implement transfer rate limitation in Nginx. As this limitation is User agent based, it can be very meaningful for high transfer volume websites like download site,  a mirror site like mirror.centos.org, etc.

The configuration itself is easy to understand:

server {
        listen   80;
        server_name  test.admon.org;
        access_log  /var/log/nginx/mirror.access.log;
        location / {
                root   /data/mirrors;
                autoindex on;
                index  index.html index.htm;
                if ($http_user_agent ~ "MSIE") {
                        limit_rate 2k;
                }
                if ($http_user_agent ~ "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)") {
                        return 404;
                }
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
                root   /var/www/nginx-default;
        }
}

From the above line, we can see that, If visitors’ User-Agent strings contains MSIE, the transfer rate is limited to 2000 Bytes only, and the other agent Firefox is not allowed to download any file from this site.

Let’s go further with some tests:
For Internet Explorer:

$ wget --no-cache --user-agent="Something with MSIE; bla bla" http://test.admon.org/file.gz -O /dev/null
--14:07:59--  http://test.admon.org/file.gz
           => `/dev/null'
Resolving test.admon.org... 94.75.214.118
Connecting to test.admon.org|94.75.214.118|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8,051,367 (7.7M) [text/plain]

 0% [                                     ] 49,152         2.08K/s  ETA 1:02:38

For Normal User:

$ wget --no-cache http://test.admon.org/file.gz -O /dev/null
--14:08:28--  http://test.admon.org/file.gz
           => `/dev/null'
Resolving test.admon.org... 94.75.214.118
Connecting to test.admon.org|94.75.214.118|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8,051,367 (7.7M) [text/plain]

100%[====================================>] 8,051,367      6.34M/s             

14:08:29 (6.33 MB/s) - `/dev/null' saved [8051367/8051367]

For a Bad User:

$ wget --no-cache --user-agent="Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)" http://test.admon.org/file.gz -O /dev/null
--14:09:22--  http://test.admon.org/file.gz
           => `/dev/null'
Resolving test.admon.org... 94.75.214.118
Connecting to test.admon.org|94.75.214.118|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
11:35:01 ERROR 404: Not Found.

So you can use it for some purposes, like:

  • Implement limitations for your visitors to get rid of some bad activities.
  • Limit SE’s crawling rate. (You can define crawl rate in some SEs like google, but not all of the Search Engines support this feathure.

Dont forget that this is just a simple way to limit the transfer rate, things in real world is that the User-Agent for a web client can be easiliy modified.
The Chinese version for this post is available here, and if you have any issues, welcome to our webmaster forums, thanks!

Share Button

Leave a comment

Your email address will not be published. Required fields are marked *