Apache

How to Enable Dynamic robots.txt with Virtual Hosts

The following configuration can enable the rewrite rules of your virtual hosts and make robots.txt point to any other dynamic file (e.g. robots.php or robots.asp) which can change content dynamically.

Add rewrite rules into <Directory> within the Virtual host.
[root@test ~]# vi /etc/httpd/conf/httpd.conf
...
<VirtualHost *:80>
    DocumentRoot "/var/www/html/example"
    ServerName www.example.com
    ...
    <Directory "/var/www/html/example">
        AllowOverride All
        RewriteEngine On
        RewriteRule ^robots.txt$ robots.php [NC,L]

    </Directory>
    ...
</VirtualHost>

Validate the syntax of httpd.conf.
[root@primary01 ~]# apachectl -t
Syntax OK

Restart httpd service.
[root@primary01 ~]# service httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]

Now you can verify your robots.txt. For example, access http://www.example.com/robot.txt

Leave a Reply

Your email address will not be published. Required fields are marked *