Jan 14
Once folks start administering their own web server, they immediately understand the enormity of the task. There are so many new things to learn. The task can seem daunting. You start encountering things that you didn’t know about. One such thing is robots.txt. What is this file and what does it do?
private directories
robot robots txt spiders 
