To understand what the robots.txt file does, it’s important to take a quick step backward. The internet contains many robots. These are programs that scour the internet. They are also called spiders, crawlers, etc. Basically, they go out and try to find what’s out there. In the case of someone like Google, they are trying to see what content is on the web so that it can index it. It can then present that information to people searching the internet in a organized fashion. I hope that makes sense.

Well, the robots.txt file tells participating robots what to and what not to index. So, you can tell a robot that you don’t want it to see anything. The problem with this in terms of completely hiding parts of your site is that the robots.txt file is easy to find. Anyone can read it. But it’s a good way to keep certain parts of your site out of search engines. And that is part of the battle to securing site. Now, let’s look at a sample robots.txt file:

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /tmp/
Disallow: /private/

This robots.txt file tells robots to stay out of the cgi-bin, images, tmp, and private directories.  There are many thing you can do with a robots.txt file, including specifying where a sitemap is located.  Hmmm, maybe we should cover that topic sometime in the near future.  So, the robots.txt file is a simple text file that tells internet spiders where they can and cannot go.  It’s that simple.  Well, unless you get more involved.

You may be wondering why in the world you would pay the extra money to get a private server virtual? Well, a virtual dedicated server (just another way of saying it), affords you more power and control over any shared hosting package. Just keep in mind, however, that those advantages are coupled with complexity.

Shared hosting packages are basic, low cost hosting packages. You typically get the most bang for the buck with these types of packages. But that ease of use and low cost is offset by slow response and inflexibility. If you are needing more control and better response times, then our virtual dedicated server packages could be exactly what you want. These packages will give you SSH access to your server. This way you can alter anything about your server box. You can install any software that you want. You can reprovision (wipe your server clean) and start all over. You can tweak your apache, mysql, and php configurations. But, you can also royally mess your system up. And you also don’t have an easy way to install most packages. That’s for you to figure out. But, if you don’t mind tinkering and learning, you will love a virtual dedicated server. There really is no going back. We have many configuration packages for virtual dedicated servers for you to choose from. You can go the linux route, choosing between Fedora Core or CentOS. Or, you can go the Windows 2003 Server way. We have also included an Assisted Server Plan recently. The Assisted Server Plan is only available on the Plesk Control Panel. It includes the following:

Initial Server Setup:

• Including installation of operating system and Plesk control panel.
• Setup for the following services:
  • Domain Name & DNS
  • Hosting Account
  • SSL Certificate
  • Resource Allocation

Backup Services:

• Critical server information will be backed up daily and stored for 7 days
• Restores provided at your request via phone or email ticket.

Security Services:

• Security scanning is offered once a month per server, with review and report of security scans, including any recommended remediation actions.

Emergency Services:

• Direct access to advanced support for emergencies including:
  • Service restart or reboot
  • Server or service down
  • Tier-3 Support troubleshooting

Patching Services:

• Patching services** for the base operating system, such as Fedora Core 6 or Windows 2003, which includes:
  • Official updates released by the Fedora or Fedora Legacy Project.
  • Official updates, bug fixes and security patches released by Microsoft®.

Monitoring Services:

• Per server monitoring of basic services such as Ping and HTTP.

Custom Support Access:

• Access to our flat-rate custom support services

This isn’t a bad option for folks who want that power and flexibility but don’t want some of the hassle that comes with that power.