What is the Robot Text File?

By: Alan murray  

What is the Robot Text File?

The robot text file is used to disallow specific or all search engine spider’s access to folders or pages that you don't want indexed.

Why would you want to do this?

You may have created a personnel page for company employees that you don't want listed. Some webmasters use it to exclude their guest book pages so to avoid people spamming. There are many different reasons to use the robots text file.

How do I use it?

You need to upload it to the root of your web site or it will not work - if you don't have access to the root then you will need to use a Meta tag to disallow access. You need to include both the user agent and a file or folder to disallow.

What does it look like?

It's really nothing more than a "Notepad" type .txt file named "robots.txt"

The basic syntax is

User-agent: spiders name here
Disallow:/ filename here

If you use

User-agent: *

The * acts as a wildcard and disallows all spiders. You may want to use this to stop search engines listing unfinished pages.

To disallow an entire directory use

Disallow:/mydirectory/

To disallow an individual file use

Disallow:/file.htm

You have to use a separate line for each disallow. You cannot you for example use

Disallow:/file1.htm,file2.html

You should use

Use-agent/*
Disallow:/file1.htm
Disallow:/file2.htm

For a list of spider names visit http://www.robotstxt.org/wc/active/html/

Make sure you use the right syntax if you don't it will not work. You can check you syntax here http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

For help on creating robot text files there is a program call robogen.

There is a free version and an advanced version, which costs $12.99 http://www.rietta.com/robogen/

About The Author Alan Murray is a Certified Internet Webmaster Professional and Provides SEO Services and Website design. http://www.designprofessional.co.uk/SEO-Services.htm

This article was posted on November 01, 2005  

VoIP Phone - The Pros And Cons

Return to Index

UK Financial Services
Sub-categories: Accounting | Banks | Building Societies | Corporate Finance | Credit Services | Currency Services | Debt Collection | Directories | Factoring | Financial Consultants | Fund Management | Independent Financial Advisors | Information Services | Insurance | Leasing | Merchant Services | Offshore Services | Organisations | Personal Finance | Publications | Regulation | Socially Responsible Investment | Stockbrokers | Taxation
Article Index: 1 - 2 - 3 - 4 - 5 - 6 - 7 - 8

World Financial Services
Sub-categories: Associations | Automobile Buying and Financing | Banking Services | By Region | Cash Flow | College Financing | Commercial Lending | Computer Financing | Cooperatives | Credit and Collection | Directories | Education and Training | Employment | Field Service Companies | Financial Consultants | Financial Planning | Freelancer-Contractor Services | Holding Companies | Information Services | Insurance | Investment Banks | Investment Services | Leasing Services | Loans | Marketing and Advertising | Medical Billing | Merchant Services | Mortgages | News and Media | Offshore Services | Payroll Services | Surety Bonds | US Government Financial Sites | Venture Capital | World War II Dormant Accounts

Get "IN TO" Vicky's "rr-tit-calls" (Vicky's a Yorkshire Lass)
Latest News & Previews 

Another day, another dinner to prepare. Are you having a difficult time finding easy to prepare meals that can satisfy your growing family? You want to give your family the best, but time is in short supply and preparing a healthy meal has become a chore. Too often take out or frozen dinners have to do. Is there a way to combine healthy eating with convenience?Healthy Family Meals - Better for Children, Easier for YouYes! Quick and easy recipe, new methods of cooking and easy to prepare staples can change your eating habits and lead to a better eating style for your family. According to the. . .