Ray´s Guide to HTML: Defining Meta tag Robot value

A robot is a type of program that search engines use to search the Web site documents and update their databases. Robots use the meta information to index if for the engine. You as a developer can use control how much a robot reads of your site.

Enter a meta tag in your head section. use the name attribute and set it to robots. Instruct robots to read the entire page and follow all the links within it, by setting the content to all, follow

Example:
------------------------------------



<head>

<title>Defining Meta tag Robot value</title>

<meta name="keywords" content="Ray´s Guide to HTML, Guide, HTML, Hypertext Markup Language, HTML Tutorial" />

<meta name="descrition" content="Ray´s Guide to HTML - An introductory guide for the beginner" />

<meta name="author" content="yourname" />

<meta name="expires" content="wed, 15 June 2011 02:00:00:00 GTM" />

<meta http-equiv="refresh" content="5" />



<meta name="robots" content="all, follow" /">



</head>

------------------------------------
Don´t forget the forward slash at the end of meta tag

Note - The reason for allowing a robot to crawl your site are obvious. However there are times you don´t want this to happen; for example a document you don´t want the world to se yet and pages for only a limited audience.

To instruct robots to read all, but retain from following links:

<meta name="robots" content="all, nofollow" />

To instruct robots not to read at all:

<meta name="robots" content="none" />