File: /home/seopositiveltd.co.uk/tutorials-sub/category/robots-txt.html
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="https://www.w3.org/1999/xhtml" dir="ltr" lang="en-US">
<head profile="https://gmpg.org/xfn/11">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>
robots.txt « SEO Positive: SEO Tutorials, PPC Pay Per Click Tutorials, Web Design & Web Development Tutorials</title>
<link rel="canonical" href="https://www.seopositiveltd.co.uk/tutorials-sub/category/robots-txt" />
<link rel="stylesheet" href="https://tutorials.seopositiveltd.co.uk/wp-content/themes/seopositive/style.css" type="text/css" media="screen"/>
<link rel="pingback" href="https://tutorials.seopositiveltd.co.uk/xmlrpc.php"/>
<!--[if IE 6]>
<link rel="stylesheet" href="/wp-content/themes/seopositive/ie6.css" type="text/css" media="screen" />
<![endif]-->
<!--[if IE 7]>
<link rel="stylesheet" href="/wp-content/themes/seopositive/ie7.css" type="text/css" media="screen" />
<![endif]-->
<script src="../../ajax/libs/jquery/1.4.2/jquery.min.js" type="text/javascript"></script>
<link rel="stylesheet" id="contact-form-7-css" href="https://tutorials.seopositiveltd.co.uk/wp-content/plugins/contact-form-7/styles.css?ver=2.4.2" type="text/css" media="all"/>
<script type="text/javascript" src="https://tutorials.seopositiveltd.co.uk/wp-includes/js/jquery/jquery.js?ver=1.4.2"></script>
<link rel="index" title="SEO Positive: SEO Tutorials, PPC Pay Per Click Tutorials, Web Design & Web Development Tutorials" href="https://tutorials.seopositiveltd.co.uk"/>
<script src="https://tutorials.seopositiveltd.co.uk/wp-content/themes/seopositive/js/AC_RunActiveContent.js" type="text/javascript"></script>
</head>
<body class="archive category category-robots-txt">
<div id="wrapper">
<div class="hidden"></div>
<div id="header">
<div class="hidden">
</div>
<div id="logo"><a href="https://www.seopositive.co.uk/"></a></div>
<div id="header-info">
<div class="menu-top-container"><ul id="menu-top" class="menu"><li id="menu-item-4228" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-4228"><a href="https://my.seopositive.co.uk/">Client Login</a></li>
<li id="menu-item-4227" class="menu-item menu-item-type-post_type menu-item-object-page menu-item-4227"><a href="https://www.seopositive.co.uk/careers.php">Careers</a></li>
<li id="menu-item-4229" class="menu-item menu-item-type-post_type menu-item-object-page menu-item-4229"><a href="https://www.seopositive.co.uk/partners.php">Partners</a></li>
</ul></div>
<div class="clear"></div>
<div class="call-us"> 0800 088 6000</div>
<div class="clear"></div>
</div>
<div class="clear"></div>
</div>
<div id="content-container">
<div id="content" class="narrowcolumn" role="main">
<ul id="menu-main">
<li><a href="/">Home</a>
<ul class="first"><li><a href="/">Home</a></li></ul>
</li>
<li><a href="../../about-us">About us</a>
<ul><li><a href="../../about-us">About us</a></li></ul>
</li>
<li><a>Services</a>
<ul class="dropdown"><li><a>Services</a></li></ul>
<ul class="services-bg">
<ul>
<li><a href="../../search-engine-optimisation">Search Engine Optimisation</a></li>
<li><a href="../../pay-per-click">Pay Per Click</a></li>
<li><a href="https://www.webdesignpositive.co.uk/">Website Design</a></li>
<li><a href="../../website-design-portfolio">Design Portfolio</a></li>
</ul>
<ul>
<li><a href="../../email-marketing">Email Marketing</a></li>
<li><a href="../../reputation-management">Reputation Management</a></li>
<li><a href="../../link-building">Link Building</a></li>
<li><a href="../../hosting">Hosting</a></li>
</ul>
<div class="clear"></div>
</ul>
</li>
<li><a href="../../search-engine-optimisation">SEO Services</a>
<ul><li><a href="../../search-engine-optimisation">SEO Services</a></li></ul>
</li>
<li><a href="../../pay-per-click">PPC Services</a>
<ul><li><a href="../../pay-per-click">PPC Services</a></li></ul>
</li>
<li><a>Testimonials</a>
<ul class="dropdown"><li><a>Testimonials</a></li></ul>
<ul class="testimonials-bg">
<ul>
<li><a href="../../seo-testimonials">Search Engine Optimisation</a></li>
<li><a href="../../ppc-testimonials">Pay Per Click</a></li>
<li><a href="../../web-design-testimonials">Website Design</a></li>
</ul>
<div class="clear"></div>
</ul>
</li>
<li><a href="../../awards">Awards</a>
<ul><li><a href="../../awards">Awards</a></li></ul>
</li>
<li><a href="../../contact-us">Contact us</a>
<ul><li><a href="../../contact-us">Contact us</a></li></ul>
</li>
<li><a>Blog</a>
<ul class="last"><li><a href="../../blog">Blog</a></li></ul>
</li>
</ul>
<div id="main-content">
<div id="content-left">
<div class="post-721 post type-post hentry category-robots-txt tag-adding-a-simple-robots-txt-file" id="post-721">
<h2><a href="/" rel="bookmark" title="Permanent Link to Adding A Simple Robots Txt File">Adding A Simple Robots Txt File</a></h2>
<hr/>
<p><small>July 13th, 2011 <!-- by admin --></small></p>
<div class="entry">
<div class="tweetmeme_button" style="float: right; margin-left: 10px;"><iframe src="https://api.tweetmeme.com/button.js?url=http%3A%2F%2Ftutorials.seopositiveltd.co.uk%2Fadding-a-simple-robots-txt-file.php&style=normal" height="61" width="50" frameborder="0" scrolling="no"></iframe></div>
<p>If your website doesn’t currently have a robots.txt file, your site may not be ranking as highly as you’d like within the search engines. This tutorial will teach you how to set up the file.</p>
<p>The robots.txt file is especially important because it tells the spiders from any of the major search engines which pages to index and which pages to ignore. The file itself needs to be placed within your root directory as a simple text file (no need for HTML). A typical robots.txt file may look like the following example:</p>
<p>User-agent: *</p>
<p>Disallow: /</p>
<p>By inserting a *, you are telling the search engine bots that this command refers to all of them. You can also insert the specific name of a search engine crawler here, which would consequently make all of the commands relevant to that bot alone.<br/>
The / indicates that the bots are not to index any of the pages on the site (this isn’t desirable if you’re working hard on your SEO, of course).</p>
<p>There isn’t an ‘Allow ‘, so therefore the following file will enable all of the search engines to index every page on your site:</p>
<p>User-agent: *<br/>
Disallow:</p>
<p>Leaving the Disallow field blank indicates that there are no indexing restrictions.</p>
<p>Bear in mind that the commands should always be used in the order of ‘user-agent’>’disallow’, as any variations to this are unlikely to work.</p>
<p>You can also disallow the search engine spiders to index specific pages within your website, if you’d rather that some information wasn’t made available to the masses. A good example would be the following:</p>
<p>User-agent: *<br/>
Disallow: /example-page/<br/>
Disallow: /sub-example/<br/>
Disallow: /second-sub-example/</p>
<p>Take into consideration that the bots will not recognise a long-tailed line of subpages, so the following example would be incorrect:</p>
<p>User-agent: *<br/>
Disallow: /example-page/sub-example/second-subexample/</p>
</div>
<p class="postmetadata">Tags: <a href="/" rel="tag">Adding A Simple Robots Txt File</a><br/> Posted in <a href="../../tutorials-sub/category/robots-txt" title="View all posts in robots.txt" rel="category tag">robots.txt</a></p>
</div>
<div class="navigation">
<div class="alignleft"><p></p></div>
<div class="alignright"><p></p></div>
</div>
<div class="clear"></div>
<hr/>
</div>
<div id="sidebar">
<h3 class="widget-title">Categories</h3> <ul>
<li class="cat-item cat-item-10"><a href="../../tutorials-sub/category/apache" title="View all posts filed under apache">apache</a>
</li>
<li class="cat-item cat-item-3"><a href="../../tutorials-sub/category/brain-candy" title="View all posts filed under Brain Candy">Brain Candy</a>
</li>
<li class="cat-item cat-item-214"><a href="../../tutorials-sub/category/cms" title="View all posts filed under CMS">CMS</a>
</li>
<li class="cat-item cat-item-66"><a href="../../tutorials-sub/category/other-technology/cold-fusion" title="View all posts filed under cold fusion">cold fusion</a>
</li>
<li class="cat-item cat-item-29"><a href="../../tutorials-sub/category/command-line-editors" title="View all posts filed under Command Line Editors">Command Line Editors</a>
</li>
<li class="cat-item cat-item-206"><a href="../../tutorials-sub/category/pay-per-click-ppc/conversion-tracking-pay-per-click-ppc" title="View all posts filed under Conversion Tracking">Conversion Tracking</a>
</li>
<li class="cat-item cat-item-11"><a href="../../tutorials-sub/category/css" title="View all posts filed under css">css</a>
</li>
<li class="cat-item cat-item-133"><a href="../../tutorials-sub/category/pay-per-click-ppc/display-network" title="View all posts filed under Display Network">Display Network</a>
</li>
<li class="cat-item cat-item-174"><a href="../../tutorials-sub/category/google-1" title="View all posts filed under Google +1">Google +1</a>
</li>
<li class="cat-item cat-item-132"><a href="../../tutorials-sub/category/pay-per-click-ppc/google-adwords-2" title="View all posts filed under Google Adwords">Google Adwords</a>
</li>
<li class="cat-item cat-item-115"><a href="../../tutorials-sub/category/web-analytics/google-analytics" title="View all posts filed under google analytics">google analytics</a>
</li>
<li class="cat-item cat-item-102"><a href="../../tutorials-sub/category/shopping-feeds/google-product-feed" title="View all posts filed under Google Product Feed">Google Product Feed</a>
</li>
<li class="cat-item cat-item-190"><a href="../../tutorials-sub/category/shopping-feeds/google-shopping-shopping-feeds" title="View all posts filed under Google Shopping">Google Shopping</a>
</li>
<li class="cat-item cat-item-200"><a href="../../tutorials-sub/category/google-webmaster" title="View all posts filed under Google Webmaster">Google Webmaster</a>
</li>
<li class="cat-item cat-item-12"><a href="../../tutorials-sub/category/htmlxhtml" title="View all posts filed under html/xhtml">html/xhtml</a>
</li>
<li class="cat-item cat-item-128"><a href="../../tutorials-sub/category/illustrator" title="View all posts filed under Illustrator">Illustrator</a>
</li>
<li class="cat-item cat-item-13"><a href="../../tutorials-sub/category/javascript" title="View all posts filed under javascript">javascript</a>
</li>
<li class="cat-item cat-item-42"><a href="../../tutorials-sub/category/joomla" title="View all posts filed under joomla">joomla</a>
</li>
<li class="cat-item cat-item-22"><a href="../../tutorials-sub/category/jquery" title="View all posts filed under jQuery">jQuery</a>
</li>
<li class="cat-item cat-item-21"><a href="../../tutorials-sub/category/jquery-tutorial" title="View all posts filed under jQuery Tutorial">jQuery Tutorial</a>
</li>
<li class="cat-item cat-item-27"><a href="../../tutorials-sub/category/linux" title="View all posts filed under Linux">Linux</a>
</li>
<li class="cat-item cat-item-41"><a href="../../tutorials-sub/category/mootools" title="View all posts filed under mootools">mootools</a>
</li>
<li class="cat-item cat-item-14"><a href="../../tutorials-sub/category/mysql" title="View all posts filed under mysql">mysql</a>
</li>
<li class="cat-item cat-item-71"><a href="../../tutorials-sub/category/mysql/mysqli" title="View all posts filed under MySQLi">MySQLi</a>
</li>
<li class="cat-item cat-item-131"><a href="../../tutorials-sub/category/pay-per-click-ppc" title="View all posts filed under Pay Per Click PPC">Pay Per Click PPC</a>
</li>
<li class="cat-item cat-item-67"><a href="../../tutorials-sub/category/other-technology/perl" title="View all posts filed under perl">perl</a>
</li>
<li class="cat-item cat-item-19"><a href="../../tutorials-sub/category/photoshop" title="View all posts filed under Photoshop">Photoshop</a>
</li>
<li class="cat-item cat-item-9"><a href="../../tutorials-sub/category/php" title="View all posts filed under php">php</a>
</li>
<li class="cat-item cat-item-51"><a href="../../tutorials-sub/category/php/php-framework" title="View all posts filed under php framework">php framework</a>
</li>
<li class="cat-item cat-item-69"><a href="../../tutorials-sub/category/php/programming-php" title="View all posts filed under Programming PHP">Programming PHP</a>
</li>
<li class="cat-item cat-item-169 current-cat"><a href="../../tutorials-sub/category/robots-txt" title="View all posts filed under robots.txt">robots.txt</a>
</li>
<li class="cat-item cat-item-134"><a href="../../tutorials-sub/category/pay-per-click-ppc/search-network" title="View all posts filed under Search Network">Search Network</a>
</li>
<li class="cat-item cat-item-23"><a href="../../tutorials-sub/category/seo" title="View all posts filed under seo">seo</a>
</li>
<li class="cat-item cat-item-101"><a href="../../tutorials-sub/category/shopping-feeds" title="View all posts filed under Shopping Feeds">Shopping Feeds</a>
</li>
<li class="cat-item cat-item-1"><a href="../../tutorials-sub/category/uncategorized" title="View all posts filed under Uncategorized">Uncategorized</a>
</li>
<li class="cat-item cat-item-28"><a href="../../tutorials-sub/category/unix" title="View all posts filed under Unix">Unix</a>
</li>
<li class="cat-item cat-item-114"><a href="../../tutorials-sub/category/web-analytics" title="View all posts filed under web analytics">web analytics</a>
</li>
<li class="cat-item cat-item-62"><a href="../../tutorials-sub/category/website-design" title="View all posts filed under website design">website design</a>
</li>
<li class="cat-item cat-item-39"><a href="../../tutorials-sub/category/wordpress" title="View all posts filed under wordpress">wordpress</a>
</li>
</ul>
<h3 class="widget-title">Archives</h3> <ul>
<li><a href="../../tutorials-sub/2011/11" title="November 2011">November 2011</a></li>
<li><a href="../../tutorials-sub/2011/10" title="October 2011">October 2011</a></li>
<li><a href="../../tutorials-sub/2011/09" title="September 2011">September 2011</a></li>
<li><a href="../../tutorials-sub/2011/08" title="August 2011">August 2011</a></li>
<li><a href="../../tutorials-sub/2011/07" title="July 2011">July 2011</a></li>
<li><a href="../../tutorials-sub/2011/04" title="April 2011">April 2011</a></li>
<li><a href="../../tutorials-sub/2011/03" title="March 2011">March 2011</a></li>
<li><a href="../../tutorials-sub/2011/02" title="February 2011">February 2011</a></li>
<li><a href="../../tutorials-sub/2011/01" title="January 2011">January 2011</a></li>
<li><a href="../../tutorials-sub/2010/12" title="December 2010">December 2010</a></li>
<li><a href="../../tutorials-sub/2010/11" title="November 2010">November 2010</a></li>
<li><a href="../../tutorials-sub/2010/10" title="October 2010">October 2010</a></li>
<li><a href="../../tutorials-sub/2010/09" title="September 2010">September 2010</a></li>
<li><a href="../../tutorials-sub/2010/08" title="August 2010">August 2010</a></li>
<li><a href="../../tutorials-sub/2010/07" title="July 2010">July 2010</a></li>
<li><a href="../../tutorials-sub/2010/06" title="June 2010">June 2010</a></li>
<li><a href="../../tutorials-sub/2010/05" title="May 2010">May 2010</a></li>
</ul>
<h3 class="widget-title">Blogroll</h3>
<ul class="xoxo blogroll">
<li><a href="https://www.seopositive.co.uk/cheap-seo.php">Cheap SEO</a></li>
<li><a href="https://www.integratedcctv.co.uk/" title="Integrated CCTV, Security, news and reviews">Integrated CCTV</a></li>
<li><a href="https://www.payperclick.org/">Pay Per Click Management</a></li>
<li><a href="https://www.payday-loans.co.uk/">Payday Loans</a></li>
<li><a href="https://www.seopositive.co.uk/pay-per-click.php">PPC Marketing</a></li>
<li><a href="https://www.searchengineoptimisation.org.uk/">Search Engine Optimisation</a></li>
<li><a href="https://www.seopositive.co.uk/">SEO</a></li>
<li><a href="https://www.seopositive.co.uk/blog">SEO Blog</a></li>
<li><a href="https://www.seopositive.net/">SEO Positive</a></li>
<li><a href="https://my.seopositive.co.uk/">SEO Positive</a></li>
<li><a href="https://news.seopositive.co.uk/">SEO Positive News</a></li>
<li><a href="https://tutorials.seopositive.co.uk/">SEO Positive Tutorials</a></li>
<li><a href="https://www.promocodes.co.uk/">voucher codes website</a></li>
<li><a href="https://www.seopositive.co.uk/website-design.php">Website Design Company</a></li>
</ul>
</div>
<div class="clear"></div>
</div>
</div>
</div>
<div id="footer" role="contentinfo">
<div id="footer-content">
<div class="footer-section">
<h3>Additional Pages</h3>
<div class="menu-additional-pages-container"><ul class="menu" id="menu-additional-pages"><li class="menu-item menu-item-type-custom menu-item-645" id="menu-item-645"><a href="../../my-sub/index">Client Login</a></li>
<li class="menu-item menu-item-type-post_type menu-item-44" id="menu-item-44"><a href="../../terms">Terms & Conditions</a></li>
<li class="menu-item menu-item-type-post_type menu-item-270" id="menu-item-270"><a href="../../privacy-policy">Privacy Policy</a></li>
<li class="menu-item menu-item-type-custom menu-item-2699" id="menu-item-2699"><a href="../../tutorials-sub/index">Tutorials</a></li>
</ul></div></div>
<div class="footer-section">
<h3>Sitemaps & Feeds</h3>
<div class="menu-sitemaps-feeds-container"><ul class="menu" id="menu-sitemaps-feeds"><li class="menu-item menu-item-type-post_type menu-item-697" id="menu-item-697"><a href="../../sitemap">Sitemap</a></li>
<li class="menu-item menu-item-type-custom menu-item-699" id="menu-item-699"><a href="https://www.seopositiveltd.co.uk/sitemap.xml">XML Sitemap</a></li>
<li class="menu-item menu-item-type-custom menu-item-2740" id="menu-item-2740"><a href="../../blog">Blog</a></li>
<li class="menu-item menu-item-type-custom menu-item-2761" id="menu-item-2761"><a href="../../news-sub/index">News</a></li>
</ul></div></div>
<div class="footer-section footer-section-big">
<h3>Our Address</h3>
<p><strong>Chelmsford Head Office</strong><br/>Milstrete House<br/>29 New Street<br/>Chelmsford<br/>CM1 1NT</p>
</div>
<div class="footer-section">
<h3> </h3>
<p><strong>London Head Office</strong><br/>3rd Floor<br/>1 Ropemaker Street<br/>London<br/>EC2Y 9HT</p>
</div>
<div class="footer-section footer-section-small no-padding-right">
<h3>Social Links</h3>
<a href="/" target="_blank" rel="nofollow"></a>
<a href="https://twitter.com/seopositiveltd" target="_blank" rel="nofollow"></a>
<a href="https://www.youtube.com/user/seopositive" target="_blank" rel="nofollow"></a>
<h3>W3C Compliant</h3>
</div>
<div class="clear"></div>
</div>
<div id="footer-btm">
<p>© Copyright 2010 SEO Positive Ltd | Registered company in England & Wales | Company Number: 06643590 | VAT Number: 974449769</p>
</div>
</div>
</div>
<script src="https://tutorials.seopositiveltd.co.uk/wp-content/themes/seopositive/js/jqueryui.js" type="text/javascript"></script>
<script src="https://tutorials.seopositiveltd.co.uk/wp-content/themes/seopositive/js/jquery.anythingslider.js" type="text/javascript"></script>
<script src="https://tutorials.seopositiveltd.co.uk/wp-content/themes/seopositive/js/mac-screen-slider.js" type="text/javascript"></script>
<script src="https://tutorials.seopositiveltd.co.uk/wp-content/themes/seopositive/js/class.horinaja.jquery.js" type="text/javascript"></script>
<script src="https://tutorials.seopositiveltd.co.uk/wp-content/themes/seopositive/js/javascript.js" type="text/javascript"></script>
<script src="https://tutorials.seopositiveltd.co.uk/wp-content/themes/seopositive/js/read-more.js" type="text/javascript"></script>
<script type="text/javascript" src="https://tutorials.seopositiveltd.co.uk/wp-content/plugins/contact-form-7/jquery.form.js?ver=2.47"></script>
<script type="text/javascript" src="https://tutorials.seopositiveltd.co.uk/wp-content/plugins/contact-form-7/scripts.js?ver=2.4.2"></script>
</body>
</html>