There is no greater force driving your Search Engine Position than your content. If the content on your site is unique and informative, in no time, your site ranking for keywords related to that content steadily increases to top position.
Speed of this rise is proportional to the particular search engine’s crawling rate.
To allow all search bots entry to all pages in your site, create a text file named “robots.txt” using notepad.exe and upload it to your site root. (Directory “public_html” if your site is hosted on a linux server and “wwwroot” if it is on a windows server)
If you would like to restrict the bot from accessing files in specific directories, add a line
Disallow: /directory full path/
Where “directory full path” gives the full path of the directory you want to hide from visitors. (Don’t forget the colon!)
You nay also allow and disallow specific bots by writing their name instead of “*” in the code line
Sitemaps help the search bots index pages that are not otherwise visible to them. A sample sitemap for Google is given below.
<?xml version=”1.0″ encoding=”UTF-8″?>
<video:player_loc allow_embed=”yes” autoplay=”ap=1″>http://www.example.com/videoplayer.swf?video=123</video:player_loc>
<video:title>Grilling steaks for summer</video:title>
<video:description>Get perfectly done steaks every time</video:description>
And that was a single entry for a URL that includes an image and a video!!!
Don’t worry if this looks a bit complex. I can create sitemaps for your site content in a few hours and send it to you by email (if you want).
If you want to control what Google does, you’re going to have to gain a profound understanding of the algorithms they use and keep up to date as the code changes. You’re contending with some of the shrewdest computer scientists in the world.
Just for fun I’ve included links to Google’s own PageRank algorithm by Sergei Brin & Lawrence Page and the Hilltop algorithm by Krishna Bharat to get you started. Be warned that if you’re caught you’ll be black-listed by Google and will have to crate a new website. But it may be fun reading if you are a hacker.