Robots is useful to prevent your website from search engines indexing. You can simply add robots.txt in your website as well as set the default robots settings from admin.
Go to configuration->system->design section to set the value for robots. You can simply set the value to Follow, index or no index, follow or no follow, no index. When you set the value to no index, follow then search engine will visit your website and follow the link but does not add link in its index. If you don't want to share the content of your page then you can set the same value.

If you want to set the robot value for particular page then you need to set the code in theme layout file local.xml

<?xml version="1.0"?>
<layout version="0.1.0">
    <cms_index_index>
        <reference name="head">
            <action method="setRobots"><value>NO INDEX,FOLLOW</value></action>
        </reference>
    </cms_index_index>
</layout >