Robots.txt is a text file with instructions for search engine robots that tells them which pages they can and cannot crawl.

These instructions are specified by “allowing” or “disallowing” the behavior of certain (or all) bots.