Skip to content
More support
Contact us
  • There are no suggestions because the search field is empty.

What Are robots.txt and llms.txt?

A simple overview of `robots.txt` and `llms.txt`, explaining how they help search engines and AI systems access, understand, and prioritize website content.

 robots.txt

robots.txt is a file that tells search-engine crawlers which parts of a website they can or cannot visit.

It is usually available at:

https://example.com/robots.txt

Example:

User-agent: * Disallow: /admin/  Sitemap: https://example.com/sitemap.xml

In this example, crawlers are asked not to access the /admin/ section.

robots.txt is not a security tool. Private pages should still be protected with passwords or proper access controls.

llms.txt

llms.txt is a file that helps AI tools and large language models understand the most important content on a website.

It is usually available at:

https://example.com/llms.txt

It can include:

  • A short description of the website
  • Links to important pages
  • Product or service information
  • FAQs
  • Policies and documentation

Example:

# Example Hotel  > A four-star hotel in central Lisbon.  ## Important pages  - [Rooms](https://example.com/rooms) - [Facilities](https://example.com/facilities) - [FAQ](https://example.com/faq) - [Booking](https://example.com/book)Main Difference
  • robots.txt tells crawlers where they can or cannot go.
  • llms.txt helps AI systems find and understand important information.

They can be used together, but they do not replace each other.