MWZ

MINDWAREZONE

Tutorials

Laravel Python

Laravel Python Blog

Technical SEO tool

Tell search crawlers which areas they may access and point them to your sitemap—without writing directives by hand.

Changes appear in the preview instantly.

Default crawler access

Allow all Custom Block all

Blocking crawling does not securely hide private content. Protect sensitive pages with authentication.

User agent

Crawl delay (optional)

Googlebot ignores this directive.

Allowed paths

Enter one path per line.

Disallowed paths

Paths are case-sensitive.

Sitemap URLs

Use absolute URLs, one per line.

Valid format

Save the file as robots.txt and place it at your site root, such as https://example.com/robots.txt.