How to make robots? How to write robots?

How to make robots? How to write robots? For robots, first of all, it is a protocol, which plays a very important role in the crawling of websites. The significance of creating robots is that it allows spiders to obey orders and arrangements, solves some privacy and unnecessary crawling pages of the website, and also emphasizes that a certain part must be crawled.

Robots may seem like a very simple operation, but if you don’t pay enough attention or operate it too much, the robots you write may not be correct, because it involves some instructions, which may affect the crawling of the website if not written correctly. When the editor Dongguan SEO was making a website, due to robots errors, the website has not been included. Today, the editor Feng Chao will share with some friends who have just engaged in website SEO optimization how to quickly create a robots.txt file.

How to make robots? How to write robots?

1. Open the online tool: http://tool.chinaz.com/robots/

Change the format of your own website and site map according to your needs. If you do not prohibit spider crawling, it is allowed by default, as shown in the figure:

2. After the change, generate the code and create a notepad, name it robots.txt, copy the code and save it (upload it to ftp)

Note:

The generated valid code is as follows:
User-agent: *
Disallow:
Sitemap: http://www.yunmajp.com/sitemap.xml

in:
① The wildcard "*" in "User-agent: *" represents all robots.
②“Disallow: ” means spiders are not prohibited from crawling the page. Do not write “Disallow: /”. The missing “/” means spiders are prohibited from crawling all pages.
③Sitemap: http://www.yunmajp.com/sitemap.xml: This is the address of the website map, which guides spiders to crawl your website map

Come create a robots.txt file for your website now!