I've written custom crawling and analysis code for my audits, but if you aangezien to avoid coding, I recommend using Screaming Frog's SEO Spider to perform the site crawl (it's free for the first 500 URIs and £99/year after that). Alternatively, if you aangezien a truly free tool, you can https://gunnerhxkvc.therainblog.com/27232185/de-basisprincipes-van-robots-txt