Following, it is best to come to a decision how you wish the crawler to deal with numerous World-wide-web technologies. There exists an ongoing discussion regarding the intelligence of internet search engine crawlers. It's not totally clear When they are full-blown headless browsers or just glorified curl scripts (or https://www.youtube.com/watch?v=EVeTP0aV8gY