Go Crawling: Mastering Concurrent Data Fetching with Durability
A production crawler is not a loop around http.Get. It is a durable work queue attached to an unreliable network: URLs multiply, hosts push back, databases become temporarily un...