Nope, this has nothing to do with multicast, sadly. My current solution is to just get off EC2 and run the crawler locally.
I have a DHT-layer ping utility, and observed that the failing node could not ping some peers that the successful node (crawling the other network) could, so I hypothesize that either EC2 or the file-sharing network was doing some rate limiting. The puzzling thing is that the local scanner has been running for hours without showing any signs of failure.
I have a DHT-layer ping utility, and observed that the failing node could not ping some peers that the successful node (crawling the other network) could, so I hypothesize that either EC2 or the file-sharing network was doing some rate limiting. The puzzling thing is that the local scanner has been running for hours without showing any signs of failure.