Platform :preferably Java

Based on Robot Exclusive Model.

A scalable web crawler capable of fetching, archiving, and analyzing the full diversity and breadth of internet-accessible content or part of it(as time permits the group).