What is a web crawler, really?

/ Search Off the Record