A Search Engine From Scratch, Part 3: Teaching It What a Good Result Is
By the end of part 2 the search engine had a corpus that could grow without filling a disk, on hardware that could hold it. That raised a different question. A broad crawl finds the same page under many URLs, mirrors and copies, thin and spammy pages, and whole sites in languages nobody will search for here.