The crawler has to make a decision about which URLs should be downloaded first, which URLs should be downloaded later and which URLs are not worthy downloading at all.
there is some work in ordering the frontier for a crawl according to the predicted quality of the unknown pages [Cho, Garcia-Molina and Page, 1998;Eiron et al., 2004].