Non-fiction: BackRub (early Google research project)

Overview
BackRub was Larry Page's 1996 Stanford research project that reframed the web as a graph whose structure could be measured and exploited for search. Conceived while Page was a PhD student, and quickly joined by Sergey Brin, the project investigated whether the pattern of hyperlinks, especially backlinks, could reveal page importance more reliably than keyword frequency. The name "BackRub" playfully underscored its core idea: ranking by backlinks.

Research Aim and Insight
The central question was how to bring order to a rapidly expanding web where early engines relied on on-page text and crude heuristics. Page hypothesized that links function like scholarly citations: a link from one page to another is an endorsement, and endorsements from prominent pages should count more than those from obscure ones. This recursive notion of authority led to a global ranking scheme rather than purely local, per-document relevance measures. Brin contributed probabilistic and data-mining perspectives that sharpened the formalism and experiments.

System Design
BackRub comprised a crawler, an indexer, a link database, and a query engine, built from commodity hardware on Stanford servers. The crawler harvested pages and their outbound links, storing raw HTML in a compressed repository. The indexer parsed text and anchors, assembling a lexicon and posting lists while separately constructing the web's link graph. Special emphasis fell on anchor text, the words used in hyperlinks, which often summarized target pages better than the targets themselves. The query engine combined term-matching scores with the link-derived importance metric to produce ranked results.

PageRank and Relevance
BackRub's signature contribution was PageRank, a mathematical model of link-based importance inspired by a "random surfer" who follows links with some probability and occasionally jumps to a random page. A page earns high rank if many pages link to it, and even more so if those linking pages themselves have high rank. The rank is computed iteratively over the link graph and converges to a stable distribution. At query time, BackRub multiplied text relevance signals, term frequency, inverse document frequency, field weighting, and anchor text evidence, by PageRank to surface pages that were both topically relevant and globally authoritative. This synthesis countered tactics like keyword stuffing and rewarded credible, well-referenced sources.

Experiments and Results
Early tests within the Stanford domain, then across broader crawls, showed substantial improvements in perceived result quality. Demonstrations highlighted results where anchor text corrected for sparse on-page keywords, and where authoritative sources appeared near the top without elaborate, domain-specific tuning. Faculty, students, and lab users informally validated precision through everyday queries, providing feedback that shaped synonym handling, duplicate detection, and stemming. Even at modest scale for the time, millions of pages and a large fraction of the publicly crawlable links, BackRub showed that link analysis produced cleaner, more trustworthy rankings than text-only methods.

Constraints and Challenges
Resource limits drove engineering choices: compressed repositories, efficient posting formats, and cautious crawling to respect bandwidth and robots exclusions. Freshness and update latency were persistent concerns, as recomputing global ranks over an expanding graph was computationally heavy. The team recognized potential vulnerabilities, link farms, nepotistic linking, and experimented with heuristics and damping to blunt manipulation, foreshadowing later anti-spam work.

Impact and Legacy
BackRub crystallized the principle that structure matters as much as content on the web. The project matured into Google, with PageRank and anchor-aware indexing at its core, and was later documented in papers such as "The Anatomy of a Large-Scale Hypertextual Web Search Engine". Beyond launching a company, the work helped shift information retrieval toward graph-aware algorithms, influenced research on network centrality and recommendation, and set expectations that search could be both relevant and robust at web scale.