Extraction looks for prose, so a front page, a comment thread or a search
result page correctly yields almost nothing. These render the structure
instead, and every link they emit routes back through /read.
/search?q= results from a configurable HTML endpoint
/feed?u= RSS and Atom
/read?u= now picks a site view, falling back to a link index when a
page has too little text to be an article
Feeds are scanned rather than parsed. A feed needs five fields per entry
and an HTML parser mangles XML, so this walks the tags directly: CDATA,
named and numeric entities, and Atom's preference for rel=alternate over
rel=self. No XML dependency.
Hacker News gets a real adapter: stories with score, author and a link
into the discussion, and comment threads rendered with their indent
preserved. Comment bodies go through the article renderer so links inside
them behave like every other link.
Search is deliberately engine-agnostic. Known result shapes are tried
first, then heading links, then any link, because every free HTML endpoint
eventually rate-limits a repeat visitor. When one answers with a challenge
page rather than results the reader says so and points at the config,
instead of showing an empty page; detection reads the body, since these
arrive as 200 or 202 rather than an error status. A blocked search is
never cached.
Pages that turn out to be feeds redirect to the feed view, and a page that
declares its own feed offers it.
nak0x
3f71d324c9