Extraction looks for prose, so a front page, a comment thread or a search
result page correctly yields almost nothing. These render the structure
instead, and every link they emit routes back through /read.
/search?q= results from a configurable HTML endpoint
/feed?u= RSS and Atom
/read?u= now picks a site view, falling back to a link index when a
page has too little text to be an article
Feeds are scanned rather than parsed. A feed needs five fields per entry
and an HTML parser mangles XML, so this walks the tags directly: CDATA,
named and numeric entities, and Atom's preference for rel=alternate over
rel=self. No XML dependency.
Hacker News gets a real adapter: stories with score, author and a link
into the discussion, and comment threads rendered with their indent
preserved. Comment bodies go through the article renderer so links inside
them behave like every other link.
Search is deliberately engine-agnostic. Known result shapes are tried
first, then heading links, then any link, because every free HTML endpoint
eventually rate-limits a repeat visitor. When one answers with a challenge
page rather than results the reader says so and points at the config,
instead of showing an empty page; detection reads the body, since these
arrive as 200 or 202 rather than an error status. A blocked search is
never cached.
Pages that turn out to be feeds redirect to the feed view, and a page that
declares its own feed offers it.
Following a link inside an extracted article used to drop the reader back
onto the live site. The server rewrites in-page links to /read?u=... so
browsing stays in reader mode, and caches rendered pages on disk, which
takes a revisit from ~350ms to under a millisecond.
The HTTP layer is blocking and hand-rolled: GET only, one response per
connection, a fixed pool of four worker threads so memory stays
predictable, and nothing but the loopback interface is ever bound.
Routes so far:
/ home, from ~/.config/furst/home.toml
/read?u= the article, with links routed back through the reader
/go?u= hand the original URL to the heavy browser
Pages that are not documents redirect to the original rather than being
run through an article extractor, and pages with too little text to be an
article say so and offer the escape hatches. Redirect stubs are followed
before that judgement is made.
furst-serve --open ensures a server is running, starting a detached one if
needed, then execs a browser at the reader URL. Cache entries carry a
schema number so a renderer change drops them rather than serving stale
markup.
furst-serve needs the fetch, extract and render stages, so they move behind
a lib target with the CLI as a thin consumer.
Adds Options::link_prefix. When set, http and https links are rewritten to
{prefix}{percent-encoded url} so that following one stays inside the reader
rather than dropping the viewer back onto the live site; other schemes are
left alone. render::body exposes the article markup without the standalone
document wrapper, for callers supplying their own page chrome, and CSS
becomes public so they can reuse it.
Percent encode and decode live in urljoin, next to the other URL handling.
Fetches a page, discards everything that is not the article, and renders
what is left as a self-contained HTML document with no scripts, no
stylesheets, no webfonts and no third-party requests. The reader rule in
the starter config already points at it.
Extraction follows Readability: score p/pre/td/blockquote by prose weight,
propagate to ancestors with decay, adjust by class and id names, scale by
one minus link density, then take the winner plus sibling nodes that also
read like body copy. Serialisation runs against a tag whitelist, with
unlisted elements contributing their children but no tag of their own, so
wrapper divs disappear.
Handling for what real pages actually do:
- follow meta http-equiv=refresh stubs, including inside noscript
- fall back to data-src when src holds a lazy-load placeholder
- take the first srcset candidate, the smallest, not the last
- decode via Content-Type charset, then meta charset, then UTF-8
- resolve links against the post-redirect URL so file:// output works
- ignore script and style text so a page cannot inflate its own score
Output goes to a cache file named by a hash of the URL and opens in
$FURST_BROWSER, $BROWSER, or the first light browser on $PATH; --html,
--text, --out and --stdin cover the other uses.
Restructures the repository as a workspace so the router keeps its two
dependencies and its fast build.
