nak0x
6dcb00da80
Fetches a page, discards everything that is not the article, and renders what is left as a self-contained HTML document with no scripts, no stylesheets, no webfonts and no third-party requests. The reader rule in the starter config already points at it. Extraction follows Readability: score p/pre/td/blockquote by prose weight, propagate to ancestors with decay, adjust by class and id names, scale by one minus link density, then take the winner plus sibling nodes that also read like body copy. Serialisation runs against a tag whitelist, with unlisted elements contributing their children but no tag of their own, so wrapper divs disappear. Handling for what real pages actually do: - follow meta http-equiv=refresh stubs, including inside noscript - fall back to data-src when src holds a lazy-load placeholder - take the first srcset candidate, the smallest, not the last - decode via Content-Type charset, then meta charset, then UTF-8 - resolve links against the post-redirect URL so file:// output works - ignore script and style text so a page cannot inflate its own score Output goes to a cache file named by a hash of the URL and opens in $FURST_BROWSER, $BROWSER, or the first light browser on $PATH; --html, --text, --out and --stdin cover the other uses. Restructures the repository as a workspace so the router keeps its two dependencies and its fast build.
75 lines
2.9 KiB
Markdown
75 lines
2.9 KiB
Markdown
# furst-read
|
||
|
||
Fetch a page, throw away everything that is not the article, and render what is
|
||
left as a small self-contained HTML document — no scripts, no stylesheets, no
|
||
webfonts, no third-party requests.
|
||
|
||
This is the part of the toolset that does the real work. Choosing a lighter
|
||
browser engine buys 2–3×. Not loading 3MB of JavaScript buys considerably more.
|
||
|
||
## Use
|
||
|
||
```sh
|
||
furst-read <url> # render and open in a browser (default)
|
||
furst-read --html <url> # minimal HTML to stdout
|
||
furst-read --text <url> # plain text to stdout
|
||
furst-read --out page.html <url>
|
||
furst-read --no-images <url>
|
||
curl -s <url> | furst-read --stdin --text <url>
|
||
```
|
||
|
||
The default writes to `$XDG_CACHE_HOME/furst-read/<hash>.html` and opens it in
|
||
`$FURST_BROWSER`, `$BROWSER`, or the first light browser on `$PATH`. Because the
|
||
file is named by a hash of the URL, revisiting a page reuses it instead of
|
||
littering.
|
||
|
||
Slots straight into [furst](../README.md):
|
||
|
||
```toml
|
||
[[rule]]
|
||
name = "reader"
|
||
hosts = ["wikipedia.org", "news.ycombinator.com", "*.substack.com"]
|
||
run = ["furst-read", "{url}"]
|
||
```
|
||
|
||
## How it finds the article
|
||
|
||
Readability's approach, reimplemented:
|
||
|
||
1. Score `p`, `pre`, `td` and `blockquote` by how much prose they hold —
|
||
character count and comma count, which boilerplate has neither of.
|
||
2. Push each score up to the ancestors: parent takes it whole, grandparent
|
||
half, and it thins out from there.
|
||
3. Weight by `class` and `id`. `post-content` gains, `sidebar` loses.
|
||
4. Multiply by `1 − linkDensity`, which is what separates a nav block from a
|
||
paragraph.
|
||
5. Take the winner, then pull in sibling nodes that also read like body copy,
|
||
because articles are routinely split across several divs.
|
||
|
||
Then serialise against a tag whitelist. Wrapper elements contribute their
|
||
children and no tag of their own, which is how the div soup disappears.
|
||
|
||
## Details that matter in practice
|
||
|
||
- **Redirect stubs.** A `<meta http-equiv="refresh">` is followed, including
|
||
when it is tucked inside `<noscript>`, which is how most static site
|
||
generators write them.
|
||
- **Lazy images.** A `src` holding a placeholder falls back to `data-src` and
|
||
friends, so the images are the real ones.
|
||
- **`srcset` takes the first candidate, not the last** — on old hardware the
|
||
smallest variant is the one you want.
|
||
- **Character sets.** Content-Type first, then `<meta charset>`, then UTF-8.
|
||
Latin-1 pages are still common enough to matter.
|
||
- **Links are made absolute** against the post-redirect URL, so the output
|
||
works from a `file://` path.
|
||
- Scripts and styles are never counted as text, so a page cannot inflate its
|
||
own score with an inline bundle.
|
||
|
||
## Limits
|
||
|
||
- Listing pages and forums have no single article to find. The extractor will
|
||
return whatever scored highest, which may be very little.
|
||
- Anything that renders client-side arrives empty; there is no JavaScript here
|
||
and that is the point.
|
||
- Paywalls and interstitials are not defeated, only rendered plainly.
|