Why we only store the headline and the link
Most aggregators keep a short excerpt of the article and argue it is short enough to be fair. We decided not to keep one at all, and the product got simpler as a result.
Every news aggregator has to answer the same question early on: how much of somebody else’s article are you going to keep?
The usual answer is an excerpt. Two hundred characters, maybe four hundred, on the reasoning that a short enough extract is fair dealing. It is a defensible position. It is also a position you have to keep defending, in every jurisdiction you operate in, forever.
We went a different way. We store the headline and the link. That is the whole list.
What we actually keep
For each item: the title, the URL, the canonical URL, the timestamp, the author if the feed gives us one, an image URL, and the language. Roughly three hundred bytes.
What we do not keep is any passage of the publisher’s writing. Not a truncated paragraph, not the first two sentences, and not the <description> field even when the feed hands it to us on a plate.
The summary you see is ours
Pages built on MyNewsFeed still show a line or two under each headline, because a wall of bare links is miserable to read. That line is written by us.
The mechanism is boring, which is the point. When an item arrives, the crawler fetches the article once, holds the HTML in memory, asks a model to summarise it, keeps the summary, and throws the HTML away. Nothing is written to disk. Nothing is rendered verbatim. What ends up on the page is new text about the article rather than a copy of any part of it.
Because our sources are shared across every customer, that summary is generated once and reused by everyone following that source. An item that five hundred pages carry is summarised a single time. That is what makes it affordable to do at all.
Why this is a stronger position than a short excerpt
The EU press publishers’ right exists specifically to cover snippets of press publications. It carves out hyperlinks and individual words, and it exempts “very short extracts”, which is a phrase that has kept a lot of lawyers busy.
An excerpt policy lives inside that argument. You are asserting your extract is short enough, and that assertion can be tested.
A headline and a link sits in the carve-out instead. It is also, straightforwardly, the payload an RSS feed is published to distribute. A feed exists to be syndicated. Taking the two fields it was built to hand out, and nothing else, is difficult to characterise as taking too much.
What it costs us
Less than you would think, and it forced two decisions we are happier for.
The scraper recipe format has no summary field. You cannot configure a MyNewsFeed scraper to lift a paragraph off a page, because there is nowhere for it to go. Scraped sources and syndicated feeds end up treated identically, which means the same argument covers both.
Storage got cheaper, which is a footnote rather than a reason. Under ten gigabytes a year at the volumes we are planning for, against something closer to thirty with excerpts.
The part that matters more than the law
Every item on every page links out, and names its source. Readers end up on the publisher’s site.
We are asking a few hundred organisations to be comfortable with us reading their feeds on behalf of strangers. The cheapest way to stay welcome is to take less than we are entitled to and send the traffic back. Being legally defensible is the floor, not the aim.