Yesterday, Microsoft announced at their event a new generation of “Copilot+” PCs with new features, let’s focus on Recall, which looks to be the most controversial so far.
Going back in time: Timeline
Timeline was a Windows 10 feature that appeared late in its lifecycle, on the April 2018 release. It used a set of APIs (part of Windows.ApplicationModel.Activities) for applications to explicitly signal when an event should be recorded on the timeline.
It had a number of privacy implications, including that it used a database accessible by all un-sandboxed apps within the same user context, and provided OS-wide history. It also had cloud sync support, but that was later deprecated by Microsoft.
Rewind
Rewind is a product by rewind.ai that runs in part locally, except the LLM part, which is actually offloaded to a cloud service. It’s supposed to be relatively transparent, without requiring app changes.
Limitless is the cloud-based alternative of that same company, which comes with support for a pendant physical device, advertised as recording conversations including “lunch meetings, conferences, clients calls, or doctors visits”. This poses the question of consent, and might even be illegal in one-party recording states.
Recall
Recall is Microsoft’s new announcement from this week. It revives the Timeline concept, which disappeared in Windows 11, but with a number of twists.
First, it currently doesn’t have support for syncing context to another device. This is something that can perhaps be added later, in an end-to-end encrypted manner too.
Second, and that’s a catch with far reaching implications, it doesn’t rely on applications explicitly creating activities, but can use information from that. Instead, it relies on taking regular screenshots and then doing analysis to reconstruct a context, including OCR. Those screenshots are then kept on disk for later fast search.
Why going the screenshots way instead of deeper integration?
I think that it might be because Microsoft thinks that they can’t get developer buy-in for a deeper level of integration at this point. This has a number of implications including revocation. How to make sure that data on the Recall side is deleted when the underlying data is can become close to an intractable problem with numerous failure modes if integration is only mandated to delete data.
Spotlight shows that the content search part can be fairly efficient without storing a ton of screenshots. Better handling of historical searches in a… timeline can get a long way there.