I don’t think the watch-of property has to be geared towards internet videos. The URL can be the video you’re actually watching, like on YouTube, or something that represents what you watched, like Wikipedia. That link doesn’t need to be canonical, either. For example, a corollary with my event posts (example) is that I link the location to a venue page on my site where I publish the venue information with an h-card.

If there is no meaningful URL to link to, it’s possible to use p-watch-of instead and just mark up the title. In either case, I think the title of what was watched will be the most important for consumers, at least initially. Getting into categorizing (movies, TV, YouTube, etc.) and uniquely identifying titles would be more complex. That complexity might also be part of why we haven’t seen aggregators yet.