I was thinking only instances that someone had followed me from would have the local cache of my profile. Is it also on instances I have followed someone on? If so, I think followers-only seems like a good start.