Hi @athie,
There was a similar thread recently, see Very weird (or bug?) behaviour with 2 types of "follow" relationships
Not sure if @jnestour found the root cause of the problem.
I’ve mad a POC with regular buttons, for me it worked perfectly. What I could suggest - is that you recreate the follow/unfollow buttons from scratch, and see if it works.
Otherwise it is difficult to find the cause of the problem without looking at the app itself.