I think the problem with visibility lays in the fact that the layout of other elements change depending on whether or not the icon is visible. I don’t know for sure if any of my suggestions will solve your problem but you might try one of the following:
You either make sure that both icons are enclosed in a rectangle with a fixed position. You then have to play around to find out if you first need to group the rectangle with the two icons and to find out which elements’ positions you have to lock (e.g. you can either choose to set a fixed position to the group or you can set a fixed position for each of the three elements).
Or you choose to exchange the red dot icon for a toggle that has no icon when the condition is false. In that case you have to play around with where you store the condition of the toggle and how you will set it up right.
I’ve never tried this myself, but I hope my brain wave will help you to come a little closer to a solution.