Did a quick search and realized that there should be a dialogue menu that you can use to tell it to wait or to go home, but in my game, the dog just barks when I try to initiate a dialogue. I didn't tell it to follow me because there was no dialogue menu to begin with, and I couldn't tell it to stop following me either, for the same reason. The reason why I post this is because yesterday I noticed that I couldn't "talk" to the stray dog that kept following me. You can then either edit out these overrides or disable the entire plugin. In order to that, load up your entire plugin list in TES5Edit and for convenience, select "Apply Filter for Cleaning" which will show you the plugins that override these dialogue records.
(or perhaps with other non-human followers) or in other words, you don't see the dialogue menu or certain dialogue options that are supposed to be there aren't shown, then look for a mod in your load order that overrides or redirects vanilla "dialogue topics" or "dialogue branches" related to followers (e.g. If you are unable to interact with animal followers such as Meeko, the stray dog etc.