That's why such things are optional...
I have tried on discord, without issue, both with the extension enabled w/ optional permissions and without.
Alternatively to setting in the optional permissions, I think you can toggle it on a per site basis, by right clicking the extension and clicking "always on on *.com"
Sadly there's no method to quickly toggle the permissions on FF, I tried looking into setting a hotkey to enable/disable the extension but it wasn't working...
The default behavior is to only operate on click (when you click the extension, otherwise it shouldn't be running)