You are mistaken, there is no need to access your downloads.

The required permissions are explained on the wiki:
https://github.com/gorhill/uMatrix/wiki/About-the-required-permissions