Finally got rid of that pesky sidebar. My ad blocker was already blocking the ads but Outlook was complaining about it, saying that I should upgrade if I want to get rid of ads. What a relief.

Now please, also get rid of that 'Upgrade to premium' button on the lower left :)