For me it's the opposite, when I start FF, QD need time to load his config.
Opening a new Window or Tab is very speed.
The only case I know that QD take the same time as FF start is in private mode.
Case 1 : You have an other extention that have side effect with QD
Case 2 : Your FF configuration is the same as the private mode but the delay should not increase
Case 3 : Need investigation...

EDIT: Thanks for the feedback, just tested speedly and with no custom config, seem to work for me.