Hello, yes sadly this is a limitation from the YouTube player, it does not allow going back to a previous video, only to the start of the current one.

However the extension is able to go back to the previous track in the current video, assuming that a tracklist has been extracted from the video description or comments.