If you join a game and are kicked before your lobby UI is loaded, you will still be dumped into a lobby that you won't leave automatically. I would like to thank The Cult for helping me repro this. :/