With a nudge in the right direction I managed to handle this problem even better then expected - no switching necessary.
For Mercurial solutions I use VisualHG and for Git I use Git Source Control Provider. I also installed HgSccPackage. Now I don't really know how or why this works, but I've got GitSCC selected as the Source Control Provider in the settings, but when I load a Mercurial project VisualHG takes over. All I have to do is set the SCC to VisualHG the first time I open the project, close the project, change back to GitSCC and when the project is reloaded VisualHG will be used for it.
Interestingly it does not work without HgSccPackage installed. If someone can explain this, please do.
Note that it does not seem to work the other way round.
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…