I just figured out that the "additional padding" is always 1em. In case you use an absolute number for your font-size, you can set the button's font-size to 0 and reset it for the inner element:
button{
font-size:0;
}
button span{
font-size:14px;
}
<button><span>Submit</span></button>
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…