The short answer is - what are you asking is desired, but not supported yet, as clearly stated (although not enough emphasized, but nobody likes to highlight the limitations) in the beginning of the current Many-to-many section of the official EF Core documentation (emphasis is mine):
Many to many relationships require a collection navigation property on both sides.
Also at the end of the original tracking item Many-to-many (skip) navigation properties #19003 you can see basically the same question
Hello, is there a way to avoid defining the property for one of the entities?
e.g.
builder.HasMany(p => p.Tags).WithMany(); // notice no parameter in `WithMany`
and the response from the team is
Not yet #3864
pointing straight to Support unidirectional many-to-many relationships through shadow navigations #3864, which is the corresponding current open issue, and seems like is scheduled for 6.0 release.
As of why exactly, only the team members can answer that, but most likely it's because of the usual lack of enough time to fit something in the limited timeframe for delivering a specific release. A brand new (and not fully complete) concept (skip navigations) used to implement the actual feature, with a lot of possible improvements like Support non-many-to-many skip navigations #21673, the one in question and many others - you can see the current list here Improve many-to-many, skip navigations, and indexer properties #22960. Plus technical difficulties, the lack of shadow navigation property support (although that doesn't stop the other types of relationships which can be configured even w/o a navigation at neither side (how that would be useful is another story)) etc.
A final note in case you are looking for workaround/a way to overcome the current limitation. I usually like to go beyond the EF Core limitations, but here first I don't see a value (I personally look at navigation properties more like metadata representing a relationship in the LINQ queries rather than a storage), and also attempts to overcome it with using directly internal metadata APIs, along with the ugly looking code just led to a different runtime exceptions, which to me proves that the current code really relies on having the "inverse" navigation, and that is constrained in many places.
So at minimum you need a private ICollection<User> Users
property or field, excluded from serialization and fluently configured
modelBuilder.Entity<User>().HasMany(e => e.Roles).WithMany("Users");
and live with the fact that it will be populated from the EF Core navigation fixup. Or better off, just live with the public navigation property and wait for the official implementation of the feature.