The only difference is that with Private you are not allowing proxies to cache the data that travels through them. In the end, it all boils down to the data contained in the pages/files you are sending.
For example, your ISP could have an invisible proxy between you and the Internet, that is caching web pages to reduce the amount of bandwidth needed and lower costs. By using cache-control:private, you are specifying that it shouldn't cache the page (but allowing the final user to do so). If you use cache-control: public, you are saying that it's okay for everyone to cache the page, and so the proxy would keep a copy.
As a rule of thumb, if it's something everybody can access (for example, the logo in this page) cache-control: public might be better, because the more people that cache it, the less bandwidth you'll need. If it's something that is related to the connected user (for example, the HTML in this page includes my username, so it won't be useful to anyone else) cache-control: private will be better, as the proxies would be caching data that won't be requested by other users, and they might also be keeping data that you don't want to be kept in servers that you don't trust.
And, of course, everything that is not public should have a private cache. Otherwise the data might be stored in a middle proxy server, were it could be accessed by anyone with access to it.
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…