No, it is none of our business what anyone does in their personal lives. That includes people in the public eye–not counting those who make money by selling out their privacy (e.g., the Kardashians).
The real exception is for people in leadership positions. Kennedy, Clinton, Patraeus–all of these men, despite acts of commendable service to their country, are open to scrutiny.
If you take on a leadership role, you must earn our utmost trust and you must lead by example. The enormous pressures of office may take their toll, but if you want to lead effectively, you have to avoid your vices.
Sure, everyone has their challenges, but when you are chosen to represent something larger than yourself, you have to transcend your limitations, not revel in what you can now get away with.