Well, there you go... There is no limiting resistor between the cap and the LED, so when you first switch it on, the cap spikes a large current through the LED. A few spikes and she's a DED (dark emitting diode), Jim. The limiting resistor needs to go between the LED and the cap.
<edit> More analysis: When the switch is open, the cap is sitting at 24V. When you close the switch, the cap discharges until it is down to Vf of the diode (about 1.5V for red, higher for other colors). During this discharge, a large current spike flows through the LED because there is no limiting resistor between the stored charge in the cap and the LED.
A P
<edit> Additional info: You should have a current limiting resistor for each LED. Otherwise any I-V curve differences will cause one to hog current and they won't all work (or their brightness will vary depending on how many are on). 2k-3k would probably be fine.