tl;dr version: Holy crap, that looks nice, but (1) current limit that shit, and (2) don't try to sink 47*20mA through your microprocessor!
Wow.
You're sinking 47 LEDs though an ATMega32U4 without any current limiting resistors, and it's still operational?
I'm actually really surprised this survived power-on. I'll be even more surprised if it lives more than a month. There are two things I think you should change:
The microcontroller in question is only rated to sink a maximum of 40mA through a given pin. It will sink more than that - the ATMega32U4 datasheet calls this out - but it also calls out an absolute maximum of 100mA sink for any pin group, and a total package sink maximum of 200mA. Violating that spec will reduce the lifespan of the microcontroller.
Running LEDs without limiting current is a bad idea. They'll happily consume all of the current you source them with. I've (intentionally) blown them up - literally; shards of the lens went flying - by powering them without current limiting. In this case, you're actually using the microcontroller itself as a current limit.
Here's a solution that's pretty trivial: Hang a mosfet off of one of the PWM outputs, put in current-limiting resistors for each LED, and you'll be set.
Here's a reference to the proposed low-side circuit.
This MOSFET would work just fine.
Here's a fairly straightforward explanation of why you should have one resistor per LED.
Here's a handy calculator for figuring out the right value. Keep the total power consumption for the entire thing - including microprocessor - under 500mA, or you may run into problems (host port shutdown via overcurrent protection)).