I have 4 bits I always switch between: 4mm, torx 20, torx 25 and 5mm.
I also have a lot of 40x10x5 magnets.

The magnet is gently (tightly) pushed upwards until it reaches the top, where it will gladly hold the torx bits on contact, and the drill bits are held in by friction and a little bit of magnetism. Mostly the long drill bit channels, though.

The cut edge is for me to tactilely orient the holder with the smaller bits to the left.