You can have the infill low, just make the wall count higher to make the holders stronger.
The first one I printed, I had a wall count of three and when I broke the supports out, the holder broke too.

So I re-printed at 15 wall count, which works well, but may have been more than needed.  By increasing the wall count, you make the hangers basically 100% infill.