If the bit does not fit smoothly try to pre-heat the tip of the handle with a heatgun or an hair dryer, before inserting the bit. Then leave the part to cool down with the bit inserted. That will make the perfect  size to match. Better use a brim.