Holley #3310, should the check ball be left out?

No. The ball is there to help control the movement of the secondary throttle blades. Removing it will not help performance, and if you do remove it and it does seem to help, you need to put it back in and find whats wrong with your carb.