Well it turns out the brake booster has had issues since I first installed it 8 years ago.
It was new in the box and the pedal had assist and everything, but the brakes never seemed as good as they should (e.g. would not lock on dry pavement no matter what) but worked well enough I could write it off.
I had also been searching for a vacuum leak most of this whole time which was always messing with my carb jetting and efi tunes.
Well after doing a head cam swap on the ls1 recently, there was a squeal coming from under the hood. I feared alternator or pulley damage during the dis-assembly process or something like that. But then noticed it would squeal for a couple seconds after killing the car.
The pedal would still have some power assist in the mornings. But the closer I listened to more it seemed like it was coming from the booster. So I reluctantly ordered one for $95 shipped on ebay. Put it on yesterday and it's a whole new car...no more idle hunts even on a cold start. I put it in gear and was amazed I had to completely lift off the pedal before the car tried to roll.
First couple times I tried to brake I nearly through myself into the steering wheel. Pedal feel is now a little firmer but consistent all the way instead of almost weightless for the first half then applying firm. The old booster was installed during a disc swap so I just assumed that's how it was supposed to be. Since it was new I never pointed my fingers back to the booster. I have literally replaced or swapped every other component: calipers, pads (tried 3 different brands), dot 4 racing fluid, checked lines, replaced blocks/valves, rear disc swap, and about 5 different master cylinders.
Lesson learned...out of the box brand new (not even rebuilt) should not be of much comfort when troubleshooting.
Now that the 1 thing I've never been happy with about the car is working superb I feel like a million bucks.
