Without the car in front of me my best (not really that good) guess would be that you have vacuum to the booster going the whole time.
You could start by disconnecting the vacuum line from the top of the Kelsey Hayes booster and see if this changes things. The car can be driven safely without brake boost.
Then check the air valve assembly 6 isn't jammed and that it is not being actuated with the brakes off. There is a lever on an eccentric that should be just clear of the valve with the brakes off.
Also check if the check valve c14693 on the Reservac tank is working.
Andrew.
881824, 1E21538. 889457. 1961 4.3l Mk2. 1975 XJS. 1979 MGB (supercharged).
Adelaide, Australia