There are a few possibilities...
I have seen P4 modules with two different types of light sensor. A small 2x3mm one, very thin, that is actually soldered to the two connection pads (I believe this is the later type), and a larger, thicker, 2.5x6mm one (which is identical to the one used on the P2 and P3 modules). The larger one is affixed with conductive epoxy, and is easily removed and replaced; the other requires a fine soldering tip and a steady hand.
Second, this display may have been a bit on the dim side from the very beginning; manufacturing techniques for LED displays were in their infancy back in the mid-70s, and some displays left the factory with marginal brightness levels. Only fix here would be a display change-out (a difficult and dangerous procedure, at best).
Last, bad batteries. LEDs are voltage-driven devices, and brightness is sensitive to input voltage. Silver-oxide are the best choice tp power them. If a fresh set of AgO cells don't improve the display, you are back to possibilities one and two.