Having said that, a 4-3 heart fit might play very well.. but I think I'd need better hearts to suggest that, and he might pass thinking the same thing, and then we are in a 3-3

Can I pass? His hesitation is more likely to suggest that he was close to not bidding than anything else. I really don't think that he was 'reviewing agreements'... we don't have any and he knows that.
But is bidding a logical alternative? I think it is..... if he wouldn't, for example, open 2♦ with 4=6 in the reds, then he could easily hold cards that make game reasonable (given the auction), and even if not, a hand such as x Qxx Kxxxxxx xx is enough to give us a play.
So I think I have to bid 5♦... the bid I'd make anyway