Um, it’s less of a matter of aliens picking up our signals, but more of the utter lack of signals received from aliens who would likely have been alive and technologically-advanced (far beyond our own level) for eons before the technological rise of humanity. We’ve received no identifiable signal in any spectrum — even the “Wow signal” now seems to have been explained as a natural phenomenon.

That’s what Fermi was referring to. He wasn’t assuming that others would have heard us — he was pointing out that we should have heard them by now.