talking local on 20 and 40 meters..

this maybe a dumb question but talking to a friend that lives 3 miles away. using our fan dipoles we can barely hear each other on 20/40 meters. The 2 antennas are broadside to each other so what would be the reasons we cant hear each other?
i have made plenty of contacts on my antenna on 20/40 so i know i am sending out a single..
when it comes to his A99 and my Imax i we can talk on 10 12 & 15 no problem..

and our antennas are 20 ish feet in the air..