Emails sent to Yahoo sit in our server que for hours before they are delivered.?

When I look at my qpsmtp log files, there is nothing indicating the status of the email. Communication between the two servers looks good, it just doesn't leave. I have not been able to find in the log file or trap when the email actually leaves. I some cases the email goes on the first try. In other cases the email sits in the que for hours or even a day before it leaves.