CISCO ASA 5520 & 6513 sometimes does not respond via inside vlan 1 ip

I am facing problem when trying to pinging, telnetting or http accessing cisco asa inside ip address and 6500 swt vlan 1 ip address. At peak hours. response break from these ip maximum happens with cisco asa inside ip address. During that time internet slows down.

configuration is simple only ip any any policy is aplied on cisco asa 5520. And on cisco 6513 smtp and tftp access policy is applied. Perhaps, It happens sporadically. It hampers internet access for higher authority. Someone help me this to mitigate this issue.