The best way to avoid giving a reference is to let your colleague, friend, or relative know that you are not comfortable in providing a reference and provide a diplomatic reason why. This works because you still retain the relationship and can continue to network with them.