Internet-accessible SSH servers are often scanned for by automated malware (worms and botnets). Once a server is found, the malware may make a lot of login attempts with "common" usernames and passwords, in an effort to gain access by bruteforcing a password. Some old SSH versions may have known weaknesses: automated malware may attempt to exploit these weaknesses even if they are not applicable to your SSH version.
What is your Operating System? Which version? Is it up to date with patches?
What SSH implementation are you using (OpenSSH-based or commercial)? Which version?
Can you stop SSH for a while and see if the freeze happens again without it? Or restrict SSH traffic to legitimate endpoints only?