There was a bug in the SSH service for a short time which prevented it from being used with slightly higher latency. But these have been fixed.

Can you see any error about rtty in System Logs when this happens?