There are various ways in which access can be restricted to sshd. By restricting which users and hosts/networks that can log in, the impact of stolen credentials may be minimized.
You can restrict which hosts/networks can access sshd via proper firewalling. Linux provides firewall software called "iptables". What follows is an example of allowing just 1 particular IPv4 address to connect to sshd while denying all others:
# iptables -I INPUT -p tcp -s 126.96.36.199 --dport 22 -j ACCEPT
# iptables -A INPUT -p tcp --dport 22 -j DROP
iptables : this is the iptables utility
-I : insert a rule...
INPUT : ...into the INPUT table
-p : specify the protcol...
tcp : ...as the TCP protocol
-s : specify the source IP address...
188.8.131.52 : ...as 184.108.40.206
--dport : specify the destination port...
22 : ...as 22
-j : jump to...
ACCEPT : the ACCEPT target