Securing systems is a continuous battle for sysadmins. There's no such thing as secure but there are some things you can do to better protect your systems and your users. And, just because there is a global pandemic, security issues didn't go away or subside in the least in 2020. Security requires constant vigilance and reliance on developers and programmers to deliver secure code, security updates, and documentation on how to better protect our systems and users.
Since so many sysadmins work alone or in small groups, finding good security information and implementing it is difficult. Tight budgets and the demands of daily production add to that difficulty. Our aim in posting these tutorials is to help you to quickly identify trouble spots and to provide information that will help you in your quest to better secure your systems.
The following 13 posts were some of the most read security tutorials for 2020.
(Score: -1, Redundant) by Anonymous Coward on Friday January 01 2021, @04:56AM (14 children)
From SSH tutorial:
Where's the bit about running SSH on a different port other than port 22, at least externally?
That way at least you are less likely to be pwned by worms or automated ssh scans AND if you do see lots of failed log in attempts on your ssh server you might consider it a more targeted attack and take appropriate measures.
(Score: 0) by Anonymous Coward on Friday January 01 2021, @04:58AM (5 children)
(Score: 0) by Anonymous Coward on Friday January 01 2021, @04:07PM (4 children)
I always wondered if you used a 256-bit port number then the chance of somebody ever hitting it by accident is ~0. Then you don't need any password / authentication nonsense.
(Score: 2) by RS3 on Friday January 01 2021, @04:36PM (2 children)
That's an interesting idea, but how would you implement it? It seems that IP TCP and UDP headers are limited to 16 bits. https://en.wikipedia.org/wiki/User_Datagram_Protocol [wikipedia.org] https://en.wikipedia.org/wiki/Transmission_Control_Protocol [wikipedia.org] , etc.
(Score: 0) by Anonymous Coward on Saturday January 02 2021, @12:56PM
Reimplement it all over HTTP, that's how Google does it /s
(Score: 0) by Anonymous Coward on Saturday January 02 2021, @11:59PM
(Score: 1, Touché) by Anonymous Coward on Sunday January 03 2021, @12:02AM
(Score: 2, Informative) by Anonymous Coward on Friday January 01 2021, @06:01AM (7 children)
If you use public key and disable password auth, you are pretty safe-- openssh has a pretty good security record.
Changing the default port might reduce log spam from scripts trying to brute force passwords. But, you still get some hits which means you won't be safe if there is an exploit in your sshd.
Another fun one with dubious added security is to use haproxy, nginx, or sslh to mutiplex ssh with your webserver (and your vpn) on tcp 443. But, it is pretty easy to tell ssh is there unless you don't start talking ssh unless the client does first. In haproxy, it is easy to setup a rule to look for "SSH 2.0" in the first bytes of the packet from the client to verify the client is trying to talk ssh before handing traffic to the sshd backend. But, even giving up _less_ information with this check (foil nmap -A / nmap -sV), all it takes is a client to try ssh, and they will be able to do anything they could do if your sshd was on port 22. And, unless you setup transparent proxying, you won't have good logs in your sshd since sshd doesn't support proxy protocol.
You can also add a proxycmd on your client that wraps your ssh traffic with stunnel, and then use sni to select the ssh backend on the proxy with TLS terminated on the proxy. This one is pretty good as far as not being discoverable. More overhead though. And, without transparent proxy your sshd logs are worthless.
Port knocking (fwknopd) is yet another step up, and protects sshd while eliminating log spam. No add'l overhead, and no negative impact on logging.
I've used all the above, but, lately I leave the listener, in the open, on port 22, and scrape logs for these bots, and add the IPs to a blocklist. Blocking by IP is safe since they have to do a full tcp 3 way handshake, so they cannot be spoofing their src address. The only collateral damage is folks behind the same NAT address (like CGNAT). I use progressively longer blocking periods to reduce impacts on the innocent. I.e., first trigger gets a block for 10m, another abuse rule hit from even something that could be spoofed during that 10m period from the same IP extends it to 60m, 4r, 12hr, then all the way to 24hrs, and then anything that hits within the 24 hr period just resets the timer on the 24hr ban-- I've seen the same IPs in the blocklist for over a week from persistent bots. One rule I have that is really quick to see ssh abuse relies on the fact that a popular python ssh library (used by the majority of these bots) doesn't support more secure HMACs, so when I see an attempted connection which cannot negotiate the HMAC (ever since Snowden released that NSA slide, "ssh and openvpn are not a problem for the NSA" I harden allowed ciphers etc. on sshd), it is an instant ban without even a single successful connection from the bot.
Lots of extra things you can do. Some more effective than others. Some more of a pita to setup than others or come with other negatives. But, if you don't care about the log spam, you are probably safe just turning off password auth.
(Score: 2) by leon_the_cat on Friday January 01 2021, @06:08AM
After reading this i have gone back to using rsh.
(Score: 0) by Anonymous Coward on Friday January 01 2021, @07:57AM (1 child)
More info for the new year. Please. A step-by-step; not so much. But a little more info...
Thanks...
(Score: 1, Informative) by Anonymous Coward on Friday January 01 2021, @06:35PM
Not sure which you wanted more info on, but hopefully something in here is useful for you. Happy New Year!
.
Hardening ssh:
.
# change pref order
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
# DSA disabled by default now
.
# We generated more secure host keys
# rm /etc/ssh/ssh_host_*
# ssh-keygen -t rsa -b 4096 -f /etc/ssh/ssh_host_rsa_key -N ""
# ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key -N ""
# And, short (less than 3071) dh moduli removed with:
# mv /etc/ssh/moduli /etc/ssh/moduli.dist
# awk '$5 >= 3071' /etc/ssh/moduli.dist > /etc/ssh/moduli
KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org,diffie-hellman-group18-sha512,diffie-hellman-group16-sha512,diffie-hellman-group-exchange-sha256
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,umac-128-etm@openssh.com
.
(you probably want aes256-gcm first. The above is from a VPS where aes-ni extensions are being masked)
.
.
.
To use better encryption to protect the private keys on your client (the old / default encryption is weak):
.
Using ed25519 keys will automatically use the new more secure encryption for private keys:
ssh-keygen -t ed25519
.
If, for compatibility reasons, you need to stick to RSA, you can add the -o switch when you generate your rsa keypair, and ssh will use the newer more secure private key encryption.
.
You can upgrade existing keys with ssh-keygen -p -o -f privatekeyfilename
.
You shouldn't be using dsa (openssh has deprecated dsa due to insecurities). And, if you give credence to NSA conspiracy theories, maybe skip ec stuff besides ed25519. I'm kind of ambivilant on this, but exclusively used ed25519 since other ec curves didn't provide anything that this trusted curve does not, and on the off chance that the NSA did do something nefarious, I'm not using those back-doored curves. I don't think anyone has enough information to say for sure that any position on the official curves is right or wrong, but it isn't pure speculation either since the NSA is known to have done similar.
.
Switching to ed25519 will reduce the time it takes for ssh handshakes since the keysize is tiny, but supposedly equivalent to a massive RSA key.
.
.
.
And, here is the rule for matching ssh from a client in haproxy:
# first ssh packet body begins SSH-2.0 below matches
# payload at offset 0, and next 7 characters and looks
# for SSH-2.0 in hex.
acl ssh payload(0,7) -m bin 5353482d322e30
.
# We need to get the info about sni buffered, so we
# can do inspection. tcp-request content inspection
# accomplishes this.
# If no "tcp-request content" rules are matched,
# the default action already is "accept", so this
# doesn't change what is allowed / denied further on.
tcp-request content accept if { req_ssl_hello_type 1 }
.
# wait on client up to Ns for sni / ssh header
tcp-request inspect-delay 5s
.
# terminate_tls_http backend points to terminate_tls_http frontend
# also running on this haproxy instance.
use_backend bk_terminate_tls_http if { req_ssl_sni -i example.org }
use_backend bk_terminate_tls_http if { req_ssl_sni -i www.example.org }
# terminate_tls_tcp backend points to terminate_tls_tcp frontend
# also running on this haproxy instance.
# ssh tunneled in tls
use_backend bk_terminate_tls_tcp if { req_ssl_sni -i ssh.example.org }
# ocserv terminates tls for itself
use_backend bk_ocserv if { hdr_dom(host) -i vpn.example.org }
# bare ssh over 443
use_backend bk_ssh if ssh
.
# default reject, if nothing above matches
.
.
The above is excerpt of relevant bits of the tcp front-end. Because I also want to terminate tls on haproxy, this frontend loops back to another http front-end for web traffic and ssh-in-tls (a tcp front-end cannot terminate tls).
.
# just loops through to haproxy front-end, "terminate_tls_http", on this same instance via a unix socket
# note send-proxy, we maintain the original src ip throughout
backend bk_terminate_tls_http
mode tcp
server terminate_tls_http /run/haproxy/terminate_tls_http send-proxy
.
# terminate tls for backend web containers
# note, accept-proxy, we maintain the original src ip throughout
frontend terminate_tls_http
bind /run/haproxy/terminate_tls_http user haproxy group haproxy ssl crt-list /etc/haproxy/example-org-cert-sni-map accept-proxy
.
.
.
.
I still have obfuscated ssh including ssh in tls as additional options for trying to get around blocks when I travel, but ssh on 22 is also running in the open (I kinda like its honeypot properties-- I intend to add rules to make it so any IP that was blocked in the last year will be primed to be blocked again on a single transgression, even to a rule where ip spoofing is possible-- a hair trigger for ips with a bad reputation-- currently spoofable violations only get rate limited). I also have ocserv vpn which uses standard tls and dtls running on 443 tcp/udp also, for trying to get around dpi firewalls when traveling.
.
I used to run haproxy in transparent mode, but I never got it working right when I moved haproxy into a container. So now, I run a dedicated sshd container with cloudflare's mmproxy inside that container that haproxy sends the IP to via proxy protocol, so I get the real src ips in the sshd logs.
.
To get around the overhead of containers, I use an LD_PRELOAD hack with libksm_preload.so, to make the containers use kernel same page merging (memory deduplication). There are security implications for doing this, but in my case efficient use of limited memory is more important. I also do containers weird from what most folks do. My containers are host snapshots that have unnecessary to the container files deleted, other files customized, uid/gid ranges shifted, and the modified snapshot mounted readonly as rootfs for the container with a few writable bits mounted on top (all with quotas enabled). I have hooks into apt that schedule new snapshots after (automatic) updates on the main host, so container "patching" is automatic. And, the snapshot based containers use pretty close to no additional disk space over not using containers on my crappy VPSs.
.
If using nginx (I used to run nginx as a transparent proxy since it was also the webserver, to have less stuff running on the limited vps , but haproxy was better for this muliplex 443 proxy without giving up so much information, and I use haproxy as load balancers at work and nginx only as webservers at work, so making haproxy do what I wanted was easier _for me_ / _I think_ it is not possible with nginx):
.
The below needs to be in a stream block. You can do the same loopback over a unix socket trick to get nginx to also terminate tls / be its own back-end webserver. Sorry, I don't have those old configs to excerpt from.
.
I went with something like this (it will show up as sshd as the listener in a port scan though):
.
# sni for ssh = "". Anything, not explicitly listed (and not null), we blackhole.
map "$ssl_preread_server_name" $name_protocol_key {
"vpn.example.org" vpn;
"www.example.org" web;
"example.org" web;
"" ssh;
default blackhole;
}
.
(dots '.' added to get around SN spam filter that didn't care for the blank lines in this message)
(Score: 0) by Anonymous Coward on Friday January 01 2021, @04:23PM (1 child)
I never understood why "no password" is always tied up with setting SSH to be key auth, yet everywhere else people rage about 2FA. I use both on my ssh. First key, then password.
PubkeyAuthentication yes
PasswordAuthentication yes
AuthenticationMethods publickey,password
Okay, I do understand the no password cases for automation tasks.. but still.
(Score: 0) by Anonymous Coward on Friday January 01 2021, @07:10PM
What you do is fine, but, personally, I would find it hella annoying.
Your private key should have a _strong_ passphrase on it already. So, you already get "password protection". The difference is you can use ssh-agent, so it isn't annoying to use. And, because it isn't annoying to use, you can use a much stronger/longer e.g., 60+ character passphrase on your private key than you would for a password you had to enter on every connection. And, you can still use scripts to automate things without breaking out expect.
If someone was able to keylog your private key passphrase, they can keylog your extra password layer too. I guess if you ignore all advise and use agent forwarding to untrusted hosts, your way would be safer, but it is a contrived example since no one would ever use agent forwarding with untrusted hosts. Hell, I wouldn't use agent forwarding at all. It is trivial to setup a proxycmd to use netcat to achieve the same thing as agent forwarding through a jump host without any of the negatives of agent forwarding. And, with recent-ish versions of ssh, you don't have to add a proxycomand line for every host entry in your config, something like this will automatically route all host entries in your ~/.ssh/config through the jump host using netcat to forward things so you can avoid agent forwarding:
~/.ssh/config:
for arg in "$@"; do
Match !exec "nc -z -w 1 %h %p >/dev/null 2>&1 || nc -z -w 1 %h.example.com %p >/dev/null 2>&1" host 10.*.*.*,*.example.com,*.*.example.com,*.example.org *.*.example.org, !host jumphost.example.org,otherhost.example.com
ProxyJump example-jump
Host example-jump
HostName jumphost.example.org
# this host connection is auto routed via jumphost and use nc instead of agent forwarding
Host somehost.somesubdomain
HostName somehost.somesubdomain.example.com
# so will this host
Host someotherhost
HostName 10.1.1.1
# this host will bypass the jumphost
Host otherhost
Hostname otherhost.example.com
(Score: 2) by RS3 on Friday January 01 2021, @04:27PM
Awesome ideas, thanks. I've let port 22 stand, and used something similar to fail2ban, similar to what you've described, with very aggressive IP blocking. I forget the parameters, but maybe 3 incorrect guesses over a wide time-frame (days) and you're blocked.
Of course the attacks come from many IP addresses (botnets) but it's been kept under control.
I've also blocked wide IP address ranges- several countries where much of the attacks come from- in the router because the servers are US based and only need to serve US-based users.
Not sure if this would be annoying (I find many 2fa schemes to be annoying) but the guessing is using common dictionary words / usernames, so one could require the login name to be longish and not normal / simple names.
(Score: 0) by Anonymous Coward on Monday January 04 2021, @06:35AM
Thanks for the analysis.
(Score: -1, Spam) by Anonymous Coward on Friday January 01 2021, @09:12AM
"They're laughing at us!"
"You know you're evil."
"Repent!"
- TERRY A. DAVIS, THE BEST OF, ALBUM 1, SIDE 1
(Score: -1, Flamebait) by Anonymous Coward on Friday January 01 2021, @04:00PM (2 children)
Why only 13? When I need to do something in Linux I need to edit at least 20 different config files and sudo apt get a dozen dependencies some of which I have to compile myself after appropriate modification of the C++ source. If you're only reading 13 tutorials, you're not serious.
(Score: 0) by Anonymous Coward on Sunday January 03 2021, @06:38AM (1 child)
C++? Feh. You're not a serious user unless you're editing the compiled binary by hand.
(Score: 0) by Anonymous Coward on Monday January 04 2021, @11:34AM
pfft... editing compiled binaries is for n00bs you're not serious unless you're modifying running programs by editing their in-memory state