SoylentNews is people
SoylentNews
Software engineer Chris Wellons writes about tar-pitting nefarious SSH probes. Anyone with a publicly-facing SSH server knows that it is probed from the moment it is turned on. Usually, the overwhelming majority of incoming connection attempts are malevolent in nature. There are several ways to deal with these attempts, one method is to drag out the response for as long as possible.
This program opens a socket and pretends to be an SSH server. However, it actually just ties up SSH clients with false promises indefinitely — or at least until the client eventually gives up. After cloning the repository, here’s how you can try it out for yourself (default port 2222):
[...] Your SSH client will hang there and wait for at least several days before finally giving up. Like a mammoth in the La Brea Tar Pits, it got itself stuck and can’t get itself out. As I write, my Internet-facing SSH tarpit currently has 27 clients trapped in it. A few of these have been connected for weeks. In one particular spike it had 1,378 clients trapped at once, lasting about 20 hours.
(Score: 0) by Anonymous Coward on Monday March 25 2019, @12:03PM (1 child)
As far as developers go, he's such an incredibly somber person that you have to watch your time going to his site. There's great fun and insight to be had checking-up on his projects and experiments from time to time. Check out his work on hash-functions after you spent some time with endlessh.
(Score: 2) by driverless on Tuesday March 26 2019, @10:36AM
In this case, unfortunately, it's relying on a quirk of badly-written software, I assume OpenSSH since that's the de facto standard. I've just connected with the SSH client I use and got.. hmm, closed the window, something like "Server sent excessive amounts of data without an SSH ID string", all in a fraction of a second. So it's an exploit of an OpenSSH (?) bug more than anything else.
(Score: 0) by Anonymous Coward on Monday March 25 2019, @12:52PM (6 children)
This is a neat trick, but I wonder who the cretins are doing the probing. Some organized group(s)? Buncha random individuals/script kiddies?
(Score: 1, Interesting) by Anonymous Coward on Monday March 25 2019, @01:14PM (1 child)
In the comments someone said to put "Tiananmen square" in the response, then they stop. I guess for New Zealand you can put that manifesto now.
(Score: 0) by Anonymous Coward on Monday March 25 2019, @05:07PM
Using the Chinese Firewall against them. Nice.
(Score: 0) by Anonymous Coward on Monday March 25 2019, @01:33PM (3 children)
Probably infected bots ran by people that never update their software.
(Score: 3, Funny) by DannyB on Monday March 25 2019, @01:55PM (1 child)
Non infected bots might do a better job. Especially if those bots can be distributed to many Windoze PCs without their owners' knowledge.
Vaccinate your bots!
When trying to solve a problem don't ask who suffers from the problem, ask who profits from the problem.
(Score: 0) by Anonymous Coward on Monday March 25 2019, @09:18PM
Don't tell our resident Genuine People Personality, it might crush its diodes.
(Score: 2) by maxwell demon on Monday March 25 2019, @07:20PM
There are people who don't update their bots? Shame on them!
The Tao of math: The numbers you can count are not the real numbers.
(Score: 3, Informative) by stormreaver on Monday March 25 2019, @01:14PM
This could easily lead to the host machine running out of file descriptors, so be sure to use only disposable machines. Before I firewalled off my ssh port, I was getting thousands of connection attempts per day, so it wouldn't have taken long for my machine to become unusable using this technique.
(Score: 0) by Anonymous Coward on Monday March 25 2019, @01:14PM (9 children)
I see minimal probes if I move my ash port to something other than 22, and instead of wasting a connection for a week, why not just add the remote to the firewall after a few minutes?
(Score: 2, Insightful) by Anonymous Coward on Monday March 25 2019, @01:30PM
Resources that the bot has to waste on your machine, can't be used to probe other machines on the net. The tarpit sends data to the bot that it needs to check before it can do attempts to login... this data never validates... so it will never login, but the connection stays alive.
(Score: 2) by NateMich on Monday March 25 2019, @05:56PM (7 children)
Make sure you use something that isn't also a bunch of 2's. I've seen so many customers servers getting slammed on 222, 2222, and 22222 that it isn't funny. There is almost zero advantage to a port based on 22.
Pick something truly random.
(Score: 5, Funny) by maxwell demon on Monday March 25 2019, @07:24PM (5 children)
I'd just use the port number written backwards. :-)
The Tao of math: The numbers you can count are not the real numbers.
(Score: 0) by Anonymous Coward on Tuesday March 26 2019, @03:43AM (4 children)
you use port 104?
commander data
(Score: 2) by maxwell demon on Tuesday March 26 2019, @05:30AM (3 children)
Off by 2.
The Tao of math: The numbers you can count are not the real numbers.
(Score: 2) by maxwell demon on Tuesday March 26 2019, @05:31AM (2 children)
Err … I shouldn't try to calculate while listening to the news …
The Tao of math: The numbers you can count are not the real numbers.
(Score: 2) by maxwell demon on Tuesday March 26 2019, @05:34AM (1 child)
I forgot to give the correct correction: Off by 5. And yes, this time I'm sure.
The Tao of math: The numbers you can count are not the real numbers.
(Score: 2) by maxwell demon on Tuesday March 26 2019, @05:35AM
Err … no, off by 7 -- off by 5 from my previous wrong result!
The Tao of math: The numbers you can count are not the real numbers.
(Score: 2) by driverless on Tuesday March 26 2019, @10:42AM
I moved my SSH to port 80, and I found I was getting thousands of probing attacks a day there.
(Score: 0) by Anonymous Coward on Monday March 25 2019, @01:22PM
What happens to the cheap-o router/firewall with all the hanging connections? Sounds like a good test for it. I wouldn't put the sshd behind a NAT port forward.
It might be nice to add the IP from the ones that don't hang to a drop all list. Those connections are the ones that are likely to be a real person.
(Score: 2, Interesting) by DannyB on Monday March 25 2019, @01:57PM (14 children)
So I assume that this cannot run along side a real SSL implementation on your system?
Eg, real SSL and tarpit SSL are mutually exclusive?
When trying to solve a problem don't ask who suffers from the problem, ask who profits from the problem.
(Score: 2, Informative) by Anonymous Coward on Monday March 25 2019, @02:25PM (2 children)
Literally has nothing to do with SSL. SSH is a tool for logging in remotely to your machine in order to control it. SSL is a protocol for validating the trustworthyness of a server and establishing an encrypted connection.
As for it living along side proper SSH, yes it should be fine. Just move SSH off to some random port that only you know and have the tarpit running on port 22.
There is an earlier comment about possibly running out of descriptors, this is true. You could run out of descriptors using this tool and thereby be unable to connect to your own machine even with SSH off on some other port. So exercise caution.
Best solution is to disable SSH password auth and switch over to using key based auth only. Do this and SSH will instantly deny anyone trying to connect who doesn't have the proper key to do so. But the tarpit is an amusing toy to mess with script kiddies.
I used to allow auth with any username and password combo, then have it pipe all input to /dev/null and pipe /dev/random to the offender. Fun times!
(Score: 2) by DannyB on Monday March 25 2019, @03:12PM
Yes, you're right.
Not enough caffeine yet. I meant SSH. I wrote SSL.
When trying to solve a problem don't ask who suffers from the problem, ask who profits from the problem.
(Score: 3, Interesting) by DannyB on Monday March 25 2019, @03:13PM
I also am aware that it is possible to have both SSH and SSL on the same port.
SSLH – Share A Same Port For HTTPS, SSH, And OpenVPN
https://www.ostechnix.com/sslh-share-port-https-ssh/ [ostechnix.com]
Applicative Protocol Multiplexer (e.g. share SSH and HTTPS on the same port)
https://github.com/yrutschle/sslh [github.com]
sslh – A ssl/ssh multiplexer
http://www.rutschle.net/tech/sslh/README.html [rutschle.net]
SSLH: A SSL/SSH Multiplexer for Linux
https://www.unixmen.com/sslh-a-sslssh-multiplexer-for-linux/ [unixmen.com]
(See also: Proxytunnel)
When trying to solve a problem don't ask who suffers from the problem, ask who profits from the problem.
(Score: 2) by FatPhil on Monday March 25 2019, @02:33PM (8 children)
My firewall NATs 6 machines, so I just have a bank of ports 222x, one forwarded to sshd on each machine, and so this tarpit on 22 would work for me - it could be forwarded to the unimportant RasPi that doesn't run any other services. However, I use fail2ban on all the machines anyway, so I'm mostly safe from idiotic probes, this is mostly a kind of vaccination to protect others by limitting the number of attacks that can take place on the population as a whole.
Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
(Score: 5, Interesting) by canopic jug on Monday March 25 2019, @02:45PM (1 child)
If you use fail2ban or sshguard, you could maybe use the triggered firewall rule to route the bot to an interal port running the tar pit. Normal connections would pass through to the SSH daemon, as expected, but offending IP addresses would get routed internally to somewhere else.
Money is not free speech. Elections should not be auctions.
(Score: 2) by DannyB on Monday March 25 2019, @03:17PM
I like that approach.
When trying to solve a problem don't ask who suffers from the problem, ask who profits from the problem.
(Score: 3, Insightful) by DannyB on Monday March 25 2019, @03:17PM (3 children)
Sorry I wrote SSL when I meant SSH.
For SSH, I have one forwarded to a non standard port in the 3xxx range. I could forward more as you describe.
I suppose then the standard SSH port could run the tarpit.
But it would be really cool to be able to mix the tarpit with a real SSH implementation if that were possible to allow a real login to proceed.
Or another approach: something like fail2ban could redirect certain 'ssh visitors' to the tarpit once it is clear that they are nothing but a brute force attack.
When trying to solve a problem don't ask who suffers from the problem, ask who profits from the problem.
(Score: 2) by FatPhil on Monday March 25 2019, @03:42PM (2 children)
I did plan to write a lightweight fail2ban replacement at some point (installed size half a meg? for something that just calls notify/poll and matches a few regexps and optionally calls some scripts, that sounds like bloat to me), I'll remember to consider that kind of option when I do.
Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
(Score: 2) by canopic jug on Monday March 25 2019, @03:50PM (1 child)
If your needs are not complex, then the replacement can be quite short. I have one which is less than 10 lines of AWK. Though it depends on probes making specific errors, it seems to get nearly all of them quickly. I should connect it to a tar pit maybe but would want to re-write the tar pit in perl so that I don't have to add python to the mix. It looks simple enough to try.
Money is not free speech. Elections should not be auctions.
(Score: 2) by RS3 on Monday March 25 2019, @06:22PM
Some years ago I had tried fail2ban and either had trouble getting it to work or just considered it too complicated. Too many years to remember.
Have any of you tried "pam shield"?
(Score: 3, Informative) by NateMich on Monday March 25 2019, @06:00PM (1 child)
Stop using port numbers with "22" in them. Those are targeting by scans almost as much as the actual ssh port, 22 is. Trust me. Be random.
(Score: 3, Informative) by maxwell demon on Monday March 25 2019, @07:37PM
So if you've got a reason to have a real sshd on port 22, it might be a good idea to tarpit all the other ports with "22" in their number?
The Tao of math: The numbers you can count are not the real numbers.
(Score: 2) by canopic jug on Monday March 25 2019, @02:36PM (1 child)
This for SSH, not SSL. As to runing both a normal SSH server and a tar pit on the same port, it might be possible to do some kind of multiplexing with sslh [github.com]. But I don't think so. Off the top of my head I'd have to say I'm not sure how it could work since both an attack and a legitimate log would be using the same start of the same protocol. Remember, this tar pit kicks in long before the encryption is even negotiated.
As mentioned by the AC, putting them on different ports definitely works.
Money is not free speech. Elections should not be auctions.
(Score: 2) by DannyB on Monday March 25 2019, @03:14PM
Sorry not enough caffeine when I wrote SSL when I meant SSH.
Yes it is possible to have SSL and SSH on the same port, see my reply a few up above.
When trying to solve a problem don't ask who suffers from the problem, ask who profits from the problem.
(Score: 1, Interesting) by Anonymous Coward on Monday March 25 2019, @02:10PM (1 child)
I always feel like going after the IPs doing these connections and wipe their systems, but apart from being illegal, the time spent to fight back against an endless flood of attackers won't give me pleasure with the prospect of wasting most of my life doing it. I do like everyone else and set up a proper firewall rule for SSH and an automated IP banner on malicious activity from the allowed subnet.. and get on with my life.
If they encountered more of these tarpits, they would run out of resources, but they would also adapt and implement a reasonable timeout, so while being a fun revenge it is not the global revenge we need ;)
(Score: 0) by Anonymous Coward on Monday March 25 2019, @02:52PM
I used to run a honeypot. These guys would leave big text files full of username/passwords and sites.
Bad news is most of the time they're coming from other hacked boxes.
(Score: 3, Funny) by All Your Lawn Are Belong To Us on Monday March 25 2019, @02:38PM (2 children)
This sig for rent.
(Score: 0) by Anonymous Coward on Monday March 25 2019, @05:20PM (1 child)
Are you likely to be eaten by a grue?
(Score: 2) by maxwell demon on Monday March 25 2019, @07:31PM
I didn't see any mention that it is pitch black. Of course it might be that I just couldn't see that text because it is pitch black …
The Tao of math: The numbers you can count are not the real numbers.
(Score: 5, Interesting) by ledow on Monday March 25 2019, @07:51PM (3 children)
Rather than waste my resources responding to obviously malicious communications, I just block them.
The way I do this is looked-down-upon for reasons I can't fathom.
I don't pretend it's secure. I claim that it stops random automated spam rather than specifically-targeted attacks, which are the main category most people receive.
I use portknocks.
Unless you hit, say, ports 1234, 2345, and 3456 of my IP *IN THAT ORDER* with nothing else in-between from your IP, then you suddenly get visibility of the SSH port 22. For 30 seconds, to initiate your connection.
Simultaneously, anyone else on any other IP, or without the correct knock, or just trying sequential/random ports stands an infinitesimal chance of even activating the port-knock, let alone realising that it actually did anything (port knocks can be any length, so you'd have to scan in-between each knock attempt within 30 seconds in order to see if it opened anything), gets absolute zip. Rather than "everyone can see you have port 22 open", you can literally slide into the SSH server without any other IP even getting a valid packet to it.
And yet, when *I* want access, I have apps on my phone that will connect, knock and SSH into my machine. Hell, in a pinch, I just download a client or "tap" certain ports with any standard util that opens a connection (e.g. nc, telnet, etc.) and then connect as normal.
Replay attacks? You can combat those (by using portknocks that are cryptographically time-dependent, etc.).
Network sniffing? So my port 22 opens to me for 30 seconds. You'd have to replay it that same knock (assume it's not dependent on the IP it came from or the current time), and that would grant you access to port 22 for 30 seconds... to do what? Hit the normal layers of security that you would have without portknocks (i.e. no passwords, public-key-only, etc. etc.)
Worse-case: You get to my port 22 for 30 seconds which would have been wide-open to the net anyway.
Since I put port-knocking (with Knockd) on my SSH-running servers over ten years ago, I literally have never seen the logs let anyone open the port except me. The closest was a "Stage 1, Stage 2, Fail" (i.e. they didn't get in, because they just happened to hit the first two knock-ports but the third was wrong).
Same yourself some time rather than "security through pathetic obscurity" (moving SSH to a non-standard port that can be discovered in 60 seconds with nmap), maintaining huge blocklists and tying up resources (e.g. fail2ban, etc.) or just plain "put port 22 out in the open". Deploy your standard software, implement port-knocking (apt-get install knockd and then edit /etc/knockd.conf to do whatever you like on whatever knock you like), and watch your logs fall silent while never denying YOU access.
People really look down on it for some reason, can't fathom why, but they accept the above as "solutions"? Next best thing to literally just only opening port 22 to your known external IP's (but let's hope with that that you never have to connect over another connection in an emergency, eh?)
(Score: 2) by dwilson on Monday March 25 2019, @08:38PM (1 child)
Probably because port-knocking looks and sounds needlessly complicated, given that the usual goal isn't to prevent unauthorized login attempts, but to prevent unauthorized logins.
Myself, I lock it down by only allowing pubkey authentication and only certain users to connect, a few other changes, and then stop worrying about it. Logfiles haven't shown a break-in yet.
stribika's ssh guide [github.io] is my usual go-to when setting up a new sshd instance.
- D
(Score: 3, Interesting) by ledow on Monday April 01 2019, @10:16AM
That's all fine... I have pub-key-only too. That's where your actual security is.
But if you open port 22, you get thousands of automated attempts against it, and testing of your SSHd for vulnerabilities constantly.
It only takes one remote hole, weak key, etc. to bring that down. And I'd rather my logs weren't filled with spammed access attempts except serious ones.
Learning to ignore / not check your security logs is a very bad habit to get into.
(Score: 0) by Anonymous Coward on Monday March 25 2019, @09:02PM
As I've mentioned above in another post. I use a whitelist to limit the problem. I know which subnet the authorized logins are going to come from, no reason to let the whole world open a connection. Having limited the scope of the problem, an automated log-monitoring system for banning unauthorized access is not going to be a demanding task.
I think port knocking is pretty cool, but I haven't got a use for it yet.
(Score: 0) by Anonymous Coward on Monday March 25 2019, @09:24PM
Create a fake user that has a name formed from a sequence of random characters and numbers. Set SSH to only allow logins from that user.
Assign that user the shell /bin/rbash Then, have real users immediately 'su' to their actual account.
'fail2ban' failed logins.
(Score: 0) by Anonymous Coward on Monday March 25 2019, @09:47PM
The article doesn't mention the TARPIT netfilter (iptables) target. It seems that it would solve a lot of the problems discussed, as only netfilter (in the kernel) has to track the connection, so less memory is needed than running a userland process, no file descriptors to run out of, etc. A process could listen to decide when to start the TARPIT (or use fail2ban, etc), and then just add a netfilter rule and let it do its thing.
(Score: 3, Interesting) by darkfeline on Tuesday March 26 2019, @04:10AM (2 children)
What's the point of this? I hope everyone realizes that these probes are all bots. No one's time is wasted except your own setting up the honeypot. The client sleeping on the socket read is using the same amount of electricity as your server sleeping on the socket write, and it's probably not even the attacker's electricity, but some poor grandma's couple of cents of electricity, whose laptop is part of a botnet. Do you think the attacker cares if ten of his billions of SSH processes across all of his bots are hanging for a day? He probably doesn't even know it's happening.
This is like trying to get revenge against Trump/ by spending a hour of your time thinking their name really hard.
Join the SDF Public Access UNIX System today!
(Score: 0) by Anonymous Coward on Tuesday March 26 2019, @05:16AM
WRONG! People (witches) who cast spells on the President need to be Investigated. Lock Her up!
(Score: 0) by Anonymous Coward on Tuesday March 26 2019, @09:49AM
I have no idea looks like a small tutorial on threads, polls and sockets?