SoylentNews is people
SoylentNews
OpenSSH 8.8 has been released and with it comes a heads up that there will be major changes to how the scp utility operates, starting in one of the next releases. Specifically, scp has been retooled to use the SFTP protocol under the hood. This will leave most behavior unchanged and most times there will be no perceived difference. However, some scripts which make use of globbing might need minor adjustment to work properly in the future:
A near-future release of OpenSSH will switch scp(1) from using the legacy scp/rcp protocol to using SFTP by default.
Legacy scp/rcp performs wildcard expansion of remote filenames (e.g. "scp host:* .") through the remote shell. This has the side effect of requiring double quoting of shell meta-characters in file names included on scp(1) command-lines, otherwise they could be interpreted as shell commands on the remote side.
This creates one area of potential incompatibility: scp(1) when using the SFTP protocol no longer requires this finicky and brittle quoting, and attempts to use it may cause transfers to fail. We consider the removal of the need for double-quoting shell characters in file names to be a benefit and do not intend to introduce bug- compatibility for legacy scp/rcp in scp(1) when using the SFTP protocol.
Another area of potential incompatibility relates to the use of remote paths relative to other user's home directories, for example - "scp host:~user/file /tmp". The SFTP protocol has no native way to expand a ~user path. However, sftp-server(8) in OpenSSH 8.7 and later support a protocol extension "expand-path@openssh.com" to support this.
The new behavior is now present in scp but currently off by default. It can be tested using the temporary -s option. Later, the -O option will force use of the original scp/rcp protocol for the cases where SFTP may be unavailable or incompatible.
Compared to scp/rcp, SFTP is a new protocol but only relatively speaking. Importantly, it has been engineered from the ground up to operate as securely as possible. In contrast, scp has been written without a formal specification other than to operate like the late rcp did, but over SSH. Currently, scp requires expansion of glob patterns using the remote system's shell. That can be eliminated by dropping scp and switching to SFTP beneath it all.
Previously:
(2019) Oh, SSH, IT Please see this: Malicious Servers can fsck with your PC's Files During scp Slurps
(2018) OpenSSH SFTP Chroot Code Execution
(2014) OpenSSH No Longer has to Depend on OpenSSL
(Score: 0, Offtopic) by c0lo on Tuesday September 28 2021, @08:20AM (24 children)
What is to comment about it?
I can't even pretend I have something to be nostalgic in re scp, never had any adventure with it, not even a one-night stand or a pub crawling on its account.
Q: should we treat them differently, together with vulnerabilities/security announcements? Like this one [soylentnews.org], does it worth keeping the "story time slot" for its entirety? Its a piece of info, you read it then either "Ok, I need to do something about it" (unlikely) or "Meh... et alors?" (a lot likely)
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 5, Interesting) by maxwell demon on Tuesday September 28 2021, @08:41AM (3 children)
You could for example discuss whether you agree or disagree with the design decision. After all, breaking backwards compatibility is not to be taken lightly, and it surely would not have been hard to simply detect and remove extra quotes from file names, which should fix the majority of compatibility issues.
The Tao of math: The numbers you can count are not the real numbers.
(Score: 3, Interesting) by canopic jug on Tuesday September 28 2021, @09:07AM
Indeed. I gather that another option would have been just to pull the plug on scp(1) and let people deal with the fallout. That would have been interesting, to say the least but was probably never a serious option even if discussed. It would have had my vote though.
This looks like it might be a viable compromise keeping most of the interface the same while removing just the legacy protocol which was causing the problems. It's not something that has been done in haste. I recall people discussing the future demise of scp(1) more than 10 years ago and recommending a move to SFTP even then, though not specifically like this use of SFTP.
Money is not free speech. Elections should not be auctions.
(Score: 2, Interesting) by c0lo on Tuesday September 28 2021, @11:29AM
Honest to God, cross-your-heart... if I would not have posted my comment, would the topic of your example popped into your mind as worthy to comment on?
(I have this hunch you wouldn't, because you didn't consider important enough to open a special thread for it. It resembles more of a "Now that you mentioned it, how about..."
Of course I can't be sure and of course I'm not judgemental no matter your answer)
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 2) by FatPhil on Tuesday September 28 2021, @06:33PM
Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
(Score: 3, Insightful) by canopic jug on Tuesday September 28 2021, @08:51AM (9 children)
There's nothing nostalgic about scp(1). Good riddance. scp(1) has needed to go for a while so that system updates can propagate the needed changes though the world. The scp(1) problem has had a few decades to spread and become pervasive and passive clean up has not done the job. scp(1) was the go-to method for file transfer for a long time and all too many guides and tutorials, new and old, still persist in mentioning it instead of Rsync or SFTP. And even young people are thus prompted to turn to the less-than-optimal solution of scp(1) in their new scripts. Once the habit takes, it is hard to change. Now push has come to shove and the only way turns out to be to leave the user interface in place and swap out the underlying protocol.
The reason I wrote the post is that most remotely managed systems use SSH, usually OpenSSH at that. I'd say therefore that people here probably use SSH frequently in one capacity or another. Many may even have scripts, either their own scripts or ones that they have inherited. Some of the scripts may have scp(1) hidden in there instead of using SFTP or Rsync. Granted the various distros will take a long time to push out just these changes, especially with the long term support editions. But the changes affect two common annotations, so this is the kind of change that needs a lead time to deal with properly and all new file transfer scripts need to work with the new scp(1) or, better, directly with SFTP or Rsync.
Money is not free speech. Elections should not be auctions.
(Score: 0) by Anonymous Coward on Tuesday September 28 2021, @09:04AM (4 children)
I feel that there is some sort of message in your spelling of "scp" as "scp(1)", but I don't get it.
If it's an important message, could you please phrase it more clearly?
(Score: 2) by canopic jug on Tuesday September 28 2021, @09:10AM
No message, it is just the relevant section of the manual pages. It's the annotation that was used in the announcement but got omitted from the summary.
Money is not free speech. Elections should not be auctions.
(Score: 0) by Anonymous Coward on Tuesday September 28 2021, @09:16AM
It's meaningless unless you use *NIX at a terminal. Just a shorthand for those who might be reading the manuals.
(Score: 3, Informative) by Anonymous Coward on Tuesday September 28 2021, @09:47AM (1 child)
The "name(number)" is used in man pages to distinguish the man page called "name" in the section with that number from other pages called "name" that appear in other sections. In the case of "scp(1)" it is being used to specifically refer to the item that corresponds with the page called "scp" in section 1 "general commands." It is a quick way and unambiguous way to signify that you are talking about a program called "scp" and not the protocol called "scp" which would be denoted with "scp(7)" or "scp(5)" depending on the section system used.
(Score: 0) by Anonymous Coward on Tuesday September 28 2021, @10:43AM
Thank you.
(Score: 2) by maxwell demon on Tuesday September 28 2021, @11:10AM
Of course what is replaced is not the tool, but the protocol. The tool still exists, it just uses the sftp protocol underneath, and has a few incompatibilities to the current scp.
The Tao of math: The numbers you can count are not the real numbers.
(Score: 2) by c0lo on Tuesday September 28 2021, @11:19AM
Apologies, that was a tongue-in-cheek on which I forgot to (grin).
Which makes from it a "community announcement" - beware guys there are some changes that may bite your ass.
The best comment I can post of this kind of post is "Thank you for the tip" (if it helps me). I don't think that a whole page of "Thank you" comments worth the entire time slot reserved for a story.
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 0) by Anonymous Coward on Tuesday September 28 2021, @06:50PM (1 child)
scp is in muscle memory. And, it will continue to be after this back-end switch-- it is simple and handy for simple file transfers. For more complex transfers, I've always just run remote commands (e.g., tar, dump/restore, etc.) via ssh (not an interactive shell) and piped output/input. I think this is fairly common. And, folks doing this to handle the more complex stuff won't notice anything, from this change, either.
The way the openbsd team are handling this is great, though. For most folks, the change probably won't be noticed. If you are doing something tricky with scp you might get bit by this, but also nice how they are announcing this way ahead of time, and hiding old behavior behind a switch as an emergency escape for, "oh f@%$, my script is broke, how do I fix it *now*?". And, having a switch ahead of time to pre-test your scripts is even better.
They have been warning us for many years that this was their eventual plan. At work, most scripts that used scp were re-written to use sftp both in preparation for deprecation (10 yrs ?or more? they have been warning of deprecation of scp) and, also, so we can chroot the users easily.
There is one script that uses undocumented behavior of scp when combined with a forced command to override dest path+file that will need to be updated still. No way that I know of of forcing sftp (server-side) to only write to a single file at a single path regardless of what the client side specifies. I guess sftp into a chroot will have to suffice.
(Score: 0) by Anonymous Coward on Tuesday September 28 2021, @09:21PM
I tend to use the -e ssh option of rsync in most cases for those kind of processes, but, you can always tunnel other sorts of things between computers.
(Score: 4, Interesting) by janrinok on Tuesday September 28 2021, @10:34AM (2 children)
If you are not a programmer, administrator, or a frequent user of the command line it will probably not affect how you use your computer.
However, for those working in such roles this is an important issue. It is quite possible, as Canopic Jug has pointed out, that scripts that are in frequent use might suddenly stop working the way that we expect them to. I don't know what proportion of our community this story will affect currently but in the past it was certainly a significant chunk - so much so that we actually have the Sortware topic for such stories.
But not all stories interest everyone in our community. We have stretched the definitions to allow a relatively wide span of topics to be discussed, but this is precisely the sort of discussion that many enjoyed in the early days of the site. It is informative and, for some, might be an important topic if scp is used in their daily professional work where this might actually result in a company experiencing an unexpected problem.
(Score: 1, Offtopic) by c0lo on Tuesday September 28 2021, @11:11AM
I'm making quite a good living software-engineering new things (not jusr programming them). Not that doing so necessary require the use of ssh or scp, it has been quite a long time since I needed it. You see, the use of a SCM (e.g. git) is much more prevalent in the category of "File operations" and automatic deploy are more frequently done with specific tools.
I'm not saying those kind of stories don't have a good role. But I don't expect extensive comments on such a story, it's mostly "informational" in nature (this is why my use of "public community announcement").
I didn't suggest to eliminate them completely, just don't let them live alone on the top for a whole 3 hours or whatever the cadence of publishing new stories, I don't think they worth it.
Well, times change. The latest ones that I'm seeing, barely make 5 comments and it's not unusual to find a couple of "Frost piss" vandalism among those comments.
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 4, Informative) by fliptop on Tuesday September 28 2021, @11:48AM
I use scp every day, many times per day, to move files around between machines. I've been doing sysadmin since the late 90's and a lot of my legacy scripts use it for copying db dumps to the backup server. I use rsync too for static files and mail backups, but I like having a separate cron job for copying db dumps so the report of what happened doesn't get buried in a mile-long rsync report.
To me, stories like this are much appreciated.
Our Constitution was made only for a moral and religious people. It is wholly inadequate to the government of any other.
(Score: 5, Interesting) by PiMuNu on Tuesday September 28 2021, @10:58AM (4 children)
Quite often I read these sorts of stories and think "thanks, that was really useful, but I don't have a comment". So I see interesting stories going past with low comment tallies and hope that the eds don't think the story is worthless just because of the low tally. Makes me wish that I could mod against a story (but clearly such a system is open to abuse)
(Score: 1, Offtopic) by c0lo on Tuesday September 28 2021, @11:14AM (2 children)
My suggestion was: post them, don't wait to get community participation.
And most of the "charm" of S/N stay rather in commenting then on the stories themselves - letting them linger the whole time until the next scheduled (commenting favorable) FA is a sorta waste.
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 3, Interesting) by PiMuNu on Tuesday September 28 2021, @11:46AM (1 child)
> most of the "charm" of S/N stay rather in commenting then on the stories themselves
I agree, but there is some "charm" in weird little technical details (much like this one). Comments alone would lean the eds towards always posting another AGW/covid/etc story. That's fine once in a while, but it tends to feed the trolls too much. Maybe the eds already take that into account.
(Score: 2) by c0lo on Tuesday September 28 2021, @12:10PM
Show that you like it. I can form no idea if you say nothing, not even a "I like this story, thanks for it".
I liked this one [soylentnews.org] well enough to try to share it with whoever from soylenters would find it interesting. I have no idea if I should continue to do it or should I keep the delight for myself. Perhaps that single on-topic reaction may suggest I should continue posting from time to time such stories.
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 0) by Anonymous Coward on Tuesday September 28 2021, @04:08PM
I have nothing meaningful to say about both the story and the comment, but I agree with the comment, so am posting just to note that (and add one more comment to the "popularity content" of this story).
(Score: 1, Interesting) by Anonymous Coward on Wednesday September 29 2021, @07:41PM
For me, "zero days under active exploit, exponential growth in detections" is news. So is "compatability-breaking changes in a cli tool incoming."
In the physical world, my news stream includes local disaster news.
I live in a digital world, where local disaster news is somewhat broader. Yes, this is neighbourhood news - pileup on the old wooden bridge might force interstate bypasses lengthening trips by 20min! gasp! - but for the few people in those cars or on the way to critically timed jobs, it's news, and those of us on these forums are grepping our home and work scripts for scp, instead of checking names on a chyron for friends and relatives.
(Score: 3, Insightful) by DECbot on Tuesday October 05 2021, @11:38PM
I know I'm late to the discussion, but this is one of those stories that I appreciate but expect little discussion. Though today I'm delightfully surprised to see so much of it.
scp has stuck with me over the years despite its impending deprecation because the syntax is so like cp but with the relevant ssh bits tacked on to make the cp command transcend the system you're logged into. The syntax is obvious and I've never really needed to reference man pages or google the command to make sure I'm using the flags correctly for moving a single file or folder across the network. The article is a good reminder that I should spend more time familiarizing myself to alternatives like sftp and rsync. It is also interesting that some distros are changing the underlying functionality of the utility as opposed to removing it from the repo. This is good info to have when you're moving files from a 2012 circa server to a 2022 circa server and they're just not agree on how that is accomplished. As it's been said many times elsewhere, you'll see this a bunch in older bash scripts that still expect the vulnerable scp functionality. Instead of banging your head on the wall going "wtf! why is this not working?" you go "oh yeah, that fking changed," change the utility, and move on without wasting time troubleshooting why a deprecated utility isn't working.
cats~$ sudo chown -R us /home/base