Researchers have demonstrated they can read what people are typing during video calls by looking at shoulder movements.
This horrifying Zoom hack will deter you from ever side-chatting again:
"From a high-level perspective, this is a concern, which obviously has been overlooked for a while," says University of Texas assistant professor of computer science Murtuza Jadliwala, who led the research, examining what could happen if your video meeting were hacked. "And actually, to be really frank, we didn't start this work for COVID-19. This took a year. . . . But we started realizing in COVID-19, when everything [is in video chat], the importance of such an attack is amplified."
As Jadliwala explains, the core problem is that our face-to-face video streams are presented in high fidelity, and their pixels convey more information than we realize. Without using any special machine learning or artificial intelligence techniques, Jadliwala's team figured out how to read the subtle pixel shifts around someone's shoulders to make out their basic cardinal movements: north, south, east, and west.
Applied to a keyboard, these four directions actually mean a lot. If you are typing "cat," you start with the C, move west to the A, then back east to the T. Once researchers figured out how to read these directions through shoulder movements, they were able to create software that could cross-reference them with what they call "word profiles" built with an English dictionary, which turned the maze of directions into meaningful words.
[...] In a lab setting, with a certain chair, keyboard, and webcam—while testing a limited pool of words—the average accuracy of the software was 75%. When the team tested subjects working from home in uncontrolled setups (they were asked to visit any websites, write emails, and enter their passwords), accuracy dropped significantly. The team was able to reverse-engineer 66% of the websites visited, but only 21% of random English words, and about 18% of the passwords typed.
Also at
Tech Xplore
Distinct Today
Daily Mail
Fast Company
(Score: 4, Insightful) by The Mighty Buzzard on Thursday November 05 2020, @10:11PM (4 children)
They must have only tested with horny chicks (huntin-peckers). My shoulders don't move even close to enough for a camera to pick up unless I need to slide my right hand over to hit something right of the Enter key or use the mouse.
My rights don't end where your fear begins.
(Score: 2) by Freeman on Thursday November 05 2020, @10:28PM
Then there's people like my Dad, who makes extensive use of the Speech-to-Text feature of his iPhone. Admittedly, most people can talk faster than they type, especially on a mobile device.
Joshua 1:9 "Be strong and of a good courage; be not afraid, neither be thou dismayed: for the Lord thy God is with thee"
(Score: 2) by coolgopher on Thursday November 05 2020, @10:28PM (2 children)
Yeah I was gonna say the same. I don't type with my shoulders, the furthest you can see the wrist movements is the elbow joint.
(Score: 2) by coolgopher on Thursday November 05 2020, @10:31PM (1 child)
And even if they could pick up on that, have fun filtering out all the vim commands from that...
(Score: 0) by Anonymous Coward on Friday November 06 2020, @08:56AM
FTFY
(Score: 2) by Freeman on Thursday November 05 2020, @10:12PM
Seems like it's a bit of a stretch, but 18% of passwords typed, isn't nothing. Still, they have to know what site you're visiting, etc. The real world use for something like this, isn't likely to be very high. Yet, definitely something to think about, if you're streaming, posting on YouTube, etc.
Joshua 1:9 "Be strong and of a good courage; be not afraid, neither be thou dismayed: for the Lord thy God is with thee"
(Score: 2) by MostCynical on Thursday November 05 2020, @10:24PM (3 children)
so just use a lower res (and far cheaper!) camera.. or, if the 'system' insists on hi res camera to 'work', just make sure your camera is pointing a bit higher, with a light behind you, or have it closer focused on your face.. (note, this may freak out others on the call - try not to have the camera pointed up your nose..)
"I guess once you start doubting, there's no end to it." -Batou, Ghost in the Shell: Stand Alone Complex
(Score: 0) by Anonymous Coward on Thursday November 05 2020, @11:01PM (1 child)
Why the light behind you?
(Score: 2) by MostCynical on Friday November 06 2020, @12:13AM
you get a 'person shaped like you' with a bright area behind.. people can see you, but you're a bit darker - twilight effect.. hides lots of facial expressions..
OR just do what one of my colleagues does - camera shows from the top of his ears up, and he wears a baseball cap..
"I guess once you start doubting, there's no end to it." -Batou, Ghost in the Shell: Stand Alone Complex
(Score: 2) by Mykl on Friday November 06 2020, @01:50AM
I wonder if using virtual backgrounds obscures things enough to prevent the read?
(Score: 3, Insightful) by Freeman on Thursday November 05 2020, @10:36PM (2 children)
I looked at the paper: https://arxiv.org/pdf/2010.12078 [arxiv.org]
So, easier to detect hunt-and-peck, but touch-typing could also be deciphered.
Joshua 1:9 "Be strong and of a good courage; be not afraid, neither be thou dismayed: for the Lord thy God is with thee"
(Score: 2) by The Mighty Buzzard on Friday November 06 2020, @02:42AM (1 child)
Tiny-handed people then? My hands are big enough that I can comfortably hit F12 while my index finger is still on the J key. Ins and Del are doable as well but not particularly comfortably.
My rights don't end where your fear begins.
(Score: 2) by Runaway1956 on Friday November 06 2020, @04:22AM
Medium sized hands here. I've used keyboards that were small enough that I could reach far enough to hit F12. I've never liked them very much. I prefer a larger keyboard. I have to shift my hands and reach for all function keys, number pad, esc, backspace, grave key and the += key. I suppose that people with smaller hands might prefer smaller, more compact keyboards, or they'll have to move their hands around just to reach the number keys.
I guess the point is, not one size fits all when it comes to keyboards. If your keyboard isn't a perfect fit, you'll probably give away a lot of information when under surveillance.
Failed to mention that I'm a touch typist. Not especially fast, but I seldom need to look at the keyboard to find where I'm at.
“I have become friends with many school shooters” - Tampon Tim Walz
(Score: 0) by Anonymous Coward on Thursday November 05 2020, @10:54PM
They better not have reflective glasses or someone can read their cards by looking at the glasses? Yet I thought I've seen them wearing reflective glasses on one of those poker T.V. shows?
(Score: 1, Funny) by Anonymous Coward on Friday November 06 2020, @12:19AM
p-o-r-n-h...
(Score: 2) by jb on Friday November 06 2020, @01:35AM (3 children)
I wonder whether using a Dvorak keyboard would work around this vulnerability ... I seem to recall that reducing the requirement for lateral movement was one of the explicit design goals.
(Score: 0) by Anonymous Coward on Friday November 06 2020, @06:36AM
I mean, if you are using any keyboard layout, other than qwerty in a non - research setting, the attacker will have a real hard time figuring that out. So personally, as a touch typist with a self - made keyboard layout on an ergonomic split keyboard, I don't think I need to worry :-D
(Score: 0) by Anonymous Coward on Friday November 06 2020, @11:59AM (1 child)
You are missing the point. You are proposing security by obscurity. It doesn't work. It may throw someone off course for a bit, but then they will adjust and review what they had before and done.
The point of this vulnerability is people will know what you are typing by just observing your body without seeing your hands. They can probably do the same by sound analysis too -- each keystroke sounds different than another. Or by reflection from your glasses or whatever.
(Score: 0) by Anonymous Coward on Friday November 06 2020, @07:59PM
Would there be any obscurity? Don't most dvorak typists proclaim far and wide at every opportunity how superior dvorak is and how much more awesome their life is since they made the switch?
So it may work, if only they could keep mum about it ...
(Score: 0) by Anonymous Coward on Friday November 06 2020, @07:05PM
and
So what do they think machine learning is?