SoylentNews is people
SoylentNews
from the just-use-a-photocopier-multiple-times dept.
Google has developed and open-sourced a new JPEG algorithm that reduces file size by about 35 percent—or alternatively, image quality can be significantly improved while keeping file size constant. Importantly, and unlike some of its other efforts in image compression (WebP, WebM), Google's new JPEGs are completely compatible with existing browsers, devices, photo editing apps, and the JPEG standard.
The new JPEG encoder is called Guetzli, which is Swiss German for cookie (the project was led by Google Research's Zurich office). Don't pay too much attention to the name: after extensive analysis, I can't find anything in the Github repository related to cookies or indeed any other baked good.
There are numerous ways of tweaking JPEG image quality and file size, but Guetzli focuses on the quantization stage of compression. Put simply, quantization is a process that tries to reduce a large amount of disordered data, which is hard to compress, into ordered data, which is very easy to compress. In JPEG encoding, this process usually reduces gentle colour gradients to single blocks of colour and often obliterates small details entirely.
The difficult bit is finding a balance between removing detail, and keeping file size down. Every lossy encoder (libjpeg, x264, lame) does it differently.
(Score: 1, Insightful) by Justin Case on Sunday March 19 2017, @10:23PM (26 children)
For the love of FSM, why is JPEG compression still a thing in this millennium?
PNG is more than two decades old now, provides flawless (lossless) compression, and is not patent-encumbered. I'm not aware of any technical reason why JPEG is better, leaving momentum and ignorance as possible explanations.
Or maybe: camera makers like to brag about their billions of pixels while compressing the images so horribly that you can't see sensor defects any more -- or perhaps even see that the original image is nowhere near the advertised specs.
Give me a camera that converts the picture straight to PNG instead of shitting all over my photos. Thank you.
(Score: 0) by Anonymous Coward on Sunday March 19 2017, @10:41PM (10 children)
Uhhh... no? Imagine using lossless PNG's instead of JPEG's that have been compressed into oblivion. The load times will be much higher and bandwidth costs will be crippling. Compare a PNG and a JPG that have both been compressed to the same size ... the PNG will be essentially garbage.
(Score: 3, Interesting) by AthanasiusKircher on Sunday March 19 2017, @11:02PM (7 children)
The load times will be much higher and bandwidth costs will be crippling.
In an era where most people are streaming HQ video all the time, and even a lot of ads are video-based, this argument is specious. Bandwidth is eaten up a lot more by that nonsense than it would be by slightly larger static image files.
And in any case, I don't think this was GP's point -- obviously the possibility of lossy compression should exist for specific applications (like website optimization), but why would you want it to be saved as the default for your raw image when you're taking a photo to begin with? All it does is set in motion generational degradation from the start if you want to do any further processing of the image.
(Score: 1, Insightful) by Anonymous Coward on Monday March 20 2017, @01:12AM
In an era where most people are streaming HQ video all the time
Not even remotely true, even if you're just looking at a tiny part of the world (USA).
(Score: 0) by Anonymous Coward on Monday March 20 2017, @02:23AM (1 child)
Not all websites are serving HD videos. Compression is not done for the client, it is done for the server. A compressed file of 50% decrease in size can mean total number of users going up by 200%.
(Score: 2) by AthanasiusKircher on Monday March 20 2017, @03:51AM
Yes, I get all that, which is why I specifically said in my post that good (and lossy, if necessary) compression should exist FOR SPECIFIC APPLICATIONS, like web optimization.
The question I asked, yet again (and which I think the OP was concerned with) is why lossy JPEG is still default for a lot of devices like cameras that create this media in the first place.
(Score: 3, Interesting) by dry on Monday March 20 2017, @03:12AM (1 child)
We don't all have endless bandwidth, at home I connect at 26.4 kb/s and I'm not that far out of town. Data is also very expensive here in Canada, actually makes the States look cheap. On my pay as you go account it is a dollar for 20 mbs or about half a dozen web pages based on the average page being about 3MBs. Once they finish building a cell tower to serve my area and finish shutting off the dial up connection (officially died last Nov 16th), the best deal will be $85 for 10 GBs + a nickel a MB for overages. I won't be streaming HQ videos and every bit of bandwidth savings will be good.
To answer another of your comments, some of us don't have expensive devices with tons of memory and at least having the choice of saving in JPEG so pictures only take up about 2MBs can be good. If I want to shoot high res pictures, it won't be on my phone with its crappy little lens with palm prints all over it.
(Score: 3, Interesting) by AthanasiusKircher on Monday March 20 2017, @04:02AM
Choice is good. No argument there. At no point am I advocating that lossy JPEG should be abolished completely. But its ubiquity as default, particularly in use for camera devices, is weird these days.
And actually, regarding phone as camera, I have a close friend who is actually a kinda artsy photographer and recently gave up on buying standalone cameras, instead investing in a set of lenses for his phone. Cameras in phones have gotten really good in the past few years.
Also, regarding memory, the problem isn't generally the expense of the devices, but the ridiculous restrictions of some manufacturers of devices. Micro SSD cards are incredibly cheap these days, able to store thousands of pretty HQ photos for a pittance. The problem isn't the cost of memory, but the manufacturers that refuse to allow SSD upgrade options easily -- instead charging $100 upgrade cost for $5 worth of extra memory or whatever.
I presented my argument in an unclear way at first -- what I should have said up-front (and which I did say in my post) is that I have no problem with web optimization of images for bandwidth reasons. I think it's a good thing. But I do think that (1) many people (though certainly not all) waste huge amounts of bandwidth -- both serverside and clientside -- and for them the small optimization from images can be irrelevant, and (2) defaulting to lossy formats at the point of TAKING photos is what I find most baffling, given that it will degrade your ultimate product if you need to do further editing, conversion, etc.
(Score: 0) by Anonymous Coward on Monday March 20 2017, @01:51PM
If I stream a HD video, it is my decision, and I'll do that decision based on the current connection. If I decide that at my current available bandwidth/data I cannot afford streaming HD I won't do it (indeed, likely I won't stream video at all). When I'm using my fat DSL at home, I don't mind HD streams, nor big images. However I have little control over the bandwidth used by the images embedded into a web page.
That's what ad blockers and related technologies are for (or rather, against).
A professional photographer will use a camera that stores raw images. Note that PNGs aren't raw images either, as even the step going from sensor data to a viewable image consists of considerable processing (for example note that unless you've got a really expensive camera, on your camera there's only one physical colour channel for each pixel, while all the other channels are interpolated from the surrounding pixels; cameras which have full colour on all pixels are expensive because of the additional optics and extra CCDs required).
(Score: 2) by urza9814 on Tuesday March 21 2017, @09:10PM
Believe it or not, there are still people in this world who can't do that. My parents among them. It literally takes at least ten minutes to load CNN.com on their internet -- and usually the page won't even load entirely -- so they sure as hell aren't streaming Netflix all day long.
Man, I was shooting in raw on a $200 point and shoot back in 2007, and they've only gotten cheaper and more advanced since. These days any Android phone can do it, and I assume iPhones too. And AFAIK damn near every DSLR has had that feature from the beginning. If your device doesn't let you change the file format, the problem isn't the format, it's the device. But in most cases the photos taken on the lower end devices aren't being printed and framed, they're being texted or emailed or posted to Facebook -- the exact situations where file size matters. So slightly lossy jpeg is still a sane default. If you're trying to do higher quality photography and are looking to get top quality shots, you either know enough to change the file format or you don't know enough for it to really matter yet.
(Score: 2, Redundant) by Justin Case on Sunday March 19 2017, @11:18PM (1 child)
Compare a PNG and a JPG that have both been compressed to the same size ... the PNG will be essentially garbage.
Ummm... I don't think you have fully grasped the concept of lossless compression.
(Score: 1, Insightful) by Anonymous Coward on Monday March 20 2017, @12:27AM
He's saying that a 50KB JPEG will look a hell of a lot better than a 50KB PNG of the same image, in general. (Say, a 480p original image with 16-bit or higher color depth.)
Try a little experiment for yourself and see!
(Score: 0) by Anonymous Coward on Sunday March 19 2017, @10:42PM
For the love of FSM, why is JPEG compression still a thing in this millennium?
Cell phones.
Next question?
(Score: 3, Insightful) by turgid on Sunday March 19 2017, @10:47PM
And give me an end to the Loudness Wars so that CDs are once again CD-quality.
I refuse to engage in a battle of wits with an unarmed opponent [wikipedia.org].
(Score: 3, Interesting) by AthanasiusKircher on Sunday March 19 2017, @10:55PM (3 children)
I agree with the need for lossless algorithms to be the standard choice.
What baffles me even more is the obsession with using compressed JPEGs in an era of data bloat and excess. In just about every other electronic situation, we see data bloating more and more. From the size of basic desktop applications to the fact that more and more people have entire archives of audio and VIDEOS, as well as worrying about whether their internet speed can handle HQ streaming, why are we still compressing lossy JPEGs on a standard basis?
I'm sure there are still plenty of applications where lossy image compression is still desirable, but unless you're storing millions of photos, why would you want this to be your default? For a typical person's personal photo archive, store better photos in a lossless format and just delete one or two of your HQ downloaded videos or something... and you'll probably have the same amount of space.
(Score: 0) by Anonymous Coward on Monday March 20 2017, @12:29AM
Porn. Encrypted volumes. DVD backups.
Next question.
(Score: 1, Insightful) by Anonymous Coward on Monday March 20 2017, @02:40AM
Dunno, so photographic images on the web download instantly (or without taking excessive bandwidth and time over mobile)?
JPEG is efficient, it is good enough for many requirements, widely adopted and will be here as long as a technological society is. This means it's obsolete next week if the authoritarian left or religious right get their way!
(Score: 2) by urza9814 on Tuesday March 21 2017, @09:21PM
For most people, their "personal photo archive" is Facebook. If you're a professional (or even amateur) photographer, you want those high quality originals, and you ought to know how to change the file format settings on your camera, which any moderately decent camera will have these days (including smartphones). For everyone else, they don't want some massive photo that's too big to email, that's gonna take forever to upload to Facebook, that's going to chew through their mobile data trying to share it with their friends. They don't want to save it to their personal gallery and then convert it and then upload that, they don't even want to HAVE a personal gallery, they want it to go straight from the phone to the cloud. Often over spotty, lower speed mobile networks. And often dealing with email systems that limit attachments to 10-20MB total (while gets you 2-4 photos from a modern smartphone, even with jpeg compression). So if you increase that file size I expect FAR more people will be pissed that they can't share their photos as easily than will be happy about or even *notice* any increased quality. The few who do really want the higher quality can just press a couple buttons to change it.
(Score: 3, Insightful) by wonkey_monkey on Sunday March 19 2017, @11:28PM (2 children)
For the love of FSM, why is JPEG compression still a thing in this millennium?
Err, file size? You might be happy if every site takes 10 times longer to load its images, but I wouldn't be.
Give me a camera that converts the picture straight to PNG instead of shitting all over my photos.
There have been cameras capable of saving losslessly compressed images for decades. Maybe you should get one of those.
systemd is Roko's Basilisk
(Score: 2) by Wootery on Monday March 20 2017, @12:03PM (1 child)
Is there a good reason no-one uses modern image compression algorithms and JavaScript decoders? It's not like it's impossible -- even video decoding can be done in JavaScript. [brionv.com]
I imagine sites like Flickr and Imgur could save countless terabytes of bandwidth if they did.
(Score: 0) by Anonymous Coward on Monday March 20 2017, @02:02PM
JavaScript decoders? You like maxing out your CPU, don't you? Note that on your phone, video is typically not even decoded by machine code running on the CPU, but on specialized hardware included for just that purpose. Guess why.
Not to mention that there are already too many sites which require JavaScript …
(Score: 2) by looorg on Sunday March 19 2017, @11:50PM
Because size still matters? JPEG is better for one reason only as far as I know and that reason is File Size. It's all about the size. A small file size and still producing an okay result. A compressed PNG is still going to be many times larger then a compressed JPEG. Unless you use some really wonky settings people won't be able to tell anyway. But people would be able to tell if every time loaded something if the time spent was 5-10x longer then before. People would notice that. The quality is just good enough for what you want and with more and more people getting their "web-experience" via a smartphone or similar size matters more then ever. If you compare them fullscreen to fullscreen then sure people could probably tell the quality difference. But the image file-size-wise is going to be an enormous difference. In a TB-disk world it might not matter all that much for a home computer (enough disk space, enough ram and enough CPU so most people won't notice - but on smaller devices they will notice) but for an overall experience with surfing the web on various devices it would utterly suck if it went all PNG.
Cameras? Those have been around for ages. Just look for one that saves images in RAW. Filesize of a raw image is massive tho compared to a JPEG image - about 5-6x as large - but you'll get all the detail goodness you could want. It's just that they are in a completely different price-range from what normal people buy these days. Do people even buy dedicated cameras anymore or are most of them just fine with using their phone as a camera? I don't know. But RAW cameras exist have have for many years.
(Score: 0) by Anonymous Coward on Monday March 20 2017, @02:20AM (1 child)
Because the Flying Spaghetti Monster demands all feature films are encoded with 12bit JPEG 2000 for theatrical release?
JPEG is better for delivery because PNG was designed as a (patent encumbered) GIF replacement for use with illustrations and graphics. PNG was never designed to replace JPEG for photographic images -- even if it is widely used in production workflows.
I compiled this google tool earlier in the week. If I am compressing images for the web, I will use this compressor. My brief test (from a a 1080p still PNG ~1.9M) was 340k for this compressor Vs. 304k for imagemagick convert (both default settings: im 92%, google 95%). When I reduced the quality for the Google compressor down to 85% (fine for the web) the image size was 181k. In my brief test, the compressor was visually lossless at 1:1 (on my image) whereas I could already see artifacts in this particular IM conversion.
Would that be 8 or 16 PNG as opposed to 14 bit raw (which most cameras are capable of) and are you aware of the differences (over 8bit JPEG) in storage requirements?
Why not build Google's new compressor and do the tests for yourself before commenting? You do not appear to understand this compressor is for delivery, click a news item and it could deliver a more pleasing image faster and at lower bandwidth cost to the user. This at the expense of affordable CPU time to the distributor.
(Score: 2) by maxwell demon on Monday March 20 2017, @08:19PM
You are aware that JPEG 2000 [wikipedia.org] is not the same format as JPEG, and also offers lossless compression?
The Tao of math: The numbers you can count are not the real numbers.
(Score: 0) by Anonymous Coward on Monday March 20 2017, @04:49AM
Easy: filesize. Disk is cheap and getting cheaper all the time, but it's neither free nor infinite, and PNG compression gets awful rates with photos with the quality difference not mattering to most people.
JPEGs are fairly small, and that matters a bunch to more than 90% of people taking pictures with their camera (so they don't have to delete pictures off the card nearly as frequently). It's not the most relevant thing in the age of cheap 64GB microSD storage, but it still matters.
If the quality loss matters (read: invest in a camera that isn't cheaply made consumer-grade crap), many cameras do offer to save in a raw or uncompressed format.
Lazy, single datapoint: resaving a 5.36MB PNG image made it go down to 2.8MB as a JPG file at "100" quality.
On a camera, that's literally double the pictures you can save, and most people don't care or notice the quality drop.
On the web, that's literally half the bandwidth cost (and time spent downloading -- it'll be quick on modern connections, but downloading several 5MB images will take a noticeable amount of time for most people still), and generally you'd end up dropping the quality lower to 85 (875k for this image) or 70 (640k for this image) on the web anyway (or even lower, although fairly artifacted at this point, at quality 30, it's still fairly clear, and now only 360kB).
in addition, JPEG hasn't actually been beaten in quality to filesize enough to matter by other lossy formats (admittedly, mostly because of work like this due to JPEG being super-common and widely supported rather than it being some kind of superior lossy image format; "enough to matter" is a consideration simply because JPEG is the only widely supported lossy image format and you'd need a fairly big improvement to get people to switch)
(Score: 2) by Marand on Monday March 20 2017, @07:11AM
For the love of FSM, why is JPEG compression still a thing in this millennium?
PNG is more than two decades old now, provides flawless (lossless) compression, and is not patent-encumbered. I'm not aware of any technical reason why JPEG is better, leaving momentum and ignorance as possible explanations.
Because sometimes you don't need lossless compression, so the trade-off of increased file size isn't worth it. Bandwidth and storage concerns are "still a thing in this millennium" as you put it. PNG is awesome at what it does, and I often store images as PNG locally so I have a lossless version, but when I want to transfer one of those images to someone else I convert to JPEG unless the other person will need lossless as well.
News flash, this is also why we resize images before distribution as well. When I take a photo I might keep a lossless 10+ megapixel image (usually RAW) for myself, but when I'm showing it to someone else I resize it and convert to JPEG because they don't need the giant lossless file, so why waste time and bandwidth?
Give me a camera that converts the picture straight to PNG instead of shitting all over my photos. Thank you.
Get a camera that can save RAW instead of whatever piece of shit you've got that can't, it's a better way to store the original data.
(Score: 2) by shortscreen on Monday March 20 2017, @08:50AM
1) JPEG is plenty old enough to not be patent-encumbered anymore
2) PNG's compression algorithm is "run it through PKZip." Not terribly innovative or ideal.
3) It's true that camera makers like to brag about pixels, but your allegation that the images are over-compressed is baloney. The fact is, camera makers are counting red, green, and blue pixels separately. So one red, one green, one blue... they count that as 3 pixels, even though on your monitor it is only one pixel. A 16MP camera outputs an image file that is 5000x3333 or so, but in reality 2/3rds of that data wasn't even captured, it had to be interpolated because each pixel on the camera sensor only sees one of the three color channels.
(Score: 3, Insightful) by Runaway1956 on Sunday March 19 2017, @10:45PM (4 children)
For every individual or corporation looking for a way to save bandwidth, there are a hundred more searching for ways to waste bandwidth. It's an arms race. At the same time, for every gain in available bandwidth, more stuff is forced into the line, squandering whatever is available. It's a never ending refrain, "We need more bandwidth for better hi-def porn!" Meanwhile, advertisers believe that all that bandwidth belongs to them.
(Score: 0) by Anonymous Coward on Sunday March 19 2017, @10:49PM (2 children)
Yeah, we should never even have built the internet, it just gets used by advertisters.
::rolleyes::
(Score: 0) by Anonymous Coward on Sunday March 19 2017, @11:05PM
You jest, but your statement is not necessarily invalid...
(Score: 3, Interesting) by Azuma Hazuki on Sunday March 19 2017, @11:11PM
This is one unfortunate, and perhaps unforeseen, consequence of the internet infrastructure itself being as close to "agnostic, dumb pipe with as little overhead as possible." I'm not sure it's the worst possibility though, and it may even be the best possible one, something like that old saw about democracy being the worst form of government except all the others. The idea is to build value at the last mile, not on the pipe itself (though of course our corporate overlords somehow managed to fuck that one up too...).
Sometimes it feels like we geeks are a little like the da Vinci stand-in Vetinari keeps captive in the Discworld books, the kind of people who bring about entirely new ways of doing things yet could never imagine what kind of horrors the greedy, evil, and venal among us would wreak with them.
I am "that girl" your mother warned you about...
(Score: 2, Interesting) by Anonymous Coward on Sunday March 19 2017, @10:55PM
Google serves up thumbnails that they have created using a JPEG encoder. These are accessed billions of times (youtube, search, google images, etc.) They are in a key position to optimize the amount of data they send. On that scale, this does matter, and it is not an arms race. Netflix, Amazon, and others can benefit from this.
(Score: 1) by corey on Monday March 20 2017, @01:51AM (1 child)
As much as I dislike Google for their privacy invasive products, they do great stuff sometimes. Make jpeg better, and open source it. That's great. Also their webP,webM, vp9, etc. Good things. Still don't like their products though..
So they do some evil, but give back a bit too.
(Score: 2, Insightful) by Anonymous Coward on Monday March 20 2017, @02:16AM
Few people or organizations are pure evil.
(Score: 3, Insightful) by martyb on Monday March 20 2017, @03:15AM
Maybe it's for Google's benefit?
How many times have you downloaded the internet? =)
But seriously, Google indexes the web, provides an image search, and also provides e-mail (with attachments)... and I'm sure there is more that I am forgetting.
Every byte they can persuade YOU to compress out of YOUR images, that's one less byte THEY need to download EACH TIME they crawl the web. It's one less byte they need to receive AND to send when you send a picture to someone via e-mail.
When you are as big as Google, and have the bandwidth needs they do, then it sure makes a lot of sense.
Related: Why do you think Google came up with their Chrome browser? My theory is that they already had written code to scrape data off web pages (for their search engine), right? Why not extend that to create a new browser? They could use that to encourage people to write standards-compliant web pages... to make it easier for Google to scrape the data off your pages. Remember, search-engine optimization IS a thing.
Wit is intellect, dancing.
(Score: 2) by shortscreen on Monday March 20 2017, @08:41AM (1 child)
A JPEG quantization table is just an array of 64 bytes. After you've run an 8x8 block of pixels through a Discrete Cosine Transform then you divide each coefficient by the corresponding value in the quantization table. Since you're doing integer math, dividing by a larger number is basically throwing away more data. Pretty soon you have an array of mostly zeros, which compresses very well, but image quality suffers. It should be noted that you won't get the best image quality by using a quantization table filled with 1s. If you open a high-quality JPEG from a camera in a hex editor and look at what is in the tables (there are generally two, one for the chroma channels and one for the luma channel) you will see 2s, 3s, 4s, and 5s in there. And I'd say you can take those tables and triple all the values, encode the same image again and cut your file size in half without a noticable difference.
It sounds plausible that Google may have found gains by tweaking quantization tables. Then again, it's strange to think that nobody in the world in the past 25 years bothered to try it before. Like, the JPEGroup came up with their table and everyone else just copied it.
The really weird part is where TFA says "Guetzli, due to the more involved quantization process, is slower than libjpeg." This makes no sense. Since the quantization table is just a predefined array of bytes, changing it has a negligible effect on encoding speed. So maybe the description of what Google did here is not even accurate.
I don't know how many different JPEG implementations are out there, but I know that at least one (the only one I tested) uses static Huffman tables. Maybe this is a common weakness in JPEG encoders. Maybe Google's encoder generates its own Huffman tables and this is why their files are smaller. (Would have been nice if TFA had some test files so I could compare the output with other programs)
(Score: 1, Informative) by Anonymous Coward on Monday March 20 2017, @01:18PM
They tune the quantization table to the image, instead of using the same quantization table for all images. This is slower.
(Score: 2) by KritonK on Monday March 20 2017, @09:40AM
I just tried it. While I usually create JPEGs with a quality of 75 (the default for cjpeg) for my web pages, guetzli will refuse to create JPEGs with a quality of less than 84, unless you tweak the source, claiming that using a lower quality will introduce artifacts.
Thus, while guetzli does produce smaller files than cjpeg with a quality of 84, both of these files are larger than the file produced by cjpeg wth a quality of 75.
The quality is supposed to be better, but, quite frankly, I couldn't tell the difference among any of the above three files or the original, and this was with the sample supplied by google, which one would assume that it would have been chosen to show off guetzli's abilities.