Storage Matters: Why Xbox and Playstation SSDs Usher In A New Era of Gaming
A new generation of gaming consoles is due to hit the market later this year, and the hype cycle for the Xbox Series X and Playstation 5 has been underway for more than a year. Solid technical details (as opposed to mere rumors) have been slower to arrive, and we still know much less about the consoles than we typically know about PC platforms and components during the post-announcement, pre-availability phase. We have some top-line performance numbers and general architectural information from Microsoft and Sony, but not quite a full spec sheet.
[...] Solid State Drives were revolutionary for the PC market, providing immense improvements to overall system responsiveness. Games benefited mostly in the form of faster installation and level load times, but fast storage also helped reduce stalls and stuttering when a game needs to load data on the fly. In recent years, NVMe SSDs have provided speeds that are on paper several times faster than what is possible with SATA SSDs, but for gamers the benefits have been muted at best. Conventional wisdom holds that there are two main causes to suspect for this disappointment: First, almost all games and game engines are still designed to be playable off hard drives because current consoles and many low-end PCs lack SSDs. Game programmers cannot take full advantage of NVMe SSD performance without making their games unplayably slow on hard drives. Second, SATA SSDs are already fast enough to shift the bottleneck elsewhere in the system, often in the form of data decompression. Something aside from the SSD needs to be sped up before games can properly benefit from NVMe performance.
Microsoft and Sony are addressing both of those issues with their upcoming consoles. Game developers will soon be free to assume that users will have fast storage, both on consoles and on PCs. In addition, the new generation of consoles will add extra hardware features to address bottlenecks that would be present if they were merely mid-range gaming PCs equipped with cutting-edge SSDs. However, we're still dealing with powerful hype operations promoting these upcoming devices. Both companies are guilty of exaggerating or oversimplifying in their attempts to extol the new capabilities of their next consoles, especially with regards to the new SSDs. And since these consoles are still closed platforms that aren't even on the market yet, some of the most interesting technical details are still being kept secret.
From page 2, dedicated decompression is a key feature:
The most important specialized hardware feature the consoles will include to complement storage performance is dedicated data decompression hardware. Game assets must be stored on disk in a compressed form to keep storage requirements somewhat reasonable. Games usually rely on multiple compression methods—some lossy compression methods specialized for certain types of data (eg. audio and images), and some lossless general-purpose algorithm, but almost everything goes through at least one compression method that is fairly computationally complex. GPU architectures have long included hardware to handle decoding video streams and support simple but fast lossy texture compression methods like S3TC and its successors, but that leaves a lot of data to be decompressed by the CPU. Desktop CPUs don't have dedicated decompression engines or instructions, though many instructions in the various SIMD extensions are intended to help with tasks like this. Even so, decompressing a stream of data at several GB/s is not trivial, and special-purpose hardware can do it more efficiently while freeing up CPU time for other tasks. The decompression offload hardware in the upcoming consoles is implemented on the main SoC so that it can unpack data after it traverses the PCIe link from the SSD and resides in the main RAM pool shared by the GPU and CPU cores.
[...] The CPU time saved by these decompression units sounds astounding: the equivalent of about 9 Zen 2 CPU cores for the PS5, and about 5 for the Xbox Series X. Keep in mind these are peak numbers that assume the SSD bandwidth is being fully utilized—real games won't be able to keep these SSDs 100% busy, so they wouldn't need quite so much CPU power for decompression.
With the CPU and dedicated decompression capability alone, the PS5 will have up to the equivalent performance of 17 Zen 2 CPU cores (more than the Ryzen 9 3950X). Other capabilities may add the equivalent of another one or two cores. Future desktop CPUs may need to add some of these SoC features to handle faster SSD storage.
(Score: 5, Interesting) by takyon on Monday June 15 2020, @10:05PM (11 children)
https://www.forbes.com/sites/kevinmurnane/2019/10/19/the-super-fast-ssds-in-the-next-gen-game-consoles-may-turn-out-to-be-a-mixed-blessing/ [forbes.com]
http://www.pushsquare.com/news/2020/03/ps5_ssd_could_reduce_game_sizes_and_save_storage_space [pushsquare.com]
[SIG] 10/28/2017: Soylent Upgrade v14 [soylentnews.org]
(Score: 2) by shortscreen on Tuesday June 16 2020, @01:54AM (5 children)
Storing the same data 400 times and calling that an optimization? Yeah, right. They say there was 10GB of duped data, so this mysterious file that was duped 400 times would be .025 GB... AT MOST. It would take significantly longer to transfer that data from the device than it would for any seek. So what is the point again? And if the file is smaller than that, so much that the seek time is longer than the read time, and used that frequently, it makes no sense to not keep it in RAM.
Rapidly shuffling things from disk to RAM to the extent that you would reasonably be worried about disk access time means that the real problem is lack of RAM. Putting in an SSD so that you can use that as a less-slow substitute seems like a much worse solution than putting in more RAM. Likewise, data compression isn't going to help much if you have no free RAM to decompress the data to. Then there's this:
Huh? The SSD is faster. Game programmers don't have to do anything. That is the advantage. There's no need to make things unplayably slow from HDD just to claim that SSD is better.
Ever since I saw a game store all its assets as 20,000 PNG and WAV files inside one giant ZIP file, I'm inclined to assume that all issues with game load times are caused by simple incompetence.
(Score: 2) by takyon on Tuesday June 16 2020, @02:10AM (1 child)
Those are two separate factoids. It't not one big file. It's various smaller textures and assets duplicated. One specific example: a mailbox.
https://www.eurogamer.net/articles/ps5-ssd-hard-drive-size-expanded-storage-6300 [eurogamer.net]
The total waste due to various duplications is 10 GB.
RAM is/was limited. 8 GB on PS4.
Incompetence can be confused for convenience. Easier to program = faster/larger game development.
[SIG] 10/28/2017: Soylent Upgrade v14 [soylentnews.org]
(Score: 5, Interesting) by Immerman on Tuesday June 16 2020, @03:06AM
To make your point a little more explicit:
If all the assets (geometry, logic, characters, textures, mailboxes, etc) for a game region are duplicated into a single sequential block on disc, so you can load it all in a single pass, then the entire loading process requires only a single seek operation.
If instead you load from a "database" of assets with no duplication, then loading the region requires one seek per asset, which can easily be in the thousands. Multiply that by a 10ms seek time and you're talking tens of seconds of extra load time just for the seek operations. Asset caching can probably improve that considerably in most games, but that comes at the price of memory overhead and greater RAM fragmentation, which can hurt the game's frame rate.
(Score: 2) by Mykl on Tuesday June 16 2020, @06:08AM
It depends what you're trying to optimize. In this case, it's clearly not storage. In games, it's all about the frame rates and load times. So yes, this game with 10Gb of redundant assets is optimized for its purpose.
(Score: 1, Interesting) by Anonymous Coward on Tuesday June 16 2020, @05:18PM (1 child)
That's actually a really efficient way to do it.
Storing every asset as an individual file is powerful but inefficient. The inefficiency comes from the fact that a file system is optimized to allow files to be added, renamed, moved, or deleted. By comparison, a ZIP file is less powerful but more efficient. The ZIP file header is smaller than the file system's allocation table. That is because if you want to add, rename, or delete a file from a ZIP you basically rewrite the entire ZIP file. But games don't need to do that. The game can also cache portions of the ZIP header in memory since it knows exactly which assets it is going to need next. If the game used the file system, it would be relying on the OS to cache that, but the OS does not know what files the game will need next.
(Score: 2) by shortscreen on Wednesday June 17 2020, @03:45AM
It's less bad than 20,000 files littering your filesystem but it's still bad.
1) WAVs, and many other types of audiovisual files, don't compress well in a ZIP file. PNGs won't compress at all because PNG itself already uses the same compressor that ZIP does.
2) when you compress files individually (eg. as PNGs) then you can't deduplicate data that is shared between them (for instance, if they were successive frames of an animation)
3) searching a ZIP header for a specific file by name is slower than just having an offset compiled into the game data. Unless you are making assumptions about the order the files are listed in the ZIP header and/or the order of the data in the ZIP file then you are wasting time searching for each file.
(Score: 2) by Mojibake Tengu on Tuesday June 16 2020, @07:02AM (4 children)
Madness engineering like this is a direct consequence of another dumb approach to platform design: gaming consoles are chronically underengineered with low memory by pure evil intent. Considering what games (and subscriptions) cost to players, a fair price for a fair amount of memory would be trivial expenditure.
Sadly enough, the PS5 still follows this kind of evility further.
Respect Authorities. Know your social status. Woke responsibly.
(Score: 2) by takyon on Tuesday June 16 2020, @09:40AM (2 children)
If the next-gen consoles were PCs, they would be a great value. Likely $450-$500 price tag for either PS5 or XSX, probably $50 less for the PS5 "Digital Edition" without an optical drive. And you get an 8-core CPU, GPU around the performance of RTX 2080 or RTX 2080 Super (but with even better ray-tracing performance), a PCIe 4.0 SSD, etc. But they are not PCs, online subscriptions are a cash cow, and if you can't jailbreak it without bricking it, no sailing the high seas.
16 GB RAM does come in below what was expected (24 GB), and that amount is shared between the CPU and GPU with some reserved to the system. That's the new standard for up to several years, even if there is a mid-cycle refresh.
On the other hand, storage bottlenecks in PCs are clearly worse than in these consoles, to the point where you might need 12-16 CPU cores to keep up with the 8-core console, and more RAM than the console to make up for worse SSDs. The console will make far better use of that lower RAM for gaming, and the fast SSD will help, even if "treating the SSD like a pool of DDR2 RAM" is an exaggeration. PC gamers who are not running the latest games on an SSD will need to cut that shit out, or use a large RAM drive to compensate.
Another thing that might help: putting an SSD inside the GPU, i.e. the "SSG" concept that AMD showed off in 2016 [tomshardware.com]. Big VRAM is good, but putting an entire ~100 GB game close to the GPU could reduce latency further. 128-256 GB of fast NAND costs little in relation to a high-end $800-$1,000+ GPU, although it remains to be seen if AMD, Nvidia, or Intel will go down that path.
[SIG] 10/28/2017: Soylent Upgrade v14 [soylentnews.org]
(Score: 0) by Anonymous Coward on Tuesday June 16 2020, @12:53PM (1 child)
you can save about 2 to 3 cpu cores if you don't need to "run" winblows under the hood.
THAT's probably the main performance benefit of consoles :P
(Score: 2) by takyon on Tuesday June 16 2020, @02:50PM
Even on the Xbox Series Sex? :P
[SIG] 10/28/2017: Soylent Upgrade v14 [soylentnews.org]
(Score: 2) by Immerman on Tuesday June 16 2020, @01:43PM
Memory doesn't necessarily have much to do with it. As I mentioned a few posts above, If you're loading thousands of assets for a game-region from a database-style, duplication-free file, then it's going to take thousands of seek operations to complete, which means tens of seconds of additional load time. More RAM will allow you to cache more assets between regions, or larger chunks of raw file data during the loading process, but basically means you've got a whole bunch of RAM that's being removed from gameplay use in order to speed up load times.
There's also another loading performance bottleneck that can be eliminated with on-disk duplication: the time necessary to transform data from the on-disk representation to the in-memory representation. I don't know how common it is, but I've heard of several games that simply dumped the fully-loaded in-memory representation directly to disk. That way instead of any involved loading process you simply load the memory dump into RAM, adjust any pointers from canonical to "real" form, and start running.