Stories
Slash Boxes
Comments

SoylentNews is people

posted by Fnord666 on Monday June 15 2020, @09:14PM   Printer-friendly
from the size-matters dept.

Storage Matters: Why Xbox and Playstation SSDs Usher In A New Era of Gaming

A new generation of gaming consoles is due to hit the market later this year, and the hype cycle for the Xbox Series X and Playstation 5 has been underway for more than a year. Solid technical details (as opposed to mere rumors) have been slower to arrive, and we still know much less about the consoles than we typically know about PC platforms and components during the post-announcement, pre-availability phase. We have some top-line performance numbers and general architectural information from Microsoft and Sony, but not quite a full spec sheet.

[...] Solid State Drives were revolutionary for the PC market, providing immense improvements to overall system responsiveness. Games benefited mostly in the form of faster installation and level load times, but fast storage also helped reduce stalls and stuttering when a game needs to load data on the fly. In recent years, NVMe SSDs have provided speeds that are on paper several times faster than what is possible with SATA SSDs, but for gamers the benefits have been muted at best. Conventional wisdom holds that there are two main causes to suspect for this disappointment: First, almost all games and game engines are still designed to be playable off hard drives because current consoles and many low-end PCs lack SSDs. Game programmers cannot take full advantage of NVMe SSD performance without making their games unplayably slow on hard drives. Second, SATA SSDs are already fast enough to shift the bottleneck elsewhere in the system, often in the form of data decompression. Something aside from the SSD needs to be sped up before games can properly benefit from NVMe performance.

Microsoft and Sony are addressing both of those issues with their upcoming consoles. Game developers will soon be free to assume that users will have fast storage, both on consoles and on PCs. In addition, the new generation of consoles will add extra hardware features to address bottlenecks that would be present if they were merely mid-range gaming PCs equipped with cutting-edge SSDs. However, we're still dealing with powerful hype operations promoting these upcoming devices. Both companies are guilty of exaggerating or oversimplifying in their attempts to extol the new capabilities of their next consoles, especially with regards to the new SSDs. And since these consoles are still closed platforms that aren't even on the market yet, some of the most interesting technical details are still being kept secret.

From page 2, dedicated decompression is a key feature:

The most important specialized hardware feature the consoles will include to complement storage performance is dedicated data decompression hardware. Game assets must be stored on disk in a compressed form to keep storage requirements somewhat reasonable. Games usually rely on multiple compression methods—some lossy compression methods specialized for certain types of data (eg. audio and images), and some lossless general-purpose algorithm, but almost everything goes through at least one compression method that is fairly computationally complex. GPU architectures have long included hardware to handle decoding video streams and support simple but fast lossy texture compression methods like S3TC and its successors, but that leaves a lot of data to be decompressed by the CPU. Desktop CPUs don't have dedicated decompression engines or instructions, though many instructions in the various SIMD extensions are intended to help with tasks like this. Even so, decompressing a stream of data at several GB/s is not trivial, and special-purpose hardware can do it more efficiently while freeing up CPU time for other tasks. The decompression offload hardware in the upcoming consoles is implemented on the main SoC so that it can unpack data after it traverses the PCIe link from the SSD and resides in the main RAM pool shared by the GPU and CPU cores.

[...] The CPU time saved by these decompression units sounds astounding: the equivalent of about 9 Zen 2 CPU cores for the PS5, and about 5 for the Xbox Series X. Keep in mind these are peak numbers that assume the SSD bandwidth is being fully utilized—real games won't be able to keep these SSDs 100% busy, so they wouldn't need quite so much CPU power for decompression.

With the CPU and dedicated decompression capability alone, the PS5 will have up to the equivalent performance of 17 Zen 2 CPU cores (more than the Ryzen 9 3950X). Other capabilities may add the equivalent of another one or two cores. Future desktop CPUs may need to add some of these SoC features to handle faster SSD storage.


Original Submission

 
This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 5, Interesting) by takyon on Monday June 15 2020, @10:05PM (11 children)

    by takyon (881) <reversethis-{gro ... s} {ta} {noykat}> on Monday June 15 2020, @10:05PM (#1008364) Journal

    https://www.forbes.com/sites/kevinmurnane/2019/10/19/the-super-fast-ssds-in-the-next-gen-game-consoles-may-turn-out-to-be-a-mixed-blessing/ [forbes.com]

    Those richer images are going to involve storing more data and SSDs provide benefits here as well. When data is read from the hard drives used in today’s consoles, a disk head travels to the location where the data is stored. The journey takes time. Developers reduce this time by storing commonly used data over and over again so it’s near the location where the disk head happens to be when the data is needed again. This redundant storage can be extreme. Cerny noted that some data was duplicated 400 times in Spider-Man. SSDs are so fast you don’t need data redundancy which opens up space to store more data.

    http://www.pushsquare.com/news/2020/03/ps5_ssd_could_reduce_game_sizes_and_save_storage_space [pushsquare.com]

    However, because the PS5’s SSD is so fast, this kind of shortcut may become a thing of the past. According to an old GDC presentation, around 10GB of Marvel’s Spider-Man’s overall file size could be attributed to duplicated assets and objects, so that’s a big overall saving if this technique is rendered obsolete.

    --
    [SIG] 10/28/2017: Soylent Upgrade v14 [soylentnews.org]
    Starting Score:    1  point
    Moderation   +3  
       Interesting=2, Informative=1, Total=3
    Extra 'Interesting' Modifier   0  
    Karma-Bonus Modifier   +1  

    Total Score:   5  
  • (Score: 2) by shortscreen on Tuesday June 16 2020, @01:54AM (5 children)

    by shortscreen (2252) on Tuesday June 16 2020, @01:54AM (#1008446) Journal

    Storing the same data 400 times and calling that an optimization? Yeah, right. They say there was 10GB of duped data, so this mysterious file that was duped 400 times would be .025 GB... AT MOST. It would take significantly longer to transfer that data from the device than it would for any seek. So what is the point again? And if the file is smaller than that, so much that the seek time is longer than the read time, and used that frequently, it makes no sense to not keep it in RAM.

    Rapidly shuffling things from disk to RAM to the extent that you would reasonably be worried about disk access time means that the real problem is lack of RAM. Putting in an SSD so that you can use that as a less-slow substitute seems like a much worse solution than putting in more RAM. Likewise, data compression isn't going to help much if you have no free RAM to decompress the data to. Then there's this:

    Game programmers cannot take full advantage of NVMe SSD performance without making their games unplayably slow on hard drives.

    Huh? The SSD is faster. Game programmers don't have to do anything. That is the advantage. There's no need to make things unplayably slow from HDD just to claim that SSD is better.

    Ever since I saw a game store all its assets as 20,000 PNG and WAV files inside one giant ZIP file, I'm inclined to assume that all issues with game load times are caused by simple incompetence.

    • (Score: 2) by takyon on Tuesday June 16 2020, @02:10AM (1 child)

      by takyon (881) <reversethis-{gro ... s} {ta} {noykat}> on Tuesday June 16 2020, @02:10AM (#1008453) Journal

      They say there was 10GB of duped data, so this mysterious file that was duped 400 times would be .025 GB... AT MOST.

      Those are two separate factoids. It't not one big file. It's various smaller textures and assets duplicated. One specific example: a mailbox.

      https://www.eurogamer.net/articles/ps5-ssd-hard-drive-size-expanded-storage-6300 [eurogamer.net]

      For example, games currently must load individual areas one at a time - such as a city block in an open world game such as Spider-Man - and because of this, there is a duplication of common objects within the world - such as a mailbox.

      Game sizes are bigger than they should be as a result because, as Mark Cerny explains, there are hundreds of duplicate 'mailboxes' throughout the game package, which your PS4 can then easily access regardless of the area that's been loaded from the hard drive.

      The total waste due to various duplications is 10 GB.

      RAM is/was limited. 8 GB on PS4.

      Incompetence can be confused for convenience. Easier to program = faster/larger game development.

      --
      [SIG] 10/28/2017: Soylent Upgrade v14 [soylentnews.org]
      • (Score: 5, Interesting) by Immerman on Tuesday June 16 2020, @03:06AM

        by Immerman (3985) on Tuesday June 16 2020, @03:06AM (#1008466)

        To make your point a little more explicit:

        If all the assets (geometry, logic, characters, textures, mailboxes, etc) for a game region are duplicated into a single sequential block on disc, so you can load it all in a single pass, then the entire loading process requires only a single seek operation.

        If instead you load from a "database" of assets with no duplication, then loading the region requires one seek per asset, which can easily be in the thousands. Multiply that by a 10ms seek time and you're talking tens of seconds of extra load time just for the seek operations. Asset caching can probably improve that considerably in most games, but that comes at the price of memory overhead and greater RAM fragmentation, which can hurt the game's frame rate.

    • (Score: 2) by Mykl on Tuesday June 16 2020, @06:08AM

      by Mykl (1112) on Tuesday June 16 2020, @06:08AM (#1008496)

      Storing the same data 400 times and calling that an optimization? Yeah, right

      It depends what you're trying to optimize. In this case, it's clearly not storage. In games, it's all about the frame rates and load times. So yes, this game with 10Gb of redundant assets is optimized for its purpose.

    • (Score: 1, Interesting) by Anonymous Coward on Tuesday June 16 2020, @05:18PM (1 child)

      by Anonymous Coward on Tuesday June 16 2020, @05:18PM (#1008751)

      Ever since I saw a game store all its assets as 20,000 PNG and WAV files inside one giant ZIP file, I'm inclined to assume that all issues with game load times are caused by simple incompetence.

      That's actually a really efficient way to do it.

      Storing every asset as an individual file is powerful but inefficient. The inefficiency comes from the fact that a file system is optimized to allow files to be added, renamed, moved, or deleted. By comparison, a ZIP file is less powerful but more efficient. The ZIP file header is smaller than the file system's allocation table. That is because if you want to add, rename, or delete a file from a ZIP you basically rewrite the entire ZIP file. But games don't need to do that. The game can also cache portions of the ZIP header in memory since it knows exactly which assets it is going to need next. If the game used the file system, it would be relying on the OS to cache that, but the OS does not know what files the game will need next.

      • (Score: 2) by shortscreen on Wednesday June 17 2020, @03:45AM

        by shortscreen (2252) on Wednesday June 17 2020, @03:45AM (#1009009) Journal

        It's less bad than 20,000 files littering your filesystem but it's still bad.

        1) WAVs, and many other types of audiovisual files, don't compress well in a ZIP file. PNGs won't compress at all because PNG itself already uses the same compressor that ZIP does.
        2) when you compress files individually (eg. as PNGs) then you can't deduplicate data that is shared between them (for instance, if they were successive frames of an animation)
        3) searching a ZIP header for a specific file by name is slower than just having an offset compiled into the game data. Unless you are making assumptions about the order the files are listed in the ZIP header and/or the order of the data in the ZIP file then you are wasting time searching for each file.

  • (Score: 2) by Mojibake Tengu on Tuesday June 16 2020, @07:02AM (4 children)

    by Mojibake Tengu (8598) on Tuesday June 16 2020, @07:02AM (#1008506) Journal

    This redundant storage can be extreme.

    Madness engineering like this is a direct consequence of another dumb approach to platform design: gaming consoles are chronically underengineered with low memory by pure evil intent. Considering what games (and subscriptions) cost to players, a fair price for a fair amount of memory would be trivial expenditure.

    Sadly enough, the PS5 still follows this kind of evility further.

    --
    Respect Authorities. Know your social status. Woke responsibly.
    • (Score: 2) by takyon on Tuesday June 16 2020, @09:40AM (2 children)

      by takyon (881) <reversethis-{gro ... s} {ta} {noykat}> on Tuesday June 16 2020, @09:40AM (#1008513) Journal

      If the next-gen consoles were PCs, they would be a great value. Likely $450-$500 price tag for either PS5 or XSX, probably $50 less for the PS5 "Digital Edition" without an optical drive. And you get an 8-core CPU, GPU around the performance of RTX 2080 or RTX 2080 Super (but with even better ray-tracing performance), a PCIe 4.0 SSD, etc. But they are not PCs, online subscriptions are a cash cow, and if you can't jailbreak it without bricking it, no sailing the high seas.

      16 GB RAM does come in below what was expected (24 GB), and that amount is shared between the CPU and GPU with some reserved to the system. That's the new standard for up to several years, even if there is a mid-cycle refresh.

      On the other hand, storage bottlenecks in PCs are clearly worse than in these consoles, to the point where you might need 12-16 CPU cores to keep up with the 8-core console, and more RAM than the console to make up for worse SSDs. The console will make far better use of that lower RAM for gaming, and the fast SSD will help, even if "treating the SSD like a pool of DDR2 RAM" is an exaggeration. PC gamers who are not running the latest games on an SSD will need to cut that shit out, or use a large RAM drive to compensate.

      Another thing that might help: putting an SSD inside the GPU, i.e. the "SSG" concept that AMD showed off in 2016 [tomshardware.com]. Big VRAM is good, but putting an entire ~100 GB game close to the GPU could reduce latency further. 128-256 GB of fast NAND costs little in relation to a high-end $800-$1,000+ GPU, although it remains to be seen if AMD, Nvidia, or Intel will go down that path.

      --
      [SIG] 10/28/2017: Soylent Upgrade v14 [soylentnews.org]
      • (Score: 0) by Anonymous Coward on Tuesday June 16 2020, @12:53PM (1 child)

        by Anonymous Coward on Tuesday June 16 2020, @12:53PM (#1008582)

        you can save about 2 to 3 cpu cores if you don't need to "run" winblows under the hood.
        THAT's probably the main performance benefit of consoles :P

    • (Score: 2) by Immerman on Tuesday June 16 2020, @01:43PM

      by Immerman (3985) on Tuesday June 16 2020, @01:43PM (#1008613)

      Memory doesn't necessarily have much to do with it. As I mentioned a few posts above, If you're loading thousands of assets for a game-region from a database-style, duplication-free file, then it's going to take thousands of seek operations to complete, which means tens of seconds of additional load time. More RAM will allow you to cache more assets between regions, or larger chunks of raw file data during the loading process, but basically means you've got a whole bunch of RAM that's being removed from gameplay use in order to speed up load times.

      There's also another loading performance bottleneck that can be eliminated with on-disk duplication: the time necessary to transform data from the on-disk representation to the in-memory representation. I don't know how common it is, but I've heard of several games that simply dumped the fully-loaded in-memory representation directly to disk. That way instead of any involved loading process you simply load the memory dump into RAM, adjust any pointers from canonical to "real" form, and start running.