Stories
Slash Boxes
Comments

SoylentNews is people

posted by Fnord666 on Thursday June 21 2018, @12:46AM   Printer-friendly
from the approaching-the-singularity dept.

IBM researchers use analog memory to train deep neural networks faster and more efficiently

Deep neural networks normally require fast, powerful graphical processing unit (GPU) hardware accelerators to support the needed high speed and computational accuracy — such as the GPU devices used in the just-announced Summit supercomputer. But GPUs are highly energy-intensive, making their use expensive and limiting their future growth, the researchers explain in a recent paper published in Nature.

Instead, the IBM researchers used large arrays of non-volatile analog memory devices (which use continuously variable signals rather than binary 0s and 1s) to perform computations. Those arrays allowed the researchers to create, in hardware, the same scale and precision of AI calculations that are achieved by more energy-intensive systems in software, but running hundreds of times faster and at hundreds of times lower power — without sacrificing the ability to create deep learning systems.

The trick was to replace conventional von Neumann architecture, which is "constrained by the time and energy spent moving data back and forth between the memory and the processor (the 'von Neumann bottleneck')," the researchers explain in the paper. "By contrast, in a non-von Neumann scheme, computing is done at the location of the data [in memory], with the strengths of the synaptic connections (the 'weights') stored and adjusted directly in memory.

Equivalent-accuracy accelerated neural-network training using analogue memory (DOI: 10.1038/s41586-018-0180-5) (DX)


Original Submission

 
This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 2) by jmorris on Thursday June 21 2018, @07:04AM (4 children)

    by jmorris (4844) on Thursday June 21 2018, @07:04AM (#696073)

    I can't help it if you are having problems with multi-threaded and multi-processor code that doesn't synchronize correctly. Unless you are intentionally introducing real random numbers (and while some simulations indeed must, most prefer pseudo-random numbers for the repeatability across runs) then you should be getting 100% repeatable results. If you aren't, you need to debug some more. Math isn't generally supposed to be random. Digital computers aren't supposed to be random. The bits that aren't deterministic (cache effects, network, other I/O) are supposed to be abstracted away by the software to help you produce stable results, Unexplained little "random" variations will eventually bite yer ass at the wrong time.

    Of course you might also be one of those rare cases that had to made a deliberate decision to sacrifice correctness on the altar of performance. Sometimes that is the right solution; doesn't really matter how correct a weather forecast is if it takes two days to tell you tomorrow's weather.

    Starting Score:    1  point
    Karma-Bonus Modifier   +1  

    Total Score:   2  
  • (Score: 2) by c0lo on Thursday June 21 2018, @07:48AM

    by c0lo (156) Subscriber Badge on Thursday June 21 2018, @07:48AM (#696089) Journal

    I can't help it if you are having problems with multi-threaded and multi-processor code that doesn't synchronize correctly.

    Eh, what?
    Not all multithreading apps need to be synchronized, many still produce valid results accepting race conditions.

    --
    https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
  • (Score: 1) by anubi on Thursday June 21 2018, @07:53AM

    by anubi (2828) on Thursday June 21 2018, @07:53AM (#696092) Journal

    I get the idea they are only re-inventing the hybrid analog computer. Having the analog part do what analog is really good at, and having the digital do what the digital is really good at. The old-school type ( Like the EAI 580 I used at university ) was optimized for solving real-time input simultaneous differential equations. The continuous analog functions were done in analog, whereas the digital part controlled and monitored the whole shebang.

    Except, like everything else, things have improved by many orders of magnitude. Comparing what they have to my EAI-580 would probably be akin to comparing an 8080 to a modern hex-core CPU.

    --
    "Prove all things; hold fast that which is good." [KJV: I Thessalonians 5:21]
  • (Score: 1, Informative) by Anonymous Coward on Thursday June 21 2018, @08:08AM (1 child)

    by Anonymous Coward on Thursday June 21 2018, @08:08AM (#696096)

    you're trolling, right? did you ever hear of the Lorenz system? here you go: https://en.wikipedia.org/wiki/Lorenz_system [wikipedia.org]

    numerical analysis says that it's best to use random round-off errors when integrating a generic dynamical system (otherwise you get systematic differences from the original dynamical system).
    that means that every time you integrate N steps of the Lorenz system, you get a random sequence of N round-off errors applied to the calculations.
    when N becomes large enough, those errors build up into something big enough to completely change the trajectory, because the Lorenz system is chaotic.

    for weather and similar fluid dynamics problems, this only gets worse.

    • (Score: 2) by jmorris on Thursday June 21 2018, @08:15AM

      by jmorris (4844) on Thursday June 21 2018, @08:15AM (#696100)

      As I said, in most such cases you want pseudo-random numbers instead of real crypto quality random for those purposes so you still have repeatability. Then you can run the same program again with the same inputs and get consistent outputs, which makes testing a lot easier.