SoylentNews is people
SoylentNews
I have difficulty collaborating with other people. It may be that I'm a cantankerous curmudgeon but I suspect this is (also) a wider problem.
I write software. I dabble with analog and digital electronics. I do computer system adminstration and database administration. I prefer to write in C (for speed) and Perl (because the interpreter is everywhere). I greatly prefer to avoid proprietary environments. Although I've been described as being to the left of Richard Stallman, my interest in open licences comes primarily from not being able to use my own work. (Others take a much more hard-line approach and refuse outright to work on anything which won't be GNU Public Licence.)
I'm willing to work almost anywhere in the world. I'm willing to work with micro-controllers, super-computers, Lisp, PHP, bash, Linux, BSD, ATM, SPI, I2C, CANBus and numerous other concepts. I prefer to work with content versioning (any is better than none) and repeatable processes. However, I'm willing to work in an environment which scores 4 out of 12 on the Joel Test.
I have difficulty getting enthused about:-
- "Apps" where Apple or Google take 30%. This isn't a good deal for me or many clients.
- node.js or other JavaScript - especially in conjunction with robot control.
- C++, C#, Objective C, D or any other derivative of C.
- Anything over radio - especially IEEE802.15.4
- Block-chain systems in their current form.
- Facial recognition.
- Neural networks.
- "User experience."
- "Cloud."
I'm more enthused about FPGA and GPU programming but I like to have an abundance of hot-spares and, for me, that significantly increases a barrier to entry. This hardware also makes me edgy about being able to use my own work in the long-term.
Within these parameters, I'd like to describe some failed explorations with other techies. Without exception, I first encountered these techies at my local makerspace.
One techie is a huge fan of Mark Tilden, inventor of the RoboSapien and numerous other commerialized robots. Mark Tilden has an impeccable pedigee and has been featured in Wired Magazine and suchlike over 20 years or more. Mark Tilden espouses a low-power analog approximation of neurons with the intention of the neurons forming a combinatorial explosion of resonant states. When implemented correctly, motor back-EMF influences the neurons and therefore a robot blindly adjusts its (resonant) walking gait in response to terrain.
Anyhow, my friend was impressed that I'd used an operational-amplifier as an analog integrator connected to a VCO [Voltage Contrlled Oscillator] for the purpose of controlling servo motors without a micro-controller. My friend hoped that we'd be able to make a combo robot which, as a corollary, would perform a superset of 3D printing functionality without using stepper motors or servo motors. This is fairly ambitious but it would be worthwhile to investigate feasibility.
To complicate matters, I'm in a situation where modest financial success in this (or other) venture would gain my full-time attention. Whereas, he's currently receiving workman's compensation due to an incident at his electro-mechnical engineering job. So, I have every reason to commercialize our efforts and he has exactly the opposite motive.
There's also the scope of robot projects. They tend to be fairly open-ended and this in itself is cause for project failure. Therefore, I tried to nudge my friend into a project with a well-defined scope, such as high-fidelity audio reproduction. It is from attempts to make this situation work that I found that PAM8403 audio amplifiers were ideally suited to control motors. However, this discovery and further discoveries around it have not enthused my friend. (Note that the specification, as it exists, was to make an analog robot and avoid the use of stepper motors or servo motors. Admittedly, this fails to incorporate back-EMF but it is otherwise satisfactory.)
He's had personal problems which include his cheating Goth girlfriend dumping him and giving away his cat. He's also got his employer asking him to come back to work. Understandably, our collaboration has stalled.
An ex-colleague wants to collaborate in different projects. So far, we've had one success and one failure. The success contributed to a mutual ex-colleague's first Oscar nomination. The failure (his PHP, my MySQL) completely failed to mesh but led to a customized aggregator responsible for many articles on SoylentNews.
He's currently proposing an ambiguous deal which appears to be a skill-swap where we retain full rights on projects that we initiate. However, many of his projects involve tinkering with ADC and/or LCD. This would have been highly lucrative in the 1980s. Nowadays, it is somewhere between economizing and procrastinating. He previously worked on 100 Volt a monophonic valve amplifier but that was about three years ago and he never bothered to commercialize it.
Prior to this, I worked with friends on LEDs for an expanding market. We worked through the tipping point where the NFL and Formula 1 switched to LEDs. Two years ago, we were four years behind the market leader. Since then, we have completely failed to sell any LEDs to anyone. The majority have been using the venture for resumé padding and I've been out-voted on features even though I'm the only person who worked in the industry and worked with customers on a daily basis. After verbally being offered less than half of an equal share in my own venture, I haven't seen one of the active co-founders for 10 months. It would be reasonable to say that people are avoiding me. I find this bizarre.
Regardless, my task was to make a secure controller. The primary resumé padder wanted something which looked like an Apple TV. (This is for an industrial controller which may be exposed to 100% humidity and is required to meet ISO 13850.) I tried various options while being resource starved. I found that the software specification has yet to be achieved on a modern system. I found that the hardware specification is very difficult without an obscene budget. I also found that development on ARM has been difficult until relatively recently. The most accessible ARM system, a Raspberry Pi, didn't gain hardware floating point support under Linux until May 2016. Parties with successful projects (mostly phones) like these high barriers to entry. (How did Cobalt Networks launch successful RISC products in 1998? By using MIPS rather than ARM - and then switching to x86.)
In recent discussion about Raspberry Pi usage, you'll see how I've been influenced over the last two years or so about long-term support, hardware and packaging, micro-controller code density and other matters. I've also added options for documentation and training. However, without aligned collaborators or sensible resources, I'm not progressing. I suspect others are in diverse but analogous situations. Do you want to collaborate?
(Score: 1) by xyz on Monday July 03 2017, @11:54PM (4 children)
Sounds great.
Video CODECs are interesting. I was speculating a while back how it could look if some CODEC algorithm was to be designed to be "efficiently" implemented using GPU programmable shader capabilities. As far as I recall GPU based decoding / encoding / transcoding of "standard CODECs / streams" wasn't (in years past when I looked into it a little bit) shown to be that effective then again that was with ~1st generation GPUs (slow / limited by today's standards) and also the "standard CODECs" were designed for ASIC implementation and not efficient programmable implementation. And anyway compared to an ASIC that one is going to make millions of one can't really say that anything programmable is area / cost "efficient" but that only means that if you're Qualcomm or Broadcom or such you should prototype with FPGAs / GPUs and then make an ASIC for mass production. Programmable GPU / FPGA technologies can be very effective for cases where making ASICs is not time / cost justified including for lower volume / faster time to market stuff.
Network processing / acceleration / cacheing is also interesting. I think 1Gbit/s throughput for a lot of network processing is probably fairly readily achievable with relatively commodity technologies whether server / desktop class CPUs or consumer class GPUs or low-medium range FPGAs depending on the particular details of the task at hand. Some kinds of standard encryption relevant to networking are of course slower than others and some media encoders can be pretty resource intensive. I have seen network processor systems with 100Gb+ class throughput though they did not focus on media just data communications, and of course that gets expensive to make / buy pretty quickly.
There are off the shelf NPUs and accelerator ICs of various sorts that are designed to handle traffic at wire-line rates usually 1Gb/s and multiples of that depending on the task at hand. When / if they become cost effective to use in a system is debatable depending on the goals / requirements / resources.
For R&D purposes beyond what could be easily simulated / run in real time "at home" it could be interesting to use cloud compute instances since those are of course available "cheaply" with scalable data center class CPU / network facilities readily available as well as maybe even instance attached GPU or FPGA resources in some of the newer offerings that I haven't investigated.
Over the years I have done experiments with setting up "edge" or localhost servers (home use) for things like SQUID, Privoxy, NAS, IDS, etc. as well as small home experimental "HPC" clusters.
It does seem like there's still good opportunity to "appliance-ify" the sea of technological resources that a well equipped home or small business has access to. Relatlively powerful but still ordinary PCs have literally (1980s class) supercomputer performance for GFLOPS/s + GBys RAM. "Enthusiast" GPUs give XX TFLOP class peak performance. Even "slow" LAN networking is Gb/s class as a base level. Pretty much the smallest spinning disc you can buy is 2TBy in size for $70 or whatever. But given all that "capability" it is pretty amazing how underutilized it is relative to providing high availability / high throughput / high "value add" local IT resources. Having things like fully local offline instances of OSM, WP, project Gutenberg, heck the entire "dead tree" form information content of a good sized city's library would be something close to a drop in the bucket to store / process given such a modest system's technological capabilities. But it is not done except for very dedicated institutions. Information searches get offloaded to cloud vendors like Google but you probably have to be an IT sysadmin or running one of the very best in class local OS / IT solutions to even be able to do something comparable locally.
And with respect to network acceleration, of course it is truly amazing just how much "waste" there is in that missed opportunity. Every "web 2.0" web page one loads probably has something close to 250kBy of "dynamic but really mostly static" content -- images, scripts, markup, style sheets, etc. etc. So it is nothing unusual to see a "browser cache" explode into 100s of MBy for a single user session of some hours and RAM usage even get into the 1+ GBy range correspondingly. And of course the cost for loading all that whether DNS data or "semi-static" content through a somewhat high latency and somewhat congested pipe is lots of lost time in the "world wide wait" and also lots of needless network bandwidth and lots of needless local host resource consumption also. So I am sure when you multiply that by 2-200 users / site it is quite significant in resource "opportunity cost".
There are all these "edge" CDNs that try to speed some of the latency of it up but of course one can do even better by just having more of it local.
With respect to media caches, yes, that's also a familiar "pain point" as I have run various things like MythTV OTA boxes / media centers and so on and there should certainly be better ways to store / cache / transcode / search / etc. media whether audio, video or whatever.
(Score: 2) by cafebabe on Thursday July 06 2017, @02:52AM
You should understand the video system in outline after reading:-
Video would be divided into a number of tiles. This provides a number of advantages and disadvantages. Advantages include parallel decode via GPU and display of arbitrarily large images or video across multiple screens. Disadvantages include decrypting and caching every video tile.
1Gb/s should be sufficient for a single user not saturating an 1Gb/s link or dozens of users accessing common content cached from a slower link. Decryption and decompression occur at fairly consistent rates. Therefore, tasks can be specified as clock cycles per bit of data processed. Video decode may be an exception because the ability of a GPU to perform work in parallel depends upon common structure in video tiles. For example, if all tiles encode JPEG DCTs then GPU cores all perform DCT calculations on different data. However, if the range of video tile encodings is more diverse then many GPU cores will wait while others perform specific cases. There is also the problem that GPU cores align with screen resolution rather than media resolution. UHD 3840×2160 video is 60×34 tiles at 64×64 pixels per tile. This is 2040 tiles which may require a 400 core GPU to take six passes where the last pass incurs 10% GPU utilization.
In the case of caching combined with micro-payments, the obvious strategy is to cache everything forever. This is particularly true if the storage cost is cheaper than obtaining another copy of the data. If this is not possible then it may be beneficial to flush the cheapest content first.
1702845791×2
(Score: 2) by cafebabe on Tuesday July 11 2017, @11:18PM
More articles:-
1702845791×2
(Score: 2) by cafebabe on Sunday July 16 2017, @01:23AM (1 child)
More articles:-
1702845791×2
(Score: 2, Informative) by xyz on Sunday July 16 2017, @05:17AM
Thank you. You're on a great roll with producing excellent quality, insightful, deep, and interesting articles / summaries.
You have obviously put a great deal of thought and research into these topic areas. Some of the areas and contextual material I'm much more familiar with than others but I'm following along and am in agreement. Many of the noted deficiencies about current systems / practices are all to familiar to me while others were surprising but interesting / informative to learn about in greater depth. Altogether there is certainly a lot of opportunity to do things better / more efficiently / more usably in the realm of data / media transmission and even moreso information and media indexing, catalogging, metadata management, and content aware discovery / transmission / cacheing / storage / utilization / derivation. "Library science" principles and "semantic web / media" ones are vastly underutilized to the point of virtual nonexistence in both the internet and intranet these days. You eloquently highlighted one major deficiency (which I often lament) in aptly pointing out that commonly relevant and "highly usable" schemes to "name" / designate, and also locate articles of information online are fundamently absent, or at least severely limited in their ubiquity / usability. It was refreshing to see project Xanadu mentioned as well as other concepts for referring to resources by their identity (e.g. DOI, hash, URI, whatever) as well as by additional dimensionalities (X, Y, temporal offset, logical subdivisions,....) and how those referential operations could practically efficiencly interact with existing 'de facto' access / query facilities like DNS, UDP, TCP. Of course other data / metadata, categorical, representational, transport, encoding and query systems exist such as XML, SGML, XQUERY, XPATH, SQL, Dublin Core, Gopher, WAIS, et. al. but at the end of the day "none of the above" is really what has been the de facto choice. So in this NATed IPv4 world even hosts don't commonly have useful names / addresses wrt. DNS. What use could be made of key / identity oriented systems (SSO, OAUTH, personal certificates, personal GPG, ubiquitous host / server certificates, et. al.) has been very slow and limited in its growth and promulgation. Similarly the "usual" capabilities of DNS have become increasingly underutilized and extensions thereof even moreso.
People even settle en masse for personal email and web addresses that aren't truly "their own" and delegate EVERYTHING (email, document hosting / sharing / editing, messaging, telephony, service hosting, ...) to "here today gone tomorrow" cloud based entities which offer NOTHING of persistent value. Not names/identities (email address, domain name). Not locations (URL stability). Not data storage. Not anything. So in this nameless, metadataless, ephemeral vs. persistent, data-ful but information-less world we live.
In any case starting with A/V media as one emphasis as you have been delving deeply into is a good idea because it is something concrete. Because it is something "bulky" and therefore especially relevant to considerations of efficient location / distribution / transmission / storage. Because it is something more "abstract" in some ways (it is generally human made and not machine made. It is something which does not so easily admit machine based "understanding" / NLP / interpretation. It is something with concrete enough attributes and dimensionalities (e.g. temporal and sectional offsets), a "simple" set of relevant metadata attributes does a good job at at least basically categorizing many aspects of its nature (length, title, subject, contained A/V languages, author, location, sub-stream metadata, ...). It is something that you benefit greatly or essentially from having a computer mediate your access to (CODECs, seeking, searching, format conversion, smart streaming / cacheing). It is bulky enough and resource intensive enough that smart decisions about encoding / decoding / server location / algorithms / cache / mirroring / backup) pay off.
It is certainly one of the "pain points" of the current internet as visible in the UX. "Please wait...buffering...", "Server not found", "Jitter / freeze", unresponsive pause / play / seek / speed controls, practically nonexistent metadata / information / content management tools (search, filter, translate, subtitle, transcode, cache control, ANNOTATE, EXCERPT, CITE, ...). No particularly smart and effective use is made of real time transcoding or VBR or alternative streaming at least in a way that apparently is functional. Or at least it is often perceived to fail to function even if the heuristic intelligence of those things is unappreciated and unappreciable "when it works seamlessly" in the UX.
As you said some of that would probably be better if everything wasn't just shoved over TCP with the local client being treated as a "dumb client" barely different than a "TV receiver" to maximize remote control & monitoring at the expense of the UX and the ability to effectively use the information with a smart local "own edge" server / client.
Of course the ubiquitous enemies of "free/open internet" and net neutrality seek to monitor, throttle, deprioritize, block, translate, transform your content / media streams at their will based upon their intercepted data / metadata / subject / origin / destination / protocol and on the other hand for partly opposing but unfortunately not simply antithetical reasons other interests seek to make it all opaque, encrypted, non-standard in protocol / format / encoding / metadata. And the users (or less powerful providers) too often end up with the disadvantages of all possible "approaches". None of that is favorable to the open and standardizes implementation of endpoint or path resident cacheing, multicast, translation, useful QoS provisioning, et. al.
Nevertheless as you've well illustrated inside the present "tower of babylon" of data overload / chaos, information underload, and protocol chaos / insufficiency it is an opportunity rich environment for added efficiencies and better UXs at all levels.
So let's see what can be done. I will continue reading and reflecting upon all that has been said in detail and in breadth.