SoylentNews

I'm currently busy because I received start-up funding (and use of a ridiculously swanky office). The first development deadline is on Mon 31 Dec 2018. The second development deadline is on Thu 31 Jan 2019. The third development deadline is on Sun 31 Mar 2019. Funding currently lasts until Fri 31 May 2019. Unfortunately, this has greatly curtailed my annual fun programming project for Christmas 2018. Previous efforts have taken three days. This took three hours.

I have implemented a small system which predicts the final moderation score of a message on a forum. This uses some search engine theory, such as postings and repetition limits. However, it doesn't use n-grams, stemming, weighting by word frequency or Bayes theorem. Indeed, a significant amount of theory has not been implemented and the result only makes predictions with a relatively weak correlation. This implementation only works with the output of SoylentNews and is heavily dependent upon certain attributes currently found in the HTML. It is also heavily dependent upon line breaks typically found within boiler-plate HTML. Regardless, the concept can be generalized and adapted for other forums.

One script collates words:-

cat /path/to/saved/soylentnews/discussions/*.html | ./collate.pl > collate.txt

The other script allows predictions to be compared against actual scores:-

cat /path/to/saved/soylentnews/discussions/*.html | egrep '(comment_score_|comment_body_)' | ./estimate.pl collate.txt | sort -r -n -k 2 | more

The second script can also be used interactively. This is minor source of amusement but is generally less insightful than browsing the collated statistics:-

./estimate.pl collate.txt

If you want to be argumentative, negative or defeatist then this script will confirm that you'll be unappreciated. For example, one of our resident trolls, SaltySpice, rarely scores above 1.2. In particular, from my cache of saved discussions, "Fuck MDC" scores 0.597.

begin 644 soylentnews-score20181224.tar.gz
M'XL("`ZZ(%P"`W-O>6QE;G1N97=S+7-C;W)E,C`Q.#$R,C0N=&%R`.U76U/C
M-A3.LW^%-KC$(0FV<R$MP6EW=K?MSG1X`*:=:1P8)Q%$7<?.2DHAW0V_O4>2
MKT`&.LNTW4'?0VR?<W0NNGQ'8?$ZQ!&/\#5KL6E,<=MQOW7;[:X]C<,PX'A_
M&5:^$`Z@W^N)I]OO]HM/`;?;Z5;<3M=UX.>@`W9NO^\Z%>14_@6L&`\H0A5.
M"?NXPEN+?4S_E6+GE;UBU)Z0R%YB&J+6;X:Q@T[5KCB&78'>Q(L%O*-3L3O0
M.\;)(N`Q%5932I8<G<5@(_<*^IDP4)%I$"IS!E;6F[K84^ALCL$N8C'E9+78
M-W9`EVXV)!:!&P;%'U<$HM1",H&-5QL8QO6<A-B:X4L2X9EEAO#PCD[/WKX_
M'M;KZ).!D#F-5Q%O-`;P3BX3DUO[:-[U)Z/SX7AO:).8*5NPYOB&>])(#-@@
M'++B*#FB.E5%7\@C<3%R6M^-&]7[SG87P0=\P3%=*`?U@12#/XO!)`34^D$H
MZT?M^N?/EBG>1\X815"A]%S+Q>[XU:U][K>^5\%,&Z*D81!:4B*6X.SMNY,3
M5#6=0X0IC>DA$E&3&3A$TR"*8HYF6'@4"AEDWX^J@\0/OB'<<FZZ;I+I1I7!
M<%)JEDS]"9,SB6?K[7,#246<1"OLN=)7LCJIM+0>^_F"E-?0MV?DS^'(CWPZ
M;A2]E^=>^*BG->X&RV6XOI@&2RN7)6R62_+TG'PJ-L;&V)VM%G*D47D!8%OY
M'ZN3_@P-X#'^%V2?\G^GXU2<MN,<]#3_?SW\GTBQ,OIRTH^7.++>'S>K1\A\
M??+3KT":53B1#_:"O!%8YC4F5W/>5'S8E%Q6]]@R!-*S1\CG"8_$S9RME>TG
M:;OQU)>4*U^90GT.@""F8<PP9"<HPI1GQJNU:H,G=*J7WF>2V4H3R5H,3GL!
M<$Y*QUM*OD_NE^`RF,YA<(B1]0&O&?H&U'E-,$O9HB2+#::;0M4J<L/+UESH
M[:+QG@DNU6M:XJ90&(0HK</0R9T+U[974@]*LZUFI5GU>;7)5A/&J27&-)UF
MKZZD?\0DLFJHUE3#09C.=-JTC.?G?W4:GX-C'N/_7K>=W_\/'''_=SJNYO__
MCO]_(1,:T#7Z,:9/[`7_A/!ADZ/L=*LK_#*@P<)C<W*IR%=]W[*$)X>V?05T
M6%1Q:H]>MWX?VZ.@]=?8+H\:G0.Y@=QO^37@>V1?E0>S8C=X0'N>J6U[ZT@S
MU<E3F;89)+J+-);'G&*^HI%H&[+LC+UDV8*G/,5C9193YSQCQ1#+I/Q:(PV9
M2T%H%J093:G_0QOC48*4E_)TU+!3(*Y,ZG6*;'.OJ.S?@UI+1?+96MXQ3B[B
MTO21S(H$#.1\AX%+9%U4[ZD,!@\G*V[W#P07&Q8E&8BH:0Z*HJNE8#XO9@9?
K\$@8^4Y`]X7\B]#0T-#0T-#0T-#0T-#0T-#0T-#0T/B_X6]42<M9`"@`````
`
end

(Usual instructions for uudecode process.)

███████.com is a proper stealth mode Dot Com style start-up with more Aeron chairs than people based in Central London between the Ritz Hotel and Fortnum & Mason. We seek someone who:-

• Can write POSIX C to a subset of the MISRA standard to target GNU/Linux, BSD, MacOSX and the cheapest micro-controllers we can find; probably 16 bit AVR or possibly 32 bit Xtensa.
• Can write useful documentation in US English as plain text, HTML or OpenOffice format.
• Can occasionally invoke horrors written in PERL4 and make.
• Is willing to solder, flash and test micro-controllers.
• Alternatively, wants the thankless task of implementing a Unix GUI.

We strongly encourage part-time tele-work, part-time hours and job sharing. We encourage you to use your favorite text editor and email client. Furthermore, we:-

• Prefer HTML2 rather than HTML5.
• Prefer UDP rather than TCP.
• Prefer IPv4 rather than IPv6. (This seems to get howls of protest from people don't understand RFC1365.)
• Prefer wired rather than wireless.
• Prefer Unix rather than Windows.
• Prefer RISC rather than CISC.
• Prefer compiled programming languages.
• Prefer strong typing and static checking.
• Attempt to follow the 12 Step Joel Test without being a cargo cult.
• Are strongly skeptical about cryptographic currency and weakly supportive of block-chain.
• Are strongly pro-privacy.
• Are strongly against cloudy computing.
• Appreciate junk food including our local source of caffeinated bacon.
• Prefer jello rather than Trello. (Sorry, Joel.)
• Prefer slack rather than Slack.

Indeed, we are chilled and low drama with collective experience of South California counter-culture and hydroponics. We are very accommodating towards alternative lifestyles and disabilities. For us, wisdom is highly prized and we appreciate that wisdom is often accompanied with ailments of age. Regardless, we retain some exhuberance and immaturity. Last week, in jest, I was intentionally struck with a scrunched food wrapper. Also, a spherical cow plushie was thrown around the office. This week, it is joined by a spherical sheep, spherical ladybug and spherical tiger. We are willing to reduce such shenanigans if people require concentration. Unfortunately, this is the first of many steps going from innovation to stale, hollow corporation. However, we are doing our utmost to ensure that people don't have to tone down appearance or mannerisms for our traditional New York investors. (We're not encouraging an us-and-them mentality but you can definitely tell who has the money and who has the ideas.)

We've been given a very long leash to make the most awesome, reliable, valuable, legal product and/or service by Mar 2019 with ongoing work subject to outcome. Personally, I'd like to deliver (five sets of) micro-processor, operating system, filing system, database, streaming video desktop remoting system with HDR, 3D sound and much more besides. However, in the four months covering Christmas, it would be optimistic to deliver a draft version of one piece. Indeed, even in the long-term, we cannot be all things to all people. So, where is the best place to concentrate effort? Consider broad market trends:-

• IBM peaked in the 1960s. It is still around but it isn't feared.
• With the licensing of PCDOS in 1981, IBM ultimately aided its successor, Microsoft. Microsoft is still around but it isn't feared.
• With the distribution of Internet Explorer in 1995, Microsoft ultimately aided its successor, Google.
• Google has peaked and has difficulty obtaining and retaining staff due to ethics, censorship and brogrammer culture.

In general:-

• Trends oscillate between server and client. IBM in 1960 (centralized) to Microsoft in 1985 (decentralized) to Google in 2010 (centralized) to ??? in 2035 (decentralized). Between IBM and Microsoft, infill all of the mini-computer companies (DEC, Data General) and dedicated word processor companies (Wang, Commodore). Between Microsoft and Google, infill all of the power desktop companies (Intergraph, SGI, Sun, Dell). From Google to its logical successor, infill rich client companies (Apple, Amazon, Facebook). Indeed, Apple's peak valuation and social media valuations rely on a hybrid of local software and vast server farms. MAGAF [Microsoft, Apple, Google, Amazon and Facebook] have more than six million servers, 10 million storage units and create an electrical base load which requires 10 nuclear power stations.
• Companies at the extremes of the trend become monopolies before falling into maintenance mode. Many of the remainder chase a changing market.
• Tech monopolies create to environment for their successors.
• A successor is *almost* the opposite of the predecessor.
• Hybrid apps have reached an inevitable peak. This is likely to concede to systems with more autonomy.
• The current environment suits development of a decentralized, privacy respecting system. This is increasingly likely over the next 20 years.
• Security and privacy are awful. This is due to externalized costs and historically weak regulation. Retail banking is a relative leader in security; in part due to disproportionate impact on its own staff. However, legislation is tightening in North America and Europe; in part due to the vitriol on social media towards legislators. This regularly includes credible death threats to immediate family. In the UK, several people have been arrested or imprisoned for credible threat of rape towards politicians. Given that politicians are rarely sexually attractive, it should indicate that the problem is widespread.

Security and privacy are increasingly marketable. However, it is like King Canute attempting to fight a rising tide. It cannot be commanded or legislated. Best option is to isolate. Cannot redact data which has escaped. This is espcially true if it would be good manners.

With a rising tide of privacy infringement, the perfect product and a hypothetical contract with one bank does nothing for customers of other banks, retail breaches, medical breaches, government breaches or any other problem. There is no silver bullet which covers all cases. Regardless, anyone can be "king of their own castle" and home security is a domain where everything is within reach of a customer. If implemented correctly, the customer has complete control. Furthermore, an installation should function beyond the viability of the manufacturer. Convenience features, such as remote access, rapidly devolves into a security quagmire. Convenience features may also require more than four months of effort.

Home security dovetails with the trends of decreased driving and driving licenses, decreased drinking in bars and nightclubs, increased parcel delivery, increased food delivery, binge watching drama, increased remote study, streaming exercise classes and generally following the trends of the film: WALL-E and the short story: The Machine Stops by E.M. Forster. Overall, there is demand for a home automation system and a home security system which connects to a big screen television. The laziest method to implement such functionality is as a Kodi module. Kodi is typically deployed on a Raspberry Pi where extensive interfaces (USB, I2C, GPIO) are often unused. Obviously, this would be a lame, unreliable and insecure implementation but it would certainly be an impressive and convenient demonstration. Geeks may find Kodi's menu system to be quite tedious (and laggy). So, an expert mode is definitely required. It would be particularly useful if a scripting interface was available.

The objective is to have one system which can be used to watch films, watch television, listen to radio, listen to albums and also dip lights, adjust room temperature, brew beer, water plants, feed fish, set burgler alarm, review external security cameras and check who is at front door. Further options are possible but these compromise privacy. It is possible to forward dubious camera footage. It is also possible to integrate a house intercom, voice activated commands and video conferencing. However, interior cameras and microphones are specifically excluded because that's creepy. Even the creepy king, Mark Zuckerberg, covers his laptop camera but he'll willingly make a buck by pointing a camera at you. Our intention is to only integrate functionality which increases security without compromising privacy. How is this achieved? Everything from television to leaf node (lock, trip switch, light bulb) is fairly unconstrained. However, it would be a significant bonus if a system is:-

• Cheaper than a NanoLeaf.
• More open than Philips Hue.
• More reliable than both.

It is also a bonus if functionality overlaps with industrial, office, retail, automotive or aerospace use. For serious use, a Raspberry Pi would be replaced with a 1U rack server with ECC RAM. For very serious use, servers could be clustered with fail-over. Leafs should be as economical and as useful as possible. A friend who runs a virtual reality start-up gave a demostration of an Xtensa micro-controller with USB, GPIO, wifi and 256KB flash serial ROM. When flashed with a Lua interpreter it is possible to run scripts which are stored on a local filing system. Virtual serial over USB also allows access to a Lua REPL. From this, it is possible to join a WPA2 network, run a DHCP client and run an HTTP server. Indeed, without stopping the HTTP server, it is possible to modify the URL-space so that arbitrary paths may be handled by arbitrary functions. A fall-through case serves static files from the local filing system. My friend estimated that the minimum configuration used less than 64KB ROM.

What can be achieved in a smaller space? Chess computers have been implemented with 128 bytes RAM and a popular chess program for the Commodore VIC20 was supplied as a 2KB ROM. Graphic competitions have a 4KB category with impressive entries. (Although, Subdream's 64KB Raum Zeit octree renderer and Porter Robinson & Madeon's animated music video for Shelter remain favorites.) A smaller system is not an academic problem. A smaller system uses less energy. A smaller system also creates savings which ripple through manufacture, wholesale, retail and integration. Reducing cost of a micro-controller by 5¢ may reduce retail price by more than US$1. Alternatively, savings can be allocated to improved cipher suites. For many devices, almost any cipher would be an improvement. I hope that it is possible to include a CLI, graph library and SMART style EDI interface in a micro-controller with 4KB RAM and 16KB ROM. I also hope to implement a network switch with 2KB RAM. Even if these estimates are repeatedly raised, the result remains competitive in a market with dual-core light bulbs, quad-core watches and where 1GB RAM is regarded as embedded.

Our preferences for hiring are as follows:-

• UID1111 who is bold enough to show work, introspective and, despite protest to the contrary, cares about detail, such as string buffers.
• UID2828 who is working on the closely related problem of vehicle networking.
• UID5227, Linux device driver contributor. (I noted the quality of this work at the time.)
• Anonymous Coward who shared a fast and concisely documented matrix library.

We are heavily constrained by time. We would otherwise like to exhaust these options before seeking people more widely.

I'd like to finish with a message to people with similar sentiments to SoylentNews' Not So Anonymous Coward, SaltySpice. (When did we start naming our trolls?) We strongly agree that there are too many oxygen thieves in software development and corporate administration. If they could FOAD, we'd make more progress. In particular, computer security is a sticking plaster for developers who don't care. From our unused collation of soggy.jobs, the most generous lower bound for fake or speculative job adverts is 5%. We would be wholy unsurprised if the majority of job adverts are fake. Particular ire goes to Infosys and IBM subsidary, Aspera, for listing fake jobs. Paraphrasing from email:-

Me: Are you *sure* you're rejecting UDP, file transfer, POSIX C programmer with professional experience of your CLI and GUI?
Michelle Munson, co-founder of Aspera: Yes.

That's like the alleged musicians, Duran Duran, who came second in a lookalike competition or the guy in Charlie Brooker's ScreenWipe USA who failed a screen test to portray himself in a fictional version of his own life. However, when unemployed people are expected to seek work but employers are not obliged to hire, don't be surprised when the result is nepotism, time-wasting and abusive practices. Also don't be surprised when the unviable dregs are advertised for an extended period. My personal favorite was a job advert for a tri-lingual, Adobe, Java, Cisco expert which paid San Francisco minimum wage. This idiocy is made significantly worse by "signposting" "services" which provide another level of indirection where none is required.

It is damning that the majority of programming jobs are obtained via family, friends and interview prowess. Ability to do the job is very secondary. Caring about the project is also secondary. I would be inclined to hire SaltySpice in preference to MDC but, c'mon, meet me half way. Give me *anything* which puts you ahead of my least competent, least employable ex-colleague.

From the Wikipedia article about spherical cows:-

Milk production at a dairy farm was low, so the farmer wrote to the local university, asking for help from academia. A multi-disciplinary team of professors was assembled, headed by a theoretical physicist, and two weeks of intensive on-site investigation took place. The scholars then returned to the university, notebooks crammed with data, where the task of writing the report was left to the team leader. Shortly thereafter the physicist returned to the farm, saying to the farmer, "I have the solution, but it works only in the case of spherical cows in a vacuum".

Since then, a spherical cow breeding program has been successful.

Spherical cows are often the stereo-typical black and white Holstein Friesian cow pattern but may also be the brown and white Jersey cow pattern or numerous other patterns including plain white, brown or black. Unfortunately, spherical cows in a single color are often mistaken for mutant hamsters and shot on sight. This contributes the Holstein Friesian pattern being predominant.

Although genetic throwbacks may exhibit hooves, ears, a tail or an undesirable ellipsoid profile, a pure-breed cow is completely spherical. They are often grown in proximity to cuboid watermelons; an innovation which makes more efficient use of space within a domestic refrigerator. A side effect of the spherical cow breeding program is that spherical cow milk has an unusually high surface tension. The milk often sits as droplets rather than a homogeneous liquid. This may be corrected with a very small quantity of surfactant. However, this additive often falls afoul of legislation regarding dairy products and therefore spherical cow milk may be restricted to illicit channels; similar to raw milk.

A perfectly spherical cow has no (identifiable) legs. Indeed, a perfectly spherical cow in an idealized meadow has no cross sectional area with the ground. Therefore, spherical cows are frictionless with the ground. In a valley, a spherical cow placed near the top will roll down the valley and up the other side. If the valley is in a vacuum then this process may continue indefinitely. If the valley has idealized wind resistance, spherical cows will typically be found in a local minima. Furthermore, a herd will be arranged in a manner approximating sphere packing. If idealized air is still, a spherical cow placed near the top of a valley or hill may be found in one of the surrounding dips. However, if wind is strong then an entire herd of spherical cows may be stuck against a fence. It is strongly recommended that any fence exceeds the radius of the largest spherical cow and can withstand the force of an entire herd during strong wind. Despite being domesticated, it is completely unknown how spherical cows eat or breed. The leading theory is that spherical cows are trans-dimensional beings but agricultural researchers are unsure how to test this hypothesis.

The easiest method to determine if a spherical cow is pregnant is via the increased radius of the cow. However, it is incredibly dangerous to be around a pregnant spherical cow. When a spherical calf is born, it may shoot out like a ping-pong ball. It may travel an incredible distance across countryside; 200m has been recorded. For this reason, spherical cow birthing sheds are strongly re-inforced and the interior is covered with damping material to reduce the total number of ricochets. A newborn spherical calf is approximately 1/2 of the radius (1/8 volume) of an adult. Spherical cows are most likely to be born during a full moon. This may be the basis of an unexplored joke.

A spherical calf grows to full radius over 18 months or so and it sufficiently mature to breed in its second year. In idealized conditions, breeding pairs of spherical cows may increase in a manner which strictly follows a Fibonacci sequence.

Spherical cow leather was the preferred material for equipment used in many types of sport. Most typically, it was used for sports balls due to its equal weight distribution and equal curvature. It was also the preferred material for chess-boxing gloves. Nowadays, sports are more likely to use synthetic materials; especially in upper leagues. Foreseeably, this has contributed to a decline in spherical cow herds.

Spherical cow steak is of unusually high quality. Spherical cows may have recessed legs and therefore edible parts of a spherical cow consist almost entirely of rib, back and belly. Approximately πr³/3 of any given cow is rib meat. However, to properly sear a spherical cow rib eye steak, a particularly large circular searing pan is required. Some advocates of paleolithic diets are particularly keen to promote the benefits of spherical cow steak. One of the disadvantages of intensive farming is that beef, pork, chicken, turkey and other meat is almost exclusively single sex. This may have long-term consequences for all humans. To avoid adverse effects of xeno-steroid hormones, it may be beneficial to eat historical proportions of male and female beef. Given the foreseeable difficulty of identifying male and female spherical cows, males are often allowed to grow to adulthood. However, the increased cost associated with spherical cow farming may have fringe health benefits.

Attempts have been made to cross-breed spherical cows and flying pigs. So far, this has been unsucessful. However, this may change as genetic engineering advances. Nevertheless, it has re-ignited a debate over collective nouns. It is generally accepted that the collective noun for pigs when flying is a flock. However, accepted use beyond this case is varied. In the case of flying cows, moderates suggest that the portmanteau, flerd, be used in all cases. Unfortunately, this reasonable proposal has been met with almost universal disdain from farmers. While diary farmers look forward to new varieties of stationery, dairy farmers are concerned that any re-classification of farming may adversely affect economic subsidies for agriculture. This may further contribute to the decline of spherical cow breeding.

It is possible to make a dozen spherical cow plushies for £20 (US$30). This is a popular hobby project because they are relatively easy to make. Only cuboid plushies are easier to make but both require relatively large amounts stuffing. Spherical cow plushies require two self-similar pieces of synthetic fur material. Ideally, the synthetic fur material will have a Jersey cow print. (Among professionals, Jersey print material is known as plane cow plush.)

The two pieces fit together in the manner of a tennis ball or hacky sack. In the trivial case, this requires an ellipsoid template consisting of a semi-circle joined to a rectangle and another semi-circle. For each piece of fur, ignoring allowance for sewing seams, the straight and curved sections should have the same length. The straight section of one piece should fit with the curve of the other piece. This sets the constraint for the relative lengths. For a semi-circle with radius r, the rectangular section should have width 2r and length πr. When a piece of material is curved to fit the other, it would be reasonably assumed that πr is the curved distance between polar opposite focus points of a curved ellipsoid. However, this is not the case.

Making the reasonable assumption that the seam does not stretch but that synthetic fur may stretch to become approximately spherical, it is neccesary to include the (appropriately named) Skinner's constant or other Flannagan Finagling Factor. The four inflection points (where the seam changes direction) are equally spaced around a great circle. Relative to the radius of the ends of an ellipsoid, the inflection points are sqrt(2)r from the center of the sphere. Therefore, for a tennis ball, spherical plushie or similar with a required radius r, all the previously stated measurements for ellipsoids should be scaled down by a factor of sqrt(2) (approximately 1.41). Common worked examples follow.

When making cutting templates from paper, length should be 1+π/2 of width - prior to ends being rounded. For an A4 sheet of paper (297mm×210mm), cut to 297mm×116mm before rounding ends. For a US Letter sheet of paper (11 inch × 8 inch), cut to 11 inch × 4.3 inch before rounding ends. This is suitable to make a plushie with 16cm (6 inch) diameter. Template and/or material should be cut with allowance for seams. Professionals typically include a 6mm (1/4 inch) seam in the template. A wider seam (included or excluded in the template) may be desirable due to inexperience and/or to increase sturdiness.

It may be useful to tack a few stitches where the straight/curved point of one piece meets the curved/straight point of the other piece. It may also be useful intermediate points. Each tack works as a rip-stop. It also reduces mismatch of material when sewing along seams. Optional adornments, such as legs and tail, can be sewn separately and then stitched into the seam. The tennis ball seam provides suitable placement for four legs, two ears and a nose. Miniture cow bells can be purchased in bulk from morris dancing suppliers.

The cheapest stuffing by volume is synthetic car washing sponge. However, the result may feel lumpy and distinctly not spherical. One sponge may be cut in half and each half may be trimmed into a dome shape. Another sponge may be cut to approximate a disc. The three layers (dome, disc, dome) may be squeezed through the hole of the synthetic cow hide before the final section of the seam is sewn closed. While attempting to close the seam in the least obvious manner, increase the difficulty of the task by simultaneously considering the hairy ball problem.

Worryingly, the result meets European safety standards for flammable material and also toys with small parts. However, it does not meet the widely flouted labelling regulations. Without exception, all waste material (fur, foam, cotton) may be stuffed into a spherical calf plushie. However, this does not meet safety standards and should not be given to the type of child inclined to stuff things up its nose; nominally a child less than three years old.

Synthetic fur may be occasionally brushed with a wig brush. Stains can be removed by soaking the surface material only. Deep ingress of water is likely to encourage mold. Synthetic fur is likely to curl or melt in a clothes dryer or under an electric hair dryer. Do not spin wash or tumble dry because it makes plushies very dizzy.

This is written primarily in response to JID3525 by UID2828 but is of general interest:-

People are quite interested in your work with Arduino systems. Whatever you've achieved, people will pick fault. However, I understand that you are primarily interested in automotive applications. You also want something which is more open, accessible and educational than the predominant CANBus systems.

CANBus DRM

People may wonder why a serial protocol is neccesary in a vehicle when legacy designs had little more than a spark coil. Engine emission limits almost force a central computer. From there, security and convenience features make it worthwhile to spur numerous micro-controllers. A modern car may have more than 100 micro-controllers. The alternative is more than 3km of wires between wiring looms and an infeasible array of fuses and relays. At some point, it is easier to implement everything with software and a network topology. From Robert X. Cringely on Mon 5 Jun 2017:-

I was shocked five years ago, for example, when my friend Shoichiro Irimajiri told me that automobile wiring harnesses had reached the point where they cost as much to build as the engine and transmission for the same car. Irimajiri-san, who built Honda's first U.S. assembly plant in Marysville, Ohio, was then a board member of Delphi, the world's largest maker of car parts, and certainly knew what he was talking about. The future trend, he explained, had to be for wiring harnesses to become cheaper by turning into intelligent networks with single wire pairs replacing dozens of wires with hundreds of connections.

However, manufacturers have used this transition to enact substantial lock-in and a large amount of unproductive busiwork which is largely hidden from customers but extracted via increased repair costs. One of my ex-housemates was a car mechanic and from this I learned that car repair has become as tedious as Microsoft Windows licencing. Specifically, the Windows® Genuine Advantage*†☠ requires re-activation of a licence key after changing components such as processor, harddisk or motherboard. Most people out-source car repair and so they are generally unaware that the same DRM tedium is required after replacing an electric mirror. All such components are connected to a common one-wire serial bus and almost every car from almost every manufacturer uses the same bus network protocol. The specification for the bus is open but the address ranges, authentication and data are not consistent between manufacturers. If a person wonders why car repairs are so expensive, it is because an independent mechanic has to:-

1. Diagnose fault.
2. Obtain DRM component from authorized supplier.
3. Fit component.
4. Connect authorized diagnostic computer. (Purchased and maintained at own expense.)
5. Obtain first part of authentication handshake.
6. Telephone premium rate support line and get put "on hold" for 30 minutes.
7. Complete cryptographic authentication.
8. Discard old, unservicable component.
9. It may be possible for an erroneously substituted component to be installed in another vehicle. The manufacturer may set arbitrary limits on this process. The manufacturer may change the limits unilaterally. This includes revoking all support to a mechanic, vehicle or model.

Do you own a modern car or is it a smartphone on four wheels? Even niche luxury cars use CANBus with unwanted DRM. I investigated luxury car manufacturers and found that it is mostly a business of out-sourcing. This includes body panels, dials, engines, gearboxes, brakes and car hi-fi in addition to more mundane components, such as screws, switches and tires. (The history of CAD/CAM software suitable for curved body panels is tied to Prime Mainframes and, via Tandem Computers, back to our interests in reliable and documented serial protocols.)

Competitor Analysis 1: Vehicle Manufacture Assembly

McLaren specifies its own engines (but certainly doesn't test them where it assembles cars). The remainder send out CAD files or purchase from a catalog. Larger manufacturers have their own tweaking divisions so they can upsell their own products. The most ridiculous example is the heavily marketed Audi R8. Each edition is only available in one color. The trashy flecked orange convertible model was product placement during the test flight in the first Iron Man film where it is priced at US$135,000. However, it has a ZF transmission, tweaked with VolksWagen mechatronics before being assembled in a (hopefully hardened) Audi chassis. The transmission alone visits three or more sites during manufacture. This is an example of supply chains becoming increasingly global. Another example is Cherry Keyboards which was founded in the US before re-locating to Germany. It produced many switch and sensor products for automotive use before these products were amalgamated into ZF.

One safety feature which was initially available on an Audi R8 (and now more widely available) is headlights which selectively ignore rain and snow by using infrared sensors and a matrix of white LEDs. This system functions at 70MPH. I explained this to a person at my local makerspace and was told that it was technically infeasible. I replied that much of the required functionality has existed in optical mice for decades. It may have to work at a higher speed, with more elements and some calibration between sensors and lights but it is certainly feasible and commercially available.

I was less impressed with Aston Martin (which uses odd numbers for car models to avoid confusion with the number of cylinders in an engine). This was before the grifter banksters tried floating a company which has been bankrupt on seven occasions. Ford has owned various brands including Volvo, Jaguar, Aston Martin and Shelby. Few question Ford's apparent absence from self-driving cars but several advances have been made with Volvo cars and trucks. The most successful Aston Martin product, the DB9, is a salvaged failure and famous example of product placement fakery. The DB9 is also the least advanced product when compared to contemporaries. When Ford owned Jaguar and Aston Martin, a Jaguar design exceeded component cost and was transferred to Aston Martin. That's why a DB9 has a Ford chassis. It is also why the DB9 was manufactured assembled in a separate factory. However, the "DB9" which appears in the James Bond film: Casino Royale is a mock made from a DB7. Technicially, it is the best DB9 ever made - especially after mock DB9 exteriors were rolled for a stunt. The special Chinese edition Virage 88 doesn't come close.

Barriers To Entry

There's an old joke aout a man browsing in a Rolls Royce dealership. He asks the sales respresentative why the cars cost so much. The answer is "Try making one for less, sir." However, that isn't true any more. A back-of-an-envelope calculation shows that the Bill Of Materials for a car with the performance of an R8 and the strength of a DB9 is less than the retail cost of either - even for a single unit.

There are a few limitations for the aspiring sports-car manufacturer. For example, European safety regulations require approximately 30 vehicles to be trashed (at manufacturer expense) in front impact, side impact and similar tests. A similar number may be required for other jurisdictions. However, in the UK, regulations only apply to fully assembled vehicles. The UK has a long history of kit cars. The most famous is the Caterham Seven which was featured in the title sequence of the television series: The Prisoner. ("I am a name, not a number!" and then drives car model seven. Yeah, you're a unique snowflake - just like everyone else.)

In the UK, if a customer makes a token effort with a wrench and assumes responsibility for final assembly, a vehicle can be certified as street legal on a individual basis. I've seen some dangerous stuff certified as street legal, including a recumbent motor-trike with exposed continuous variable drive. It is also possible to pack approximately twice as many vehicles into a shipping container if they are sold in pieces.

I am concerned that the disparity between material cost and sale price occurs prior to the inflection point from hydrocarbons to electric. This could be like the situation where micro-controllers become powerful enough to drive VGA monitors at the point where VGA gets deprecated and replaced with something less accessible, less secure and more centralized. Or where analog headphone jacks and US$1 amplifiers get deprecated in favor of less accessible, less secure and more centralized digital protocols. Now that the little guy can make a better sports-car, the rules may change. Unfortunately, the result is likely to be garbage.

An electric train is sensible because the energy is supplied externally and therefore it doesn't carry its fuel. However, an electric car - even with the energy density and fueling/charging time of hydrocarbons - competes at a 2:1 disadvantage because an electric car carries the deadweight of a flat battery. A diesel (turbo) electric car is worthwhile. Specifically, an engine block connected to an alternator and an electric motor on each wheel with regenerative braking. This would be a hydrid with no gearbox or differential. Electrical losses are comparable to mechanical losses but it swaps precision machining for precision etching. It also provides the optimal acceleration of an electric car and the urban efficiency of a hybrid.

The next limitation is the industry specific CAD/CAM licencing (and formats) for body panels and interior. This can be skipped if a chassis is retro-fitted or a chassis has a strikingly angular appearance, like an F-111 Aardvark or the USS Zumwalt. I appreciate this loses mass market appeal but I merely suggest what can be trimmed from low volume production.

Competitor Analysis 2: Serial Protocols

The remaining limitation is the serial protocol which is used ubiquitously by car manufacturers. I don't want to diverge from CANBus to be a contrarian. I only want to diverge to reduce cost and risk. I want user-servicable parts - especially in a tight situation. I believe this is your primary use-case. The ability to tweak and customize is a welcome bonus but the penalty is that it is time-consuming, like every other open source project.

What is the point of maintaining wire-level compatibility with CANBus when manufacturers don't use consistent address ranges or data formats and lock-out third-party spares via DRM? The most standardized part of CANBus is agricultural accessories. However, in this case, ass-hattery is moved elsewhere. Most notoriously, John Deere refused to honor the re-sale of a firmware licence. Competitors have followed this poor practice. This greatly affects the capital value of a tractor and routinely requires tractor mechanics to break the law.

Are there any other serial protocols which are suitable for automotive use? Commercial aircraft use ARINC429 and larger ones use AFDX which is effectively ARINC77 tunnelled over UDP over dual runs of 100Mb/s twisted pair Ethernet. Many military aircraft use MIL-STD-1553 - except the F-35 Clusterfuck which uses FireWire. (I wish that I was joking.) I2C was originally developed by Philips to reduce wiring in televisions. This was eminently sensible when Philips made the custom chips and the televisions. Nowadays, robots use I2C for servo control and ultrasonic distance sensing and it is fairly apparent that each node is an independent micro-controller. Beyond aircraft and robots, there is RC5 for infra-red remote controls (possibly via an RC5/I2C bridge), MIDI for music sequencing, DMX for stage lighting, DALI for ambient lighting, X10 for home automation, the defunct iLink for hydroponics and several mutually incompatible protocols for Internet connected devices.

I don't suggest devising one universal serial protocol because that inevitably creates another standard while doing nothing to handle legacy cases. Regardless, many of these control protocols have very small address-spaces; typically 8 bit or less excluding reserved addresses. Even IPv6 over IEEE802.15.4 with stateful header compression and self-selection of addresses effectively has an 8 bit address-space rather than the expected 128 bit address-space. The most inane example of the art is HDMI's slow bus with a 4 bit address-space. Of course, this means that a television which implements I2C and HDMI may have a protocol with 7 bit address-space inside the device and a completely incompatible protocol with 4 bit address-space between devices. (Bring back SCART. All is forgiven.)

It is hoped that a vehicle has less than 127 micro-controllers. However, a small address-space hinders independent innovation and hinders auto-detection of optional peripheral components. A Hamming distance greater than one also increases reliability.

General Purpose I/O

A general purpose I/O expander system is of general interest because they are often implemented poorly. Multiple incompatible systems exist for Raspberry Pi systems. For some domains, this would be acceptable if basic database theory was applied to process control. (Error detection, read locks escalating to write locks, atomic commits.) More often, it is written for children or halfwits. And it is an insult to children. The typical scenerio is something akin to "Hey! I've got two Raspberry Pis connected via Ethernet and I'm running out of I/O pins. *Surely* I can get a program on one Raspberry Pi to flip pins on another?" And this is how we get pigpio and the numerous incompatible variants. Please don't use them; especially on hardware lacking parity RAM.

I first encountered this scenario when working with PIC micro-controllers. I found that PICs with more I/O were disproportionately more expensive because they came with additional RAM, ROM and interfaces which were not required. My idea, which was not unique, was to have an open or closed loop where all-but-one of the chips had identical firmware. This leads to the possibility that additional I/O pins don't have to be in the same place. For example, each node may control a cluster of car lights. The next consideration was addressing.

I didn't want I/O expanders with 16 I/O pins and then lose 8 pins to a hard-coded address. (Nor some variation thereof.) Nor did I want to maintain consistent addressing. So, I devised a protocol of three byte triples of the form <device>, <command>, <data>. In all cases, <data> is provided even if it is a dummy value. This allows upward compatibility and provides a placeholder within a closed loop serial stream for read operations. The part that I thought was particularly cunning was that the device number would be decremented by each I/O expander. It was hoped that every device could test against zero. This would allow the I/O expander at the head of the chain to be device zero. All subsequent devices would be numbered contiguously in ascending order and no I/O pins would be used for device addressing. Unfortunately, this scheme is very susceptible to bit error - before or after a triple is processed by the intended device. There is also the very fundamental problem that each byte of a triple may be mis-interpreted. For example, a command may be interpreted as a device number.

After many iterations of eliminating obvious flaws, I settled on a protocol which can be concisely be described as ATM Lite with a 28 bit address-space and 24 byte fixed-length cell bit-stuffed into a 256 bit frame. Somewhere along the way, synchronous communication became asynchronous. Actually, a dream caught that design error. In the dream, I was at my local makerspace and demonstrating a four node configuration but it didn't work. I woke and thought, "Phew! It was only a dream!" I then had the worrying realisation that the test case was depicted accurately and that designs over the previous two weeks didn't work.

My primary use-case is hydroponics and home automation - where everything is grounded. However, my choice of bit-stuffing would look hopelessly naïve in an ungrounded automotive or aerospace environment. A quick fix would be Manchester encoding where every bit is echoed with the reverse polarity. However, that halfs the bit-rate. In typical cases, such as bit-stuffing with an even Hamming distance, it is possible (and often desirable) to have an equal number of zeroes and ones to represent each valid symbol. This maximizes current balance and discourages accumulation of static. For my purposes, unbalanced bit-stuffing is sufficient. Indeed, it is desirable for low-speed software implementation. This would be re-considered if an ungrounded and/or high-speed, hardware implementation was ever required. Similar choices define the different generations of USB, although neither USB 1.x, 2.x nor 3.x are current balanced within one wire.

It is reasonable to assume that your protocol is somewhere between a toy implementation and a carrier-grade communication link. I hope that you have considered error detection in some form even if you have not investigated the art thoroughly. Indeed, I strongly recommend against a thorough investigation because you'll rapidly encounter an exotic zoo of unbalanced LDPCs where you can get too smart and zap your electronics. Despite this, minor tweaks to your protocol may disproportionately improve resilience. However, I would be the first to concede that there are diminishing returns to this process.

Circuit Design

I presume that you are a hardware engineer and that your designs are distinctly above average. For automotive applications, people wrongly assume that 12V is the maximum Voltage. However, 14.4V is typical for battery charging and 60V transients are typical despite surge suppression. Semiconductors, typically rated at 30V, are rapidly destroyed in such an environment. Irrespective of firmware quality, your designs would have increasing utility with increasing resilience to automotive electricity, US mains electricity, European main electricity and three phase electricity.

Your circuit board designs would also gain utility by providing mechanical compatibility with the fixing holes of the most common credit card computers. I assume that your design is considerably more compact but the (optional) additional area could be used for MOSFET heatsinks. In this scenario, a compact (automotive) version would be sufficient to switch headlights whereas a credit card (home automation) version would be sufficient to switch a kettle. Indeed, it may be trivial to design boards so that they can be crudely cut to the required size after manufacture. This would be particularly true if you provided, for example, eight MOSFETs and only six were required.

Successful Propagation

Your motive is neither fame nor fortune. You merely want to see your work being used rather than ignored. In particular, you are considering open source partly to obtain the most reach within formal education environments. I was inclined to agree with this approach until I suggested similar to a friend. My friend is extremely hostile to dumping a life's work on the Internet and finding that it is completely ignored. That would be particularly dispiriting. Instead, I strongly suggest charging about 10 times the component cost. This is not so that you can become an evil capitalist, spend the money on porn, hookers and blow or circuitously defend against litigious parties because you are worth suing. It is to cover unforeseen setbacks and spend the remainder on promotion to further your ideas. From the (cleverly named but concise) Lazy Man's Guide To Riches, retail price should be a minimum of three times the cost price (including materials, assembly, instructions, packaging, delivery and support) with the intention that more than half of the turnover is spent pursuing further sales. I understand that it appears pointlessly circular to take money to spend money to take money - while skimming whatever you can to cover equipment failure, manufacturing defects and calamity. However, if you do not engage in this practice then you exclude a very obvious method to leave a legacy.

I discourage the fixation on youth education. Without effacy, we attempt to stuff maximum knowledge while also expecting social success, civic responsibility, health and athletic prowess and many other goals in preparation for a career, financial responsibility and family. The pressure leads many to give up and some are driven to suicide. Although we should strongly encourage numeracy and rationality, adding proficiency electronics and micro-controllers is detrimental, especially for a kid who doesn't have an aptitude for STEM. Even in the case where the computer game, Minecraft, became a school activity, use of Minecraft for personal projects rapidly became uncool. Do you want your field to be treated similarly?

Neither do I suggest that you target efforts towards military veterans. For many people, joining the military is the best economic option even if aptitude lies elsewhere. However, it rewards failure to withold a STEM education until someone becomes a violent aggressor. Nor are they in the best state to learn after injuries; visible or hidden.

If you must target efforts, concentrate on people who are in mid career; who are stable, competent, unhurried - and heading towards divorce and homeless alcoholism when their job is suddenly offshored or automated. If you charge accordingly, you could personally extend credit to people in decline. In aggregate, you would cover the losses.

Suggested Strategy

Hardware and firmware may have independent merit. If nothing advances the art, overt secrecy may not be an advantage. However, if you strictly retain copyright, you have the descretion to gift licences to worthy causes. You can also avoid wasting resources by ignoring unofficial clones. It may be an advantage for clones to expand the market but it becomes problematic if they leave you with liability, take rewards and leave you in poverty. In general, if someone wants to do the money-getting while spreading your work then it gives you more time to use your talent. However, you are far less likely to spawn imitators if they cannot stake a career and mortgage on it. That definitely won't happen if you act like a hobbiest.

Anyone wishing to make DRM-free micro-controllers for automative applications may use any of this text with or without attribution.

After serving me well these past five years, my good old Nexus 7 is finally going to retire. I've gotten a new tablet, a Xiaomi Mi Pad 4, which is more or less the same form factor, just slightly bigger, and the hardware specs are really sweet. But I should have been a tad more careful reading of both the place I bought it from and the stuff on XDA about it. First off I'd bought a version that has a Chinese ROM. This shouldn't be such a big deal, except that the damn thing doesn't have Google Play or any sane app market. There's the Mi App Store, and while it has some of my essential tablet apps, some omissions are rather notable, not even including including something as basic as a workable FTP client. Since one of the main uses of my tablet is to play video files I download from a network storage server at home, this is essential. Maybe there is something there that could do that... if only I could read Chinese. F-Droid seems to help somewhat, except that there still isn't a simple FTP client that will let me download several files in a batch. The closest thing there seems to be Ghost Commander, but it downloads all the files in parallel and can't display progress for all of them. This is ridiculous. Even the web browsers like Firefox are in Chinese! The keyboard is also the Google Chinese keyboard, and there's something of a dearth of alternative keyboard apps, at least no keyboard that will let me type numbers on a password screen lock without pressing a modifier key. I fucking hate that. There is a lot of space on a 1920×1200 screen that is 8" across, I could use a row of number keys at the top thank you very much! There certainly seems to be nothing like the Hacker's Keyboard on Google Play, which you can actually use vi on a remote shell with because it gives you that all-important Esc key. :P

The other thing is that locked bootloader. I could have flashed a custom ROM to remove the useless Chinese cruft. While Xiaomi will allow me to unlock it if I want to (after giving them something like $250 for it I expect no less), they have an irksome requirement. They are making me wait 720 hours (yes a whole fucking month) before they will permit me to unlock. The fact that they impose this delay on you for your "protection" is extremely irritating and paternalistic. Sure, if the device is stolen it becomes essentially unusable if the bootloader is locked but there are better ways to make that happen I think. Given the lack of GPS or LTE circuitry in this version of the Mi Pad 4 I have, if that happens I don't think it's going to help. So do I own my device or not? Seems they want to hang onto it for a month longer before they'll actually give it to me.

In the meantime I've managed to sideload a few of the essentials after using Titanium Backup on my Nexus 7 to convert them into APKs. They largely seem to work.