SoylentNews

I'm currently busy because I received start-up funding (and use of a ridiculously swanky office). The first development deadline is on Mon 31 Dec 2018. The second development deadline is on Thu 31 Jan 2019. The third development deadline is on Sun 31 Mar 2019. Funding currently lasts until Fri 31 May 2019. Unfortunately, this has greatly curtailed my annual fun programming project for Christmas 2018. Previous efforts have taken three days. This took three hours.

I have implemented a small system which predicts the final moderation score of a message on a forum. This uses some search engine theory, such as postings and repetition limits. However, it doesn't use n-grams, stemming, weighting by word frequency or Bayes theorem. Indeed, a significant amount of theory has not been implemented and the result only makes predictions with a relatively weak correlation. This implementation only works with the output of SoylentNews and is heavily dependent upon certain attributes currently found in the HTML. It is also heavily dependent upon line breaks typically found within boiler-plate HTML. Regardless, the concept can be generalized and adapted for other forums.

One script collates words:-

cat /path/to/saved/soylentnews/discussions/*.html | ./collate.pl > collate.txt

The other script allows predictions to be compared against actual scores:-

cat /path/to/saved/soylentnews/discussions/*.html | egrep '(comment_score_|comment_body_)' | ./estimate.pl collate.txt | sort -r -n -k 2 | more

The second script can also be used interactively. This is minor source of amusement but is generally less insightful than browsing the collated statistics:-

./estimate.pl collate.txt

If you want to be argumentative, negative or defeatist then this script will confirm that you'll be unappreciated. For example, one of our resident trolls, SaltySpice, rarely scores above 1.2. In particular, from my cache of saved discussions, "Fuck MDC" scores 0.597.

begin 644 soylentnews-score20181224.tar.gz
M'XL("`ZZ(%P"`W-O>6QE;G1N97=S+7-C;W)E,C`Q.#$R,C0N=&%R`.U76U/C
M-A3.LW^%-KC$(0FV<R$MP6EW=K?MSG1X`*:=:1P8)Q%$7<?.2DHAW0V_O4>2
MKT`&.LNTW4'?0VR?<W0NNGQ'8?$ZQ!&/\#5KL6E,<=MQOW7;[:X]C<,PX'A_
M&5:^$`Z@W^N)I]OO]HM/`;?;Z5;<3M=UX.>@`W9NO^\Z%>14_@6L&`\H0A5.
M"?NXPEN+?4S_E6+GE;UBU)Z0R%YB&J+6;X:Q@T[5KCB&78'>Q(L%O*-3L3O0
M.\;)(N`Q%5932I8<G<5@(_<*^IDP4)%I$"IS!E;6F[K84^ALCL$N8C'E9+78
M-W9`EVXV)!:!&P;%'U<$HM1",H&-5QL8QO6<A-B:X4L2X9EEAO#PCD[/WKX_
M'M;KZ).!D#F-5Q%O-`;P3BX3DUO[:-[U)Z/SX7AO:).8*5NPYOB&>])(#-@@
M'++B*#FB.E5%7\@C<3%R6M^-&]7[SG87P0=\P3%=*`?U@12#/XO!)`34^D$H
MZT?M^N?/EBG>1\X815"A]%S+Q>[XU:U][K>^5\%,&Z*D81!:4B*6X.SMNY,3
M5#6=0X0IC>DA$E&3&3A$TR"*8HYF6'@4"AEDWX^J@\0/OB'<<FZZ;I+I1I7!
M<%)JEDS]"9,SB6?K[7,#246<1"OLN=)7LCJIM+0>^_F"E-?0MV?DS^'(CWPZ
M;A2]E^=>^*BG->X&RV6XOI@&2RN7)6R62_+TG'PJ-L;&V)VM%G*D47D!8%OY
M'ZN3_@P-X#'^%V2?\G^GXU2<MN,<]#3_?SW\GTBQ,OIRTH^7.++>'S>K1\A\
M??+3KT":53B1#_:"O!%8YC4F5W/>5'S8E%Q6]]@R!-*S1\CG"8_$S9RME>TG
M:;OQU)>4*U^90GT.@""F8<PP9"<HPI1GQJNU:H,G=*J7WF>2V4H3R5H,3GL!
M<$Y*QUM*OD_NE^`RF,YA<(B1]0&O&?H&U'E-,$O9HB2+#::;0M4J<L/+UESH
M[:+QG@DNU6M:XJ90&(0HK</0R9T+U[974@]*LZUFI5GU>;7)5A/&J27&-)UF
MKZZD?\0DLFJHUE3#09C.=-JTC.?G?W4:GX-C'N/_7K>=W_\/'''_=SJNYO__
MCO]_(1,:T#7Z,:9/[`7_A/!ADZ/L=*LK_#*@P<)C<W*IR%=]W[*$)X>V?05T
M6%1Q:H]>MWX?VZ.@]=?8+H\:G0.Y@=QO^37@>V1?E0>S8C=X0'N>J6U[ZT@S
MU<E3F;89)+J+-);'G&*^HI%H&[+LC+UDV8*G/,5C9193YSQCQ1#+I/Q:(PV9
M2T%H%J093:G_0QOC48*4E_)TU+!3(*Y,ZG6*;'.OJ.S?@UI+1?+96MXQ3B[B
MTO21S(H$#.1\AX%+9%U4[ZD,!@\G*V[W#P07&Q8E&8BH:0Z*HJNE8#XO9@9?
K\$@8^4Y`]X7\B]#0T-#0T-#0T-#0T-#0T-#0T-#0T/B_X6]42<M9`"@`````
`
end

(Usual instructions for uudecode process.)

███████.com is a proper stealth mode Dot Com style start-up with more Aeron chairs than people based in Central London between the Ritz Hotel and Fortnum & Mason. We seek someone who:-

• Can write POSIX C to a subset of the MISRA standard to target GNU/Linux, BSD, MacOSX and the cheapest micro-controllers we can find; probably 16 bit AVR or possibly 32 bit Xtensa.
• Can write useful documentation in US English as plain text, HTML or OpenOffice format.
• Can occasionally invoke horrors written in PERL4 and make.
• Is willing to solder, flash and test micro-controllers.
• Alternatively, wants the thankless task of implementing a Unix GUI.

We strongly encourage part-time tele-work, part-time hours and job sharing. We encourage you to use your favorite text editor and email client. Furthermore, we:-

• Prefer HTML2 rather than HTML5.
• Prefer UDP rather than TCP.
• Prefer IPv4 rather than IPv6. (This seems to get howls of protest from people don't understand RFC1365.)
• Prefer wired rather than wireless.
• Prefer Unix rather than Windows.
• Prefer RISC rather than CISC.
• Prefer compiled programming languages.
• Prefer strong typing and static checking.
• Attempt to follow the 12 Step Joel Test without being a cargo cult.
• Are strongly skeptical about cryptographic currency and weakly supportive of block-chain.
• Are strongly pro-privacy.
• Are strongly against cloudy computing.
• Appreciate junk food including our local source of caffeinated bacon.
• Prefer jello rather than Trello. (Sorry, Joel.)
• Prefer slack rather than Slack.

Indeed, we are chilled and low drama with collective experience of South California counter-culture and hydroponics. We are very accommodating towards alternative lifestyles and disabilities. For us, wisdom is highly prized and we appreciate that wisdom is often accompanied with ailments of age. Regardless, we retain some exhuberance and immaturity. Last week, in jest, I was intentionally struck with a scrunched food wrapper. Also, a spherical cow plushie was thrown around the office. This week, it is joined by a spherical sheep, spherical ladybug and spherical tiger. We are willing to reduce such shenanigans if people require concentration. Unfortunately, this is the first of many steps going from innovation to stale, hollow corporation. However, we are doing our utmost to ensure that people don't have to tone down appearance or mannerisms for our traditional New York investors. (We're not encouraging an us-and-them mentality but you can definitely tell who has the money and who has the ideas.)

We've been given a very long leash to make the most awesome, reliable, valuable, legal product and/or service by Mar 2019 with ongoing work subject to outcome. Personally, I'd like to deliver (five sets of) micro-processor, operating system, filing system, database, streaming video desktop remoting system with HDR, 3D sound and much more besides. However, in the four months covering Christmas, it would be optimistic to deliver a draft version of one piece. Indeed, even in the long-term, we cannot be all things to all people. So, where is the best place to concentrate effort? Consider broad market trends:-

• IBM peaked in the 1960s. It is still around but it isn't feared.
• With the licensing of PCDOS in 1981, IBM ultimately aided its successor, Microsoft. Microsoft is still around but it isn't feared.
• With the distribution of Internet Explorer in 1995, Microsoft ultimately aided its successor, Google.
• Google has peaked and has difficulty obtaining and retaining staff due to ethics, censorship and brogrammer culture.

In general:-

• Trends oscillate between server and client. IBM in 1960 (centralized) to Microsoft in 1985 (decentralized) to Google in 2010 (centralized) to ??? in 2035 (decentralized). Between IBM and Microsoft, infill all of the mini-computer companies (DEC, Data General) and dedicated word processor companies (Wang, Commodore). Between Microsoft and Google, infill all of the power desktop companies (Intergraph, SGI, Sun, Dell). From Google to its logical successor, infill rich client companies (Apple, Amazon, Facebook). Indeed, Apple's peak valuation and social media valuations rely on a hybrid of local software and vast server farms. MAGAF [Microsoft, Apple, Google, Amazon and Facebook] have more than six million servers, 10 million storage units and create an electrical base load which requires 10 nuclear power stations.
• Companies at the extremes of the trend become monopolies before falling into maintenance mode. Many of the remainder chase a changing market.
• Tech monopolies create to environment for their successors.
• A successor is *almost* the opposite of the predecessor.
• Hybrid apps have reached an inevitable peak. This is likely to concede to systems with more autonomy.
• The current environment suits development of a decentralized, privacy respecting system. This is increasingly likely over the next 20 years.
• Security and privacy are awful. This is due to externalized costs and historically weak regulation. Retail banking is a relative leader in security; in part due to disproportionate impact on its own staff. However, legislation is tightening in North America and Europe; in part due to the vitriol on social media towards legislators. This regularly includes credible death threats to immediate family. In the UK, several people have been arrested or imprisoned for credible threat of rape towards politicians. Given that politicians are rarely sexually attractive, it should indicate that the problem is widespread.

Security and privacy are increasingly marketable. However, it is like King Canute attempting to fight a rising tide. It cannot be commanded or legislated. Best option is to isolate. Cannot redact data which has escaped. This is espcially true if it would be good manners.

With a rising tide of privacy infringement, the perfect product and a hypothetical contract with one bank does nothing for customers of other banks, retail breaches, medical breaches, government breaches or any other problem. There is no silver bullet which covers all cases. Regardless, anyone can be "king of their own castle" and home security is a domain where everything is within reach of a customer. If implemented correctly, the customer has complete control. Furthermore, an installation should function beyond the viability of the manufacturer. Convenience features, such as remote access, rapidly devolves into a security quagmire. Convenience features may also require more than four months of effort.

Home security dovetails with the trends of decreased driving and driving licenses, decreased drinking in bars and nightclubs, increased parcel delivery, increased food delivery, binge watching drama, increased remote study, streaming exercise classes and generally following the trends of the film: WALL-E and the short story: The Machine Stops by E.M. Forster. Overall, there is demand for a home automation system and a home security system which connects to a big screen television. The laziest method to implement such functionality is as a Kodi module. Kodi is typically deployed on a Raspberry Pi where extensive interfaces (USB, I2C, GPIO) are often unused. Obviously, this would be a lame, unreliable and insecure implementation but it would certainly be an impressive and convenient demonstration. Geeks may find Kodi's menu system to be quite tedious (and laggy). So, an expert mode is definitely required. It would be particularly useful if a scripting interface was available.

The objective is to have one system which can be used to watch films, watch television, listen to radio, listen to albums and also dip lights, adjust room temperature, brew beer, water plants, feed fish, set burgler alarm, review external security cameras and check who is at front door. Further options are possible but these compromise privacy. It is possible to forward dubious camera footage. It is also possible to integrate a house intercom, voice activated commands and video conferencing. However, interior cameras and microphones are specifically excluded because that's creepy. Even the creepy king, Mark Zuckerberg, covers his laptop camera but he'll willingly make a buck by pointing a camera at you. Our intention is to only integrate functionality which increases security without compromising privacy. How is this achieved? Everything from television to leaf node (lock, trip switch, light bulb) is fairly unconstrained. However, it would be a significant bonus if a system is:-

• Cheaper than a NanoLeaf.
• More open than Philips Hue.
• More reliable than both.

It is also a bonus if functionality overlaps with industrial, office, retail, automotive or aerospace use. For serious use, a Raspberry Pi would be replaced with a 1U rack server with ECC RAM. For very serious use, servers could be clustered with fail-over. Leafs should be as economical and as useful as possible. A friend who runs a virtual reality start-up gave a demostration of an Xtensa micro-controller with USB, GPIO, wifi and 256KB flash serial ROM. When flashed with a Lua interpreter it is possible to run scripts which are stored on a local filing system. Virtual serial over USB also allows access to a Lua REPL. From this, it is possible to join a WPA2 network, run a DHCP client and run an HTTP server. Indeed, without stopping the HTTP server, it is possible to modify the URL-space so that arbitrary paths may be handled by arbitrary functions. A fall-through case serves static files from the local filing system. My friend estimated that the minimum configuration used less than 64KB ROM.

What can be achieved in a smaller space? Chess computers have been implemented with 128 bytes RAM and a popular chess program for the Commodore VIC20 was supplied as a 2KB ROM. Graphic competitions have a 4KB category with impressive entries. (Although, Subdream's 64KB Raum Zeit octree renderer and Porter Robinson & Madeon's animated music video for Shelter remain favorites.) A smaller system is not an academic problem. A smaller system uses less energy. A smaller system also creates savings which ripple through manufacture, wholesale, retail and integration. Reducing cost of a micro-controller by 5¢ may reduce retail price by more than US$1. Alternatively, savings can be allocated to improved cipher suites. For many devices, almost any cipher would be an improvement. I hope that it is possible to include a CLI, graph library and SMART style EDI interface in a micro-controller with 4KB RAM and 16KB ROM. I also hope to implement a network switch with 2KB RAM. Even if these estimates are repeatedly raised, the result remains competitive in a market with dual-core light bulbs, quad-core watches and where 1GB RAM is regarded as embedded.

Our preferences for hiring are as follows:-

• UID1111 who is bold enough to show work, introspective and, despite protest to the contrary, cares about detail, such as string buffers.
• UID2828 who is working on the closely related problem of vehicle networking.
• UID5227, Linux device driver contributor. (I noted the quality of this work at the time.)
• Anonymous Coward who shared a fast and concisely documented matrix library.

We are heavily constrained by time. We would otherwise like to exhaust these options before seeking people more widely.

I'd like to finish with a message to people with similar sentiments to SoylentNews' Not So Anonymous Coward, SaltySpice. (When did we start naming our trolls?) We strongly agree that there are too many oxygen thieves in software development and corporate administration. If they could FOAD, we'd make more progress. In particular, computer security is a sticking plaster for developers who don't care. From our unused collation of soggy.jobs, the most generous lower bound for fake or speculative job adverts is 5%. We would be wholy unsurprised if the majority of job adverts are fake. Particular ire goes to Infosys and IBM subsidary, Aspera, for listing fake jobs. Paraphrasing from email:-

Me: Are you *sure* you're rejecting UDP, file transfer, POSIX C programmer with professional experience of your CLI and GUI?
Michelle Munson, co-founder of Aspera: Yes.

That's like the alleged musicians, Duran Duran, who came second in a lookalike competition or the guy in Charlie Brooker's ScreenWipe USA who failed a screen test to portray himself in a fictional version of his own life. However, when unemployed people are expected to seek work but employers are not obliged to hire, don't be surprised when the result is nepotism, time-wasting and abusive practices. Also don't be surprised when the unviable dregs are advertised for an extended period. My personal favorite was a job advert for a tri-lingual, Adobe, Java, Cisco expert which paid San Francisco minimum wage. This idiocy is made significantly worse by "signposting" "services" which provide another level of indirection where none is required.

It is damning that the majority of programming jobs are obtained via family, friends and interview prowess. Ability to do the job is very secondary. Caring about the project is also secondary. I would be inclined to hire SaltySpice in preference to MDC but, c'mon, meet me half way. Give me *anything* which puts you ahead of my least competent, least employable ex-colleague.

From the Wikipedia article about spherical cows:-

Milk production at a dairy farm was low, so the farmer wrote to the local university, asking for help from academia. A multi-disciplinary team of professors was assembled, headed by a theoretical physicist, and two weeks of intensive on-site investigation took place. The scholars then returned to the university, notebooks crammed with data, where the task of writing the report was left to the team leader. Shortly thereafter the physicist returned to the farm, saying to the farmer, "I have the solution, but it works only in the case of spherical cows in a vacuum".

Since then, a spherical cow breeding program has been successful.

Spherical cows are often the stereo-typical black and white Holstein Friesian cow pattern but may also be the brown and white Jersey cow pattern or numerous other patterns including plain white, brown or black. Unfortunately, spherical cows in a single color are often mistaken for mutant hamsters and shot on sight. This contributes the Holstein Friesian pattern being predominant.

Although genetic throwbacks may exhibit hooves, ears, a tail or an undesirable ellipsoid profile, a pure-breed cow is completely spherical. They are often grown in proximity to cuboid watermelons; an innovation which makes more efficient use of space within a domestic refrigerator. A side effect of the spherical cow breeding program is that spherical cow milk has an unusually high surface tension. The milk often sits as droplets rather than a homogeneous liquid. This may be corrected with a very small quantity of surfactant. However, this additive often falls afoul of legislation regarding dairy products and therefore spherical cow milk may be restricted to illicit channels; similar to raw milk.

A perfectly spherical cow has no (identifiable) legs. Indeed, a perfectly spherical cow in an idealized meadow has no cross sectional area with the ground. Therefore, spherical cows are frictionless with the ground. In a valley, a spherical cow placed near the top will roll down the valley and up the other side. If the valley is in a vacuum then this process may continue indefinitely. If the valley has idealized wind resistance, spherical cows will typically be found in a local minima. Furthermore, a herd will be arranged in a manner approximating sphere packing. If idealized air is still, a spherical cow placed near the top of a valley or hill may be found in one of the surrounding dips. However, if wind is strong then an entire herd of spherical cows may be stuck against a fence. It is strongly recommended that any fence exceeds the radius of the largest spherical cow and can withstand the force of an entire herd during strong wind. Despite being domesticated, it is completely unknown how spherical cows eat or breed. The leading theory is that spherical cows are trans-dimensional beings but agricultural researchers are unsure how to test this hypothesis.

The easiest method to determine if a spherical cow is pregnant is via the increased radius of the cow. However, it is incredibly dangerous to be around a pregnant spherical cow. When a spherical calf is born, it may shoot out like a ping-pong ball. It may travel an incredible distance across countryside; 200m has been recorded. For this reason, spherical cow birthing sheds are strongly re-inforced and the interior is covered with damping material to reduce the total number of ricochets. A newborn spherical calf is approximately 1/2 of the radius (1/8 volume) of an adult. Spherical cows are most likely to be born during a full moon. This may be the basis of an unexplored joke.

A spherical calf grows to full radius over 18 months or so and it sufficiently mature to breed in its second year. In idealized conditions, breeding pairs of spherical cows may increase in a manner which strictly follows a Fibonacci sequence.

Spherical cow leather was the preferred material for equipment used in many types of sport. Most typically, it was used for sports balls due to its equal weight distribution and equal curvature. It was also the preferred material for chess-boxing gloves. Nowadays, sports are more likely to use synthetic materials; especially in upper leagues. Foreseeably, this has contributed to a decline in spherical cow herds.

Spherical cow steak is of unusually high quality. Spherical cows may have recessed legs and therefore edible parts of a spherical cow consist almost entirely of rib, back and belly. Approximately πr³/3 of any given cow is rib meat. However, to properly sear a spherical cow rib eye steak, a particularly large circular searing pan is required. Some advocates of paleolithic diets are particularly keen to promote the benefits of spherical cow steak. One of the disadvantages of intensive farming is that beef, pork, chicken, turkey and other meat is almost exclusively single sex. This may have long-term consequences for all humans. To avoid adverse effects of xeno-steroid hormones, it may be beneficial to eat historical proportions of male and female beef. Given the foreseeable difficulty of identifying male and female spherical cows, males are often allowed to grow to adulthood. However, the increased cost associated with spherical cow farming may have fringe health benefits.

Attempts have been made to cross-breed spherical cows and flying pigs. So far, this has been unsucessful. However, this may change as genetic engineering advances. Nevertheless, it has re-ignited a debate over collective nouns. It is generally accepted that the collective noun for pigs when flying is a flock. However, accepted use beyond this case is varied. In the case of flying cows, moderates suggest that the portmanteau, flerd, be used in all cases. Unfortunately, this reasonable proposal has been met with almost universal disdain from farmers. While diary farmers look forward to new varieties of stationery, dairy farmers are concerned that any re-classification of farming may adversely affect economic subsidies for agriculture. This may further contribute to the decline of spherical cow breeding.

It is possible to make a dozen spherical cow plushies for £20 (US$30). This is a popular hobby project because they are relatively easy to make. Only cuboid plushies are easier to make but both require relatively large amounts stuffing. Spherical cow plushies require two self-similar pieces of synthetic fur material. Ideally, the synthetic fur material will have a Jersey cow print. (Among professionals, Jersey print material is known as plane cow plush.)

The two pieces fit together in the manner of a tennis ball or hacky sack. In the trivial case, this requires an ellipsoid template consisting of a semi-circle joined to a rectangle and another semi-circle. For each piece of fur, ignoring allowance for sewing seams, the straight and curved sections should have the same length. The straight section of one piece should fit with the curve of the other piece. This sets the constraint for the relative lengths. For a semi-circle with radius r, the rectangular section should have width 2r and length πr. When a piece of material is curved to fit the other, it would be reasonably assumed that πr is the curved distance between polar opposite focus points of a curved ellipsoid. However, this is not the case.

Making the reasonable assumption that the seam does not stretch but that synthetic fur may stretch to become approximately spherical, it is neccesary to include the (appropriately named) Skinner's constant or other Flannagan Finagling Factor. The four inflection points (where the seam changes direction) are equally spaced around a great circle. Relative to the radius of the ends of an ellipsoid, the inflection points are sqrt(2)r from the center of the sphere. Therefore, for a tennis ball, spherical plushie or similar with a required radius r, all the previously stated measurements for ellipsoids should be scaled down by a factor of sqrt(2) (approximately 1.41). Common worked examples follow.

When making cutting templates from paper, length should be 1+π/2 of width - prior to ends being rounded. For an A4 sheet of paper (297mm×210mm), cut to 297mm×116mm before rounding ends. For a US Letter sheet of paper (11 inch × 8 inch), cut to 11 inch × 4.3 inch before rounding ends. This is suitable to make a plushie with 16cm (6 inch) diameter. Template and/or material should be cut with allowance for seams. Professionals typically include a 6mm (1/4 inch) seam in the template. A wider seam (included or excluded in the template) may be desirable due to inexperience and/or to increase sturdiness.

It may be useful to tack a few stitches where the straight/curved point of one piece meets the curved/straight point of the other piece. It may also be useful intermediate points. Each tack works as a rip-stop. It also reduces mismatch of material when sewing along seams. Optional adornments, such as legs and tail, can be sewn separately and then stitched into the seam. The tennis ball seam provides suitable placement for four legs, two ears and a nose. Miniture cow bells can be purchased in bulk from morris dancing suppliers.

The cheapest stuffing by volume is synthetic car washing sponge. However, the result may feel lumpy and distinctly not spherical. One sponge may be cut in half and each half may be trimmed into a dome shape. Another sponge may be cut to approximate a disc. The three layers (dome, disc, dome) may be squeezed through the hole of the synthetic cow hide before the final section of the seam is sewn closed. While attempting to close the seam in the least obvious manner, increase the difficulty of the task by simultaneously considering the hairy ball problem.

Worryingly, the result meets European safety standards for flammable material and also toys with small parts. However, it does not meet the widely flouted labelling regulations. Without exception, all waste material (fur, foam, cotton) may be stuffed into a spherical calf plushie. However, this does not meet safety standards and should not be given to the type of child inclined to stuff things up its nose; nominally a child less than three years old.

Synthetic fur may be occasionally brushed with a wig brush. Stains can be removed by soaking the surface material only. Deep ingress of water is likely to encourage mold. Synthetic fur is likely to curl or melt in a clothes dryer or under an electric hair dryer. Do not spin wash or tumble dry because it makes plushies very dizzy.

This is written primarily in response to JID3525 by UID2828 but is of general interest:-

People are quite interested in your work with Arduino systems. Whatever you've achieved, people will pick fault. However, I understand that you are primarily interested in automotive applications. You also want something which is more open, accessible and educational than the predominant CANBus systems.

CANBus DRM

People may wonder why a serial protocol is neccesary in a vehicle when legacy designs had little more than a spark coil. Engine emission limits almost force a central computer. From there, security and convenience features make it worthwhile to spur numerous micro-controllers. A modern car may have more than 100 micro-controllers. The alternative is more than 3km of wires between wiring looms and an infeasible array of fuses and relays. At some point, it is easier to implement everything with software and a network topology. From Robert X. Cringely on Mon 5 Jun 2017:-

I was shocked five years ago, for example, when my friend Shoichiro Irimajiri told me that automobile wiring harnesses had reached the point where they cost as much to build as the engine and transmission for the same car. Irimajiri-san, who built Honda's first U.S. assembly plant in Marysville, Ohio, was then a board member of Delphi, the world's largest maker of car parts, and certainly knew what he was talking about. The future trend, he explained, had to be for wiring harnesses to become cheaper by turning into intelligent networks with single wire pairs replacing dozens of wires with hundreds of connections.

However, manufacturers have used this transition to enact substantial lock-in and a large amount of unproductive busiwork which is largely hidden from customers but extracted via increased repair costs. One of my ex-housemates was a car mechanic and from this I learned that car repair has become as tedious as Microsoft Windows licencing. Specifically, the Windows® Genuine Advantage*†☠ requires re-activation of a licence key after changing components such as processor, harddisk or motherboard. Most people out-source car repair and so they are generally unaware that the same DRM tedium is required after replacing an electric mirror. All such components are connected to a common one-wire serial bus and almost every car from almost every manufacturer uses the same bus network protocol. The specification for the bus is open but the address ranges, authentication and data are not consistent between manufacturers. If a person wonders why car repairs are so expensive, it is because an independent mechanic has to:-

1. Diagnose fault.
2. Obtain DRM component from authorized supplier.
3. Fit component.
4. Connect authorized diagnostic computer. (Purchased and maintained at own expense.)
5. Obtain first part of authentication handshake.
6. Telephone premium rate support line and get put "on hold" for 30 minutes.
7. Complete cryptographic authentication.
8. Discard old, unservicable component.
9. It may be possible for an erroneously substituted component to be installed in another vehicle. The manufacturer may set arbitrary limits on this process. The manufacturer may change the limits unilaterally. This includes revoking all support to a mechanic, vehicle or model.

Do you own a modern car or is it a smartphone on four wheels? Even niche luxury cars use CANBus with unwanted DRM. I investigated luxury car manufacturers and found that it is mostly a business of out-sourcing. This includes body panels, dials, engines, gearboxes, brakes and car hi-fi in addition to more mundane components, such as screws, switches and tires. (The history of CAD/CAM software suitable for curved body panels is tied to Prime Mainframes and, via Tandem Computers, back to our interests in reliable and documented serial protocols.)

Competitor Analysis 1: Vehicle Manufacture Assembly

McLaren specifies its own engines (but certainly doesn't test them where it assembles cars). The remainder send out CAD files or purchase from a catalog. Larger manufacturers have their own tweaking divisions so they can upsell their own products. The most ridiculous example is the heavily marketed Audi R8. Each edition is only available in one color. The trashy flecked orange convertible model was product placement during the test flight in the first Iron Man film where it is priced at US$135,000. However, it has a ZF transmission, tweaked with VolksWagen mechatronics before being assembled in a (hopefully hardened) Audi chassis. The transmission alone visits three or more sites during manufacture. This is an example of supply chains becoming increasingly global. Another example is Cherry Keyboards which was founded in the US before re-locating to Germany. It produced many switch and sensor products for automotive use before these products were amalgamated into ZF.

One safety feature which was initially available on an Audi R8 (and now more widely available) is headlights which selectively ignore rain and snow by using infrared sensors and a matrix of white LEDs. This system functions at 70MPH. I explained this to a person at my local makerspace and was told that it was technically infeasible. I replied that much of the required functionality has existed in optical mice for decades. It may have to work at a higher speed, with more elements and some calibration between sensors and lights but it is certainly feasible and commercially available.

I was less impressed with Aston Martin (which uses odd numbers for car models to avoid confusion with the number of cylinders in an engine). This was before the grifter banksters tried floating a company which has been bankrupt on seven occasions. Ford has owned various brands including Volvo, Jaguar, Aston Martin and Shelby. Few question Ford's apparent absence from self-driving cars but several advances have been made with Volvo cars and trucks. The most successful Aston Martin product, the DB9, is a salvaged failure and famous example of product placement fakery. The DB9 is also the least advanced product when compared to contemporaries. When Ford owned Jaguar and Aston Martin, a Jaguar design exceeded component cost and was transferred to Aston Martin. That's why a DB9 has a Ford chassis. It is also why the DB9 was manufactured assembled in a separate factory. However, the "DB9" which appears in the James Bond film: Casino Royale is a mock made from a DB7. Technicially, it is the best DB9 ever made - especially after mock DB9 exteriors were rolled for a stunt. The special Chinese edition Virage 88 doesn't come close.

Barriers To Entry

There's an old joke aout a man browsing in a Rolls Royce dealership. He asks the sales respresentative why the cars cost so much. The answer is "Try making one for less, sir." However, that isn't true any more. A back-of-an-envelope calculation shows that the Bill Of Materials for a car with the performance of an R8 and the strength of a DB9 is less than the retail cost of either - even for a single unit.

There are a few limitations for the aspiring sports-car manufacturer. For example, European safety regulations require approximately 30 vehicles to be trashed (at manufacturer expense) in front impact, side impact and similar tests. A similar number may be required for other jurisdictions. However, in the UK, regulations only apply to fully assembled vehicles. The UK has a long history of kit cars. The most famous is the Caterham Seven which was featured in the title sequence of the television series: The Prisoner. ("I am a name, not a number!" and then drives car model seven. Yeah, you're a unique snowflake - just like everyone else.)

In the UK, if a customer makes a token effort with a wrench and assumes responsibility for final assembly, a vehicle can be certified as street legal on a individual basis. I've seen some dangerous stuff certified as street legal, including a recumbent motor-trike with exposed continuous variable drive. It is also possible to pack approximately twice as many vehicles into a shipping container if they are sold in pieces.

I am concerned that the disparity between material cost and sale price occurs prior to the inflection point from hydrocarbons to electric. This could be like the situation where micro-controllers become powerful enough to drive VGA monitors at the point where VGA gets deprecated and replaced with something less accessible, less secure and more centralized. Or where analog headphone jacks and US$1 amplifiers get deprecated in favor of less accessible, less secure and more centralized digital protocols. Now that the little guy can make a better sports-car, the rules may change. Unfortunately, the result is likely to be garbage.

An electric train is sensible because the energy is supplied externally and therefore it doesn't carry its fuel. However, an electric car - even with the energy density and fueling/charging time of hydrocarbons - competes at a 2:1 disadvantage because an electric car carries the deadweight of a flat battery. A diesel (turbo) electric car is worthwhile. Specifically, an engine block connected to an alternator and an electric motor on each wheel with regenerative braking. This would be a hydrid with no gearbox or differential. Electrical losses are comparable to mechanical losses but it swaps precision machining for precision etching. It also provides the optimal acceleration of an electric car and the urban efficiency of a hybrid.

The next limitation is the industry specific CAD/CAM licencing (and formats) for body panels and interior. This can be skipped if a chassis is retro-fitted or a chassis has a strikingly angular appearance, like an F-111 Aardvark or the USS Zumwalt. I appreciate this loses mass market appeal but I merely suggest what can be trimmed from low volume production.

Competitor Analysis 2: Serial Protocols

The remaining limitation is the serial protocol which is used ubiquitously by car manufacturers. I don't want to diverge from CANBus to be a contrarian. I only want to diverge to reduce cost and risk. I want user-servicable parts - especially in a tight situation. I believe this is your primary use-case. The ability to tweak and customize is a welcome bonus but the penalty is that it is time-consuming, like every other open source project.

What is the point of maintaining wire-level compatibility with CANBus when manufacturers don't use consistent address ranges or data formats and lock-out third-party spares via DRM? The most standardized part of CANBus is agricultural accessories. However, in this case, ass-hattery is moved elsewhere. Most notoriously, John Deere refused to honor the re-sale of a firmware licence. Competitors have followed this poor practice. This greatly affects the capital value of a tractor and routinely requires tractor mechanics to break the law.

Are there any other serial protocols which are suitable for automotive use? Commercial aircraft use ARINC429 and larger ones use AFDX which is effectively ARINC77 tunnelled over UDP over dual runs of 100Mb/s twisted pair Ethernet. Many military aircraft use MIL-STD-1553 - except the F-35 Clusterfuck which uses FireWire. (I wish that I was joking.) I2C was originally developed by Philips to reduce wiring in televisions. This was eminently sensible when Philips made the custom chips and the televisions. Nowadays, robots use I2C for servo control and ultrasonic distance sensing and it is fairly apparent that each node is an independent micro-controller. Beyond aircraft and robots, there is RC5 for infra-red remote controls (possibly via an RC5/I2C bridge), MIDI for music sequencing, DMX for stage lighting, DALI for ambient lighting, X10 for home automation, the defunct iLink for hydroponics and several mutually incompatible protocols for Internet connected devices.

I don't suggest devising one universal serial protocol because that inevitably creates another standard while doing nothing to handle legacy cases. Regardless, many of these control protocols have very small address-spaces; typically 8 bit or less excluding reserved addresses. Even IPv6 over IEEE802.15.4 with stateful header compression and self-selection of addresses effectively has an 8 bit address-space rather than the expected 128 bit address-space. The most inane example of the art is HDMI's slow bus with a 4 bit address-space. Of course, this means that a television which implements I2C and HDMI may have a protocol with 7 bit address-space inside the device and a completely incompatible protocol with 4 bit address-space between devices. (Bring back SCART. All is forgiven.)

It is hoped that a vehicle has less than 127 micro-controllers. However, a small address-space hinders independent innovation and hinders auto-detection of optional peripheral components. A Hamming distance greater than one also increases reliability.

General Purpose I/O

A general purpose I/O expander system is of general interest because they are often implemented poorly. Multiple incompatible systems exist for Raspberry Pi systems. For some domains, this would be acceptable if basic database theory was applied to process control. (Error detection, read locks escalating to write locks, atomic commits.) More often, it is written for children or halfwits. And it is an insult to children. The typical scenerio is something akin to "Hey! I've got two Raspberry Pis connected via Ethernet and I'm running out of I/O pins. *Surely* I can get a program on one Raspberry Pi to flip pins on another?" And this is how we get pigpio and the numerous incompatible variants. Please don't use them; especially on hardware lacking parity RAM.

I first encountered this scenario when working with PIC micro-controllers. I found that PICs with more I/O were disproportionately more expensive because they came with additional RAM, ROM and interfaces which were not required. My idea, which was not unique, was to have an open or closed loop where all-but-one of the chips had identical firmware. This leads to the possibility that additional I/O pins don't have to be in the same place. For example, each node may control a cluster of car lights. The next consideration was addressing.

I didn't want I/O expanders with 16 I/O pins and then lose 8 pins to a hard-coded address. (Nor some variation thereof.) Nor did I want to maintain consistent addressing. So, I devised a protocol of three byte triples of the form <device>, <command>, <data>. In all cases, <data> is provided even if it is a dummy value. This allows upward compatibility and provides a placeholder within a closed loop serial stream for read operations. The part that I thought was particularly cunning was that the device number would be decremented by each I/O expander. It was hoped that every device could test against zero. This would allow the I/O expander at the head of the chain to be device zero. All subsequent devices would be numbered contiguously in ascending order and no I/O pins would be used for device addressing. Unfortunately, this scheme is very susceptible to bit error - before or after a triple is processed by the intended device. There is also the very fundamental problem that each byte of a triple may be mis-interpreted. For example, a command may be interpreted as a device number.

After many iterations of eliminating obvious flaws, I settled on a protocol which can be concisely be described as ATM Lite with a 28 bit address-space and 24 byte fixed-length cell bit-stuffed into a 256 bit frame. Somewhere along the way, synchronous communication became asynchronous. Actually, a dream caught that design error. In the dream, I was at my local makerspace and demonstrating a four node configuration but it didn't work. I woke and thought, "Phew! It was only a dream!" I then had the worrying realisation that the test case was depicted accurately and that designs over the previous two weeks didn't work.

My primary use-case is hydroponics and home automation - where everything is grounded. However, my choice of bit-stuffing would look hopelessly naïve in an ungrounded automotive or aerospace environment. A quick fix would be Manchester encoding where every bit is echoed with the reverse polarity. However, that halfs the bit-rate. In typical cases, such as bit-stuffing with an even Hamming distance, it is possible (and often desirable) to have an equal number of zeroes and ones to represent each valid symbol. This maximizes current balance and discourages accumulation of static. For my purposes, unbalanced bit-stuffing is sufficient. Indeed, it is desirable for low-speed software implementation. This would be re-considered if an ungrounded and/or high-speed, hardware implementation was ever required. Similar choices define the different generations of USB, although neither USB 1.x, 2.x nor 3.x are current balanced within one wire.

It is reasonable to assume that your protocol is somewhere between a toy implementation and a carrier-grade communication link. I hope that you have considered error detection in some form even if you have not investigated the art thoroughly. Indeed, I strongly recommend against a thorough investigation because you'll rapidly encounter an exotic zoo of unbalanced LDPCs where you can get too smart and zap your electronics. Despite this, minor tweaks to your protocol may disproportionately improve resilience. However, I would be the first to concede that there are diminishing returns to this process.

Circuit Design

I presume that you are a hardware engineer and that your designs are distinctly above average. For automotive applications, people wrongly assume that 12V is the maximum Voltage. However, 14.4V is typical for battery charging and 60V transients are typical despite surge suppression. Semiconductors, typically rated at 30V, are rapidly destroyed in such an environment. Irrespective of firmware quality, your designs would have increasing utility with increasing resilience to automotive electricity, US mains electricity, European main electricity and three phase electricity.

Your circuit board designs would also gain utility by providing mechanical compatibility with the fixing holes of the most common credit card computers. I assume that your design is considerably more compact but the (optional) additional area could be used for MOSFET heatsinks. In this scenario, a compact (automotive) version would be sufficient to switch headlights whereas a credit card (home automation) version would be sufficient to switch a kettle. Indeed, it may be trivial to design boards so that they can be crudely cut to the required size after manufacture. This would be particularly true if you provided, for example, eight MOSFETs and only six were required.

Successful Propagation

Your motive is neither fame nor fortune. You merely want to see your work being used rather than ignored. In particular, you are considering open source partly to obtain the most reach within formal education environments. I was inclined to agree with this approach until I suggested similar to a friend. My friend is extremely hostile to dumping a life's work on the Internet and finding that it is completely ignored. That would be particularly dispiriting. Instead, I strongly suggest charging about 10 times the component cost. This is not so that you can become an evil capitalist, spend the money on porn, hookers and blow or circuitously defend against litigious parties because you are worth suing. It is to cover unforeseen setbacks and spend the remainder on promotion to further your ideas. From the (cleverly named but concise) Lazy Man's Guide To Riches, retail price should be a minimum of three times the cost price (including materials, assembly, instructions, packaging, delivery and support) with the intention that more than half of the turnover is spent pursuing further sales. I understand that it appears pointlessly circular to take money to spend money to take money - while skimming whatever you can to cover equipment failure, manufacturing defects and calamity. However, if you do not engage in this practice then you exclude a very obvious method to leave a legacy.

I discourage the fixation on youth education. Without effacy, we attempt to stuff maximum knowledge while also expecting social success, civic responsibility, health and athletic prowess and many other goals in preparation for a career, financial responsibility and family. The pressure leads many to give up and some are driven to suicide. Although we should strongly encourage numeracy and rationality, adding proficiency electronics and micro-controllers is detrimental, especially for a kid who doesn't have an aptitude for STEM. Even in the case where the computer game, Minecraft, became a school activity, use of Minecraft for personal projects rapidly became uncool. Do you want your field to be treated similarly?

Neither do I suggest that you target efforts towards military veterans. For many people, joining the military is the best economic option even if aptitude lies elsewhere. However, it rewards failure to withold a STEM education until someone becomes a violent aggressor. Nor are they in the best state to learn after injuries; visible or hidden.

If you must target efforts, concentrate on people who are in mid career; who are stable, competent, unhurried - and heading towards divorce and homeless alcoholism when their job is suddenly offshored or automated. If you charge accordingly, you could personally extend credit to people in decline. In aggregate, you would cover the losses.

Suggested Strategy

Hardware and firmware may have independent merit. If nothing advances the art, overt secrecy may not be an advantage. However, if you strictly retain copyright, you have the descretion to gift licences to worthy causes. You can also avoid wasting resources by ignoring unofficial clones. It may be an advantage for clones to expand the market but it becomes problematic if they leave you with liability, take rewards and leave you in poverty. In general, if someone wants to do the money-getting while spreading your work then it gives you more time to use your talent. However, you are far less likely to spawn imitators if they cannot stake a career and mortgage on it. That definitely won't happen if you act like a hobbiest.

Anyone wishing to make DRM-free micro-controllers for automative applications may use any of this text with or without attribution.

I sometimes write small programs over Christmas. I published a rotating torus and a fractal animation. A previous effort was a sudoku puzzle solver. This type of puzzle involves completing a 9×9 numerical grid (or a 16×16 hexadecimal grid for the particularly ambitious) while meeting the constraints that a digit may not be repeated in a row, column or box. While some people find this type of puzzle to be fun, I find it rather tedious and contrived. I also like to solve the general case. People have used many techniques to solve sudoku puzzles and perhaps the most ambitious was a spread-sheet with multiple layers and a very large number of conditional operators. (This is more akin to a Kohonen network or a deeper neural network.)

My technique is rather blunt: 81 levels of recursion. Superficially, this would appear to require an infeasible maximum of 9^81 attempts. However, the constraints of the puzzle bound this to something more akin to 9×8×7×6×5×4×3×2×1 attempts and it often takes far less attempts. Indeed, solving an empty grid requires relatively little computation.

Input (via stdin) is nine lines of text with any non-numerical string as a separator between digits. Use digit zero for unknown inputs. Output (via stdout) is nine lines of text followed by a count of the recursion calls:-

begin 644 sudoku-solver.pl.gz
M'XL(`/4*0EL"`YV3SV^;,!3'[_XKW(:F$+<!$DT-(IXF=9===DFE'9(<\L-I
MK+F`;-,13?SO>S8.(5%6:;L`[_F]S_=];=.["4LEPS7/PH))@1]_(-3#LW*;
M_RSQ+!?O3$+L/P>C*!X_3O#+GN'G/%.YU+Q\&Z(>K)JE>#3ZA)5>2=TD)M%3
M%&.U>F<([7+I>X)&J2>FDQB>A"8!_HTP]C1[*^AT]O+UV_?/*22^")XQJ@K!
MM1_.%]E"+C1^6)+PP98&IL;B*H.KI@D\"&E@@'N5?#L'OE<MJ6=0<_@R/36J
M$5+E&F^Y*FRUI1P,Y6"'.K1#?:2`<2%YIO$]OA\ZM0-Q&D;E5'&[R&[/E"7;
ME%(Q"Q+Y9B5\;\L*O0^HVO.=3D]I$70"F?_JAIO\;'6=5S:$?)EI0LPWWSDR
MI9/X.'C?&/>#M#MATX0!)-30C8LQJ[AN!F]0C4L+7-[0J`4Z/TZ+Q)9=8R:<
M1Q@*9J>@Y$K"Q,D;$[3)W24N!4::4N@)Q\%@]$1L"*4F'*<M<4!=C[M6YD)-
M(0?'WCDF;R=6K[#FPH].U-H\^@0!<Y[&:+]_D:1PIX)NWU$F2MM4?0T*)J!_
MD%Q@V_1_@V'7"/BZ&P?-;E5FKRYE_E+T+Z+'M[D/MH;&W<ZS.P+8$^#Z+;G6
7U:K6[:]D?YV6$$$K0G\`U/.'`+`$````
`
end

(Usual instructions for uudecode process.)

I may finish writing a web interface for this script so that I can spoil more enjoyment save more tedium among non-technical people. I planned to implement a stateful script which uses an HTML form. However, if anyone wants to extract digits from an image, that would be greatly appreciated. This would allow a sudoku puzzle to be solved by selecting an app on a smartphone and pointing the smartphone's camera at the puzzle. Or uploading an image via an HTML form. I like to solve the general case.

A friend suggested a solitaire card game solver. I give this serious consideration.

Since playing with Knoppix on a friends thinkpad circa 2001 while he was on break from uni, ive been a Debian user ever since. That extended to Ubuntu back during 6.06 LTS days and then on to Linux Mint and Cinnamon after hating unity/gnome 3. But I've been craving something more lightweight and systemd free but had yet to find an interesting enough alternative until reading this reply (Thanks urza9814 and Azuma Hazuki!).

I really liked what I saw and it felt like a cross between a modern Linux distro and light weight OpenBSD. BSD licenses for their from-scratch package manager; I don't dislike the GPL but BSD is much more permissive and allows giving back to the BSD folks. Then we have the use of runit for init which uses the very slick idea of a managed process tree (very Unix). Plus the distro is rolling which is something I've always been interested in for desktop use. LibreSSL is a big plus along with a myriad of ports to other arch's like raspberry pi and other arm boards (like open and net bsd).

I first downloaded the musl libc version to see how much it could support and surprisingly the musl libc version is plenty stable and usable. Though, I wanted to run a serious hardware box to test it out and see how it performs, and more importantly, how simple it is to install and manage. So I opted for the standard glibc version. Runs like a champ. The xbps package manager is easy to use and a few simple commands keep the system up to date and install packages and resolves dependencies just like apt or rpm. The install process is also very simple as you boot into a live desktop, open a terminal and run the installer as root using sudo. After a very quick configuration the installer does its thing and you're ready to reboot. So far I have my plan 9 tools setup such as drawterm and plan9port. Getting a usable system up and running is honestly pretty damn easy. One word that sums it up: refreshing.

My next steps are some sort of mdadm torture test to see how stable it runs between reboots. Though I'm not so sure if it's systemd/kernel/hardware related just yet. If its crap software, then my workstation/big file server gets voided. I'm also interested in switching my laptop as well.

== All aboard the 9grid Part 1.1 ==
So I've been eager to write a second chapter to my plan 9 experience and so far progress has been slow and steady. I have a much better grasp of the concept of name spaces and how the OS works but do not yet feel qualified to write a guide. I have been tinkering with my setup, a Celeron J1900 board with a 256GB SSD running the most recent version of 9ants configured as a CPU server. The box sits next to my router plugged into the network and happily hums along at about 11 watts which costs me about a buck sixty here in NYC to run 24/7. I don't allow direct access instead relying on an ssh tunnel via my ageing Debian box which is an old wyse terminal that sips power at 8W (void conversion in the future). I can pop onto my CPU server from work using cygwin on windows 10 and drawterm built under cygwin.

The feel of plan 9 is quite interesting. The GUI feels obtuse at first but after getting to know how to mouse chord and use the terminal, it becomes a pleasure. The whole idea behind the lack of a command history is because the text buffer of the window is your history and is fully editable. if you see a command example in a man page you can edit it on screen, highlight it, and "send" it which runs that command.

The plumber is a message server that receives plumb requests which are text strings that the plumber parses and matches against a set of rules in your plumber config. The rule then performs the appropriate action such as running the associated command. This is similar to the context menus in the MS Windows right click menu. Instead of having open/openwith or some menu modifier buried deep within the registry, you just have a message decoding server. If it's a url, open the web browser with that url. If it's a png open page to display it. If it's a text file which includes scripts and source files then open them in acme. etc. Very neat little service that gives you a lot of interesting functionality.

The Acme Text editor is quite interesting and quickly showing it's powerful ease of use. Open a session and you have a listing of your home directory. right click a file to open it in a new window. if the file is an image, media, or whatever, then the plumber will forward the request to page which will open the appropriate viewer. Right clicking a directory opens that directory in a new window. want to create a directory? type the shell command mkdir newdir, highlight it, and middle click it to run that command. Then click get to refresh the directory listing and you will see newdir. Acme use example:
Create a new script file in current directory window:

touch newfile, highlight, middle click.
middle click get to relist directory contents and observe newfile.
chmod +x newfile, highlight, middle click.
right click newfile to open edit buffer in new window.
add some stuff to file, e.g. #!/bin/rc \ echo "Hello world!", click put (save)
in directory window, middle click newfile
new windows opens with script output we just executed: Hello world!

All of the above commands and editing is done within acme windows. The idea is the editor works with your tools, not against them. The only issue I have had is input events are skipped causing infinite loops. This happened when using fgets from stdio and not native plan 9 input using the draw libraries.

I have to organize my notes and writings pertaining to my tinkering with plan 9. The architecture is quite interesting and simple overall. I plan on writing a bit of a basic into of the OS internals such as how the kernel works, booting, networking, graphics and more. But I am still a ways off but having a ton of fun learning by working in the OS. I have been fooling around with this guide and building the examples in plan 9 using acme and as much native plan 9 libraries as possible. Fun learning experience.

If there was any doubt about processing HTML (from stdin to stdout) into rainbow text, an example implementation is provided. It works best on long paragraphs of text and many find the result too irritating to read:-

begin 644 rainbow-text.pl.gz
M'XL(`$P&0EL"`X5376O;,!1]]Z]0$C>58SN2G&84;'F%P-C#^K(%-HA=<!)G
M\>9:0;)IH4E_^R3Y*QD9PQ:Z.N?HWJ.OT0!5@J-U5J!#RG/@?C>,$?B\?/P"
MOB99L68O8)F^EA*#"\O#9.[>@^4^!0M6",;+K'J>&B/)2NH>WWD?0)G\3@NP
MX^P9+!Z_&<8#K]-@"CU\Y[0-RT_VEM\)"(78(=C3-)D31_[XC/<47W^*K2<;
MHEJ#1@#>#`!RMDER:&:%1<4^VY5^C[&JM/KA@SBDF_/Q3YYL]=C<L)P*B24<
M=NXMERA.SZ+BD&<E1#`8N.YT\M%UPV.P>@KC27B$T7CU%(31V(]M_ZA"$)51
M$?'8>B/.[&2A3#C*GJXD+=4VZ\0:JRLTY;5)-%/$H1+[6N;<WM:_UBO?%*MP
MQS@T<QF;>:#3R,"V+;TO`&0[J,&5F4]F\>`=K8(P1BW;)))D3'5DV[XF3D;=
M7%?#"E29=%6*V^EJ)5/ZBV4%E,[ZO06`IV7%"RWPFU3_]3G8IKNL2+>P\V2=
M^=2ESI8R[6/;BQO7(,U%VL\Y,(&I.MA)EQ)UZ]&*A&J1FQ4EU)'546M*7#/Q
M+W)=D<D1J9/8I`755BD\5+7[-71JB;;2T\7ZQ('+$CLX##ZQHI3/+6><1L/1
M#?9^M"T:AC?B1@1(2<*A8[:7=:5=Q!-S;5]B1&*)C:?S3DNN:,D_M-X5K?>W
MMC\7Y^)<K,OK='$M3H;QLL_R%';GGLN.!F%S[/7P7:"5?DFVB1`3^DVH30+C
4QDL]S7*&43'42?\`&U52+MT$````
`
end

(Usual instructions for uudecode process.)

I apologize for only contributing occasionally to SoylentNews. I've been mostly off-line and, since Apr 2018, working on a website. It seem like a contradiction to have no radio, no television, no telephone, no Internet connection and work on a website but let me explain.

1. I'm ill.
2. I'm in pain and it adversely affects my concentration.
3. I require money for medical treatment.
4. I have to engage in what P.T. Barnam described as money-getting.
5. I must have some valuable skills. I've had a rather unusual career progression from Internet cafés to e-commerce to agriculture (including hydroponics) to search engines to databases (including OLAP and field consultancy) to film post-production to streaming video.

You'd think that I'd be an Internet millionaire. Maybe I am a millionaire but I'd like to make another million from another venture. (Actually, I've been a paper millionaire. From direct experience, that involved eating noodles multiple meals in a row.) With far less ethics, I would certainly be a millionaire. Unfortunately, the time has come to compromise my ethics. I hope to compromise my ethics as little as possible but this could be a succession of compromises.

A modern money-getting venture invariably requires a website. This is often the first mistake. A brief read of Joe Karbo's book: The Lazy Mans's Guide To Riches (either edition) describes mail order business principles which are incredibly similar to e-commerce. In particular, and from direct experience, a shopping basket with credit card integration will, at most, double sales. This magnifying effect means that zero doubled is still zero. A chain of logic, from emperical observations, leads to the conclusion that a venture should be profitable without a website or shopping basket. As I noted, the website is often the first mistake. This is because it is a trap.

The first failure is a venture which is focused on style rather than substance. Such websites invariably fail. A venture which correctly focuses on content may also fail but a certain level of expertise greatly helps. In part, this prevents an overly ambitious venture which is doomed to fail. It also ensures that a website is implemented competently. From emperical observation, the threshold of competence is just beyond the ability to use SQL JOIN with appropriate database indexes. I don't want to be contentious because there are numerous cases where not using SQL is a distinct advantage. However, there has to be a very good business reason to discard ad hoc queries. A search engine is a very good case for not using SQL. A search engine is a special case of partial ordered, partially materialized JOIN for the purpose of displaying the most likely results. Even if you do this, you very probably aren't doing it at the scale required to discard a conventional database.

From reading Tracy Kidder's book: The Soul Of A New Machine and the direct experience of friends, programming was a tedious practice which often occurred in absence of a computer. This involved a formal definition of a problem, use cases, data structures, flow-charts, pseudo-code and dry runs before making punch cards. The deployed program may be the second, third or fourth implementation. Nowadays, a prototype system may gain one million users before the venture is acquired by a conglomerate. Said conglomerate then grafts a subsequent implementation onto its NoSQL infrastructure. On this basis alone, software should be written for conciseness and correctness rather than efficiency. People complain that modern software is in a state of perpetual beta testing but this is a symptom of fast processors and cheap memory which only tangentially encourages poor programming practices.

Indeed, I am a victim of Moore's law. I am no longer a database field consultant because people make the same mistake on much larger systems. With the exception of some market leaders, the majority of organizations grow slower than Moore's law and the related laws for bandwidth and image quality until their function becomes trivial. As an example, it is difficult to write a spell check within 128KB RAM. A dictionary of words is typically larger than 128KB and word stemming was quite awkward to implement at speed. For this reason, when Microsoft Word required a spell check function, Microsoft merely acquired a company with a working implementation. It seems outrageous to acquire a company to obtain a spell check. It can be written very concisely in a scripting language but that doesn't work on an 8MHz system with 128KB RAM. Likewise, it is difficult to write a search engine within 16MB RAM but trivial to write in a scripting language with 1GB RAM.

Anyhow, the brief for the website is quite loose:-

1. It can say anything.
2. It can do anything.
3. It can look like anything.
4. There is no concern for efficiency, throughput or scale.
5. There is great concern about various classes of attack:-
   1. XSS and SQL injection are regarded as the same class of mis-matched string concatenation which is exacerbated by poor language design.
   2. Buffer overflow remains a problem.
   3. Denial of service remains a problem.
   4. Processor architectures don't meet specification.
   5. Attacks which are more sneaky include corrupting cryptographic keys in de-duplicated memory (and here and here). This could feasibly be used to attack a kernel, database, web server or language interpreter.

More specifically:-

1. The website should be truthful and legal. We're quite libertarian but there is no point being a martyr when there is no collective gain. Website is likely to cover medicinal use of cannabis and/or other contentious topics and therefore jurisdiction of website hosting may have to be considered carefully.
2. Majority of website will be static HTML. Remainder is contained and not essential. By law (and as a matter of professional competence), majority of content will be accessible to disabled users.
3. We choose an overly corporate style. Testing indicates that it is possible to veer significantly into vaporware and fiction when a corporate style is used.
4. Static website can be served from one thread of nginx web server or similar. HTTPS greatly complicates this scenario but a patch to FreeBSD9.0 from Netflix provides an extension to POSIX sendfile() which allows a kernel to implement hardware acceleration of HTTPS. Dynamic content may remain very vulnerable to denial of service.
5. May develop and test website in a deliberately hostile environment and then deploy in a slightly more sane environment. However, will not have physical control of servers and full administrative control is doubtful. Therefore, web hosting should be regarded as an untrusted and unreliable pocket in someone else's cloudy system.
6. Considered payment integration. We're not limited to existing options but found cryptographic currencies unsuitable as a default payment option. Block-chain and proof-of-work schemes also limit hosting options.

Plan to have a website which lists products speculatively and/or which are manufactured on demand. Trivial products include a buzzer game or an alarm clock. More substantial products may include a 5W per channel, 32 channel audio amplifier or a hydroponic controller in (matching) 1U rack-mount boxes. There is no reason to discontinue products unless parts become scarce. There is incentive to design products using the least variation of components. This allows more products to be manufactured from a fixed quantity of stock. However, it does make designs more susceptible to component scarcity. Most tediously, this may require standardizing resistor and capacitor values across products.

Although component costs may fall, overall costs, such as shipping, may rise. We don't want to be in the situation where someone orders a large quantity of products, all of the components are in stock but a sale loses money, either through labor/opportunity cost or returns on a marginal design. To counter this, component costs, labor and other costs will be defined as exponentials. For example, component cost may be defined as a specific cost on a specific day plus an increase of 10% per year. A monthly re-build of the static website plus hashes allows advertised prices to be honored - even if purchase occurs during site re-build. This may occur without tracking details too closely. If correctly configured, legacy products default to being too expensive rather than being too cheap.

Preferable to have a basic website, such as Wikipedia, CraigsList, Reddit or SoylentNews where the focus is on (allegedly) factual content. However, whipper-snappers prefer audio, video and animation. Ignoring this, desktop computer sales are steady in a growing market and the majority of casual use (and consumer spending) is on touch-screens devices of various descriptions. How is it possible to satisfy the majority of use cases while making a website which is easy to use while looking old, crufty and corporate? It is very simple and actually saves work. After the transition from the hierarchical Gopher protocol to the more free-form HTTP, people tried various navigation constructs including virtual galleries, virtual offices and virtual cities with street numbers. And then people moved back to a hierarchical structure. Steve Krug's book: Don't Make Me Think! explains in more detail. A brief overview is that navigation elements should be a blend of newspaper layout and department store signage. A trite summary would be "copy Amazon.Com slavishly" but Steve Krug notes that they've thought about it more than you and they have more turnover than you.

The website is also a place for orphaned content and software. Much of the software is CGI scripts and therefore they can be run via the website. The intention is for more people to ask "How many people update your website?" There is about 100,000 words or orphan content (excluding 80,000 words on SoylentNews) and at least 20,000 words which can be gleaned from e-mail. Also have more than four hours of video presentations. This can all be strung together with Makefile configurations and one of the numerous static HTML builders. While there is no shortage of content at the leafs of the website, it is tedious to repeatedly summarize branches towards the root. There is also the matter of intertwingularity in which anyone without a masters degree in library science is unlikely to devise a taxonomy which accomodates all content in a obvious manner.

This deficiency can be largely covered with large, clickable icons. The design of the icons is not too important. It helps if they don't change frequently but they function as landmarks in the navigational structure. Some people will merely select the red one followed by the stripey one or suchlike. Actually, icons towards the root of the navigational structure will be deliberately bad to provide fake antiquity. This is described fully in the Ye Olde section of the book: The Bluffer's Guide To Small Business. Essentially, antiquated style and unrounded pricing contributes to a corporate image of longevity, steadfastness and trustworthiness.

The icons have to be really obviously hyperlinked. This can be achieved by ignoring the fashionable, minimal, flat, Bauhuas style and having 1980s style mwm style bevelled edges. This was popularized in Microsoft Windows3.0 but was yet another innovation which was devised elsewhere. Bevelled edges can be automated with ImageMagick's convert -raise 12 or similar. What should be the size of the icons? I'm off-line and I have better things to do than find common sizes of icons. However, it should be noted that an icon bookmarked via a touch-screen interface may be promoted to the main menu icon tray typically found on a tablet or smartphone. From cached webpages, there are references to square icon sizes 16×16, 32, 48, 57, 60, 64, 70, 72, 76, 96, 114, 120, 128, 144, 150, 152, 155, 160, 167, 180, 192, 194, 256, 270 and 310 pixels. (256×256 was found on the ARM Developer website.) Cannot guarantee any size has been used by any device or that list is complete. However, much looks like Apple idiocy. In the era of MacOS7, it was common to design icons of different sizes for a 2 color MacIntosh, a 16 color MacIntosh, a 256 color MacIntosh and a 16 million color MacIntosh. Omissions were frequent (along with Balloon Help "text goes here" or suchlike). However, when all Apple devices support 16 million colors, vector image formats and MIP mapping, it is inexcusable to require multiple icon sizes in meta-data (sent to all clients) to support a device which may not be supported by Apple. Screw that, we're designing everything as 256×256 pixels and MIP mapping down by a factor of two or more. (We suspect the ARM Developers had a similar sentiment.) It is also useful to design buttons which are 256×96 pixels to be similarly scaled. This 8:3 ratio is very similar to the ratio of the legacy 88×31 pixel button format and it is quite easy to upscale legacy buttons. Also designing 8:1 ratio banners although this very probably requires further research.

Anyhow, there is lazy method to fill the void between the content. This will be complimented by text descriptions but there is primarily a casual interface which can be accessed easily on a small screen. The laziest part is that many of the icons are designed with OpenOffice Draw and pasted to GIMP. This technique casts vector to bitmap prior to MIP mapping, bevelling and other trashing of mediocre designs. Many designs use the same template but use different fonts, colors or gradients. Some designs are above average but the majority look like amateur corporate logos - and that's good enough. It looks worst on a thumb sheet where 300 logos are tightly bunched. However, it looks more moderate when there are 20 or less icons which are suitable spaced and captioned.

Icon spacing is adapted from some HTML <table> abuse learned from a ex-colleague. This is probably not original but he implemented a grid of products which automatically fills width of a web browser. This requires tweaking CSS display properties of <table>, <a> and, optionally, <span>. By default, <h1>, <p>, <table> and other HTML tags are rendered as block level elements. Whereas, <a>, <img> and other HTML tags are rendered inline. This is mutable on a per tag basis. Indeed, by having single-cell tables which are all the same width and with CSS display: inline, tables flow as a grid. When a web browser's width is changed (desktop window resize or touch-screen device is tilted), tables re-flow to a different number of columns. Furthermore, if <a> is set to CSS display: block then an icon, caption and all contiguous surround space is hyperlinked. Furthermore, where sub-categories have a hyperlink to a parent directory, use of <span> may make an up button unprintable. Therefore, any attempt to print a category of icons will omit the icon to the parent category and possibly allow the remaining icons to be printed in less space.

This menu system is most obviously applied to static HTML. It can also be applied to scripts. Indeed, it is possible to take 20 year old PERL4 scripts from Matt's Script Archive and - with suitable icons and meta-data - make them accessible like "apps" on a smartphone. But, like, in the cloud 'n' stuff. It is also possible to nest menus deeper than is typically allowed on a smartphone. However, we have yet to explain how menu and scripts all obtain the same style as the static website. This requires consideration of the history of dynamic content over HTTP. This probably omits developments but covers the important cases. Initially, there was no dynamic content. Then people wrote custom web servers so that URLs were dynamic rather than mapping directly to static files. Then there was server-side image maps with a URL suffix in the format ?(x,y). Then there was forms with HTTP Method GET parameters suffixed in the format ?foo=1&bar=2. Then there was client-side image maps. Then there was HTTP Method POST. This detacted parameters from URLs and was useful for file upload and to circumvent client implementation limitations of HTTP Auth. Then there was extended HTTP Method POST which preserved file meta-data, such as file name. Within a CGI environment on a Unix server (one of many server environments), HTTP Method is passed to a script as an environment variable. HTTP Method GET parameters are passed as a separate environment variable whereas HTTP Method POST is available via Unix stdin. This is rather messy but fits with the historical development of dynamic content. Command line arguments aren't used but this dormant mechanism can be used for testing scripts or using them outside of a CGI environment. In the laziest case, it is possible to create sub-directories within /cgi-bin/ and within each sub-directory, have a shell script called index.cgi with the following content:-

#!/bin/sh

./prog.cgi $@ | ./wrapper.cgi $@

This allows separation of content and style in a manner which was not feasible when CGI was developed in the 1990s. It remains infeasible to deploy this on a high-volume website but it significantly eases development. The shell script uses a Unix pipe to pass the output of the application logic to a styling wrapper script. This has several uses. It allows a naked application to be tested without HTML style. The shell script propagates environment variables and command line parameters to the application and the wrapper. This allows common functionality, such as session management, to be off-loaded to the wrapper. Unfortunately, only the application receives HTTP Method POST parameters. Fortunately, the majority of scripts do not require HTTP Method POST and the remainder may pass parameters of interest to the wrapper. Meanwhile, the wrapper may re-write URLs to ensure session keys are always propagated. The wrapper may perform performance measurements, grammatical checks, sampling of output, redaction and rate limiting. It is also possible for the wrapper to distill a superset of HTML into something more conventional. When such functionality is separated, programs are typically less than 10KB and the wrapper is less than 100KB (but common to all scripts written in all programming languages from all sources). It is also possible to provide a central location for HTTP cache header configuration. Obviously, as functionality increases, processing load also increases. This make the technique unsuitable for popular websites unless caching is very well tuned. Regardless, it is a technique to shrink legacy scripts while improving the quality and consistency of their output.

There are three unresolved problems when using a wrapper script. All are easily overcome. The first problem is that scripts typically make incorrect URL references to themselves. A script should use the provided environment variable. It is more common to use $0 (or similar) or a parameter name defined in the begining of the script. It is tedious to correct and test all cases. The second problem is the HTTP looks like e-mail and output from CGI scripts is no exception. Specifically, RFC822 style colon separated headers (with optional tabs, spaces and indented line breaks) are followed by a blank line and then the content. The header should specify a MIME type and therefore, in some modes, the output of a script may not be HTML. In this case, the wrapper should pass-through the content unchanged. The third problem is that trivial implementation of the wrapper may introduce considerable latency. This is greatly reduced if processing occurs incrementally. In the trivial case, it should be possible to distill </box> separately from <box> or defer processing of <rainbowtext> until the tag is closed.

Although the wrapper creates additional system bottleneck and additional latency, this is unlikely to be a significant problem if the application is I/O intensive (for example, if the application makes heavy database queries) or milti-core hosting is available. This was typically unavailable when CGI was typically deployed on 25MHz single-core servers with 16MB RAM. For low-volume use, this is not a problem for a 700MHz dual-core server with 256MB RAM. Indeed, it runs quite well on many credit-card size computers.

This is not a complete guide to boot-strapping and mega-corporation by only using electronics which will fit in a rucksack but it demonstrates that it is possible to make something which is above average without considerable skill. Joe Karbo noted that an average product or service covers half of the market. Presumably, further improvement leads to diminishing returns. Also, in a case of less is more, being the best may also imply being the most abstract and detached.

Without being too abstract, it is possible to comprehensively satisfy one part of the Joel Test. Specifically, it is possible compile a web server, compile a database, compile a script interpreter, build a website, tutorial videos and/or embedded software with one invocation. This is important because style isn't.

I will be posting considerably less technical content over the next month or three. If you want more technical content, I highly recommend an introduction to the Plan9 Operating System. I also highly recommend DTACK Grounded which describes how to make a fast, small computer using a Motorola 68000 processor optionally with a 8087 co-processor. Similar systems (and dup) have been covered on SoylentNews.

(This article has some rather blunt observations about the representation of race, nationality, sex, sexuality and religion by multiple litigious media companies. I strongly doubt that the editors of SoylentNews would ever approve official publication of such an article and therefore I publish this, without editing, in a personal capacity.)

I really want to watch some high-quality, fan produced StarTrek. However, the pipeline is exhausted. This is due to the rights holder being extremely restrictive. This leaves me quite disgruntled.

I've made friends through a shared interest in StarTrek. It has also been useful in a professional capacity because it allows some technical concepts to be conveyed more concisely. I've been persuaded to visit Pages Bar in Pages Street, Westminster, London and (due to booking error) persuaded to attend a StarTrek convention where Ethan Phillips (Neelix), Jennifer Lien (Kes) and William Shatner (the third best actor to portray Captain James Tiberius Kirk) were guests of honor.

Pages Bar was the most fun but that closed many years ago. Saturday evenings were for StarTrek. other evenings were for other science fiction themes. The bar had a large model NCC1701D Enterprise hanging from the ceiling and some of the tables were in the style of 10 Forward. About 1/3 of the patrons wore StarTrek uniforms, although ranks below commander were quite sparse. Some of the remainder dressed as Borg or Klingons. (Top tip: Wear a tampon under a Klingon prosthetic forehead to absorb sweat.) There was a dealer table run by a guy who was known as the Ferengi due to his generous discounting policy. The bar served Romulan Ale (lager with a dash of blue food coloring) and Tribble Burgers (which were probably about 90% beef and 10% horse.) It also showed official episodes of StarTrek, fan productions and promotional video for conventions. There was often one guest of honor, such as George Takei (Sulu) or Garrett Wang (Harry Kim from Voyager). Every Saturday was like a mini-convention.

When the rights for StarTrek transferred from Viacom to Paramount, the latter scoped around to see if any rights required enforcing. When it encountered Pages Bar, the reaction was akin to "WTF is this???" Paramount made a token effort to enforce rights. Romulan Ale and Tribble Burgers were dropped from the menu but it was otherwise unaffected. It was generally understood that Pages Bar pushed a little too far and shouldn't push further. This was respected by fans and Paramount gained considerable goodwill.

Similar truces stood for many years but during this period, the cost of quality cameras crashed, the cost of post-production crashed and a growing number of actors from the growing canon were willing to participate in a growing number of productions. Cribbing from reason.tv's brief history of fan productions prior to StarTrek: Axanar getting sued, this first occurred in 1985 when George Takei appeared in Yorktown: A Time To Heal and then Chekov, Uhura and Tuvok appeared in the (rather good) fan production, StarTrek: Of Gods And Men. The latter also had Gary Graham from Alien Nation.

I find most of the legacy fan productions to be unwatchable. The seven seasons of StarTrek: Hidden Frontier rivals The Next Generation by size and is widely available. However, many of the sets were rendered with less than 100MHz processing power and composited to NTSC VHS at 525i before being archived, sampled, uploaded and transcoded to 360p. Early episodes of StarTrek: New Voyages and Starship Farragut have equal distribution quality. StarTrek: New Voyages becomes extremely watchable from Episode 8: Kitumba. It helps that Episode 9: Mind Sifter has a retro 5:4 aspect ratio and is consistent with StarTrek: Of Gods And Men. Unfortunately, StarTrek: New Voyages finishes at Episode 10. Starship Farragut has an astounding set but the acting hasn't improved over 20 years. The actors have merely gone from having the presence of young middle-managers to having the presence of old and fat middle-managers. One is more suited to the rôle of dental receptionist with Stage 4 RBF rather than StarFleet Communication Officer. It is ass-clenchingly awful and not in an amusing way.

With seven episodes of StarTrek Continues each raising funding and then StarTrek: Axanar raising US$1.3 million of crowd-funding across Kickstarter and IndieGogo, Paramount/CBS (or whatever it is called nowadays) decided to set rules which prohibited anything beyond 2×15 minute productions - and no canon cast or crew allowed in *any* rôle, paid or unpaid. And Paramount/CBS set these rules with nothing ready for StarTrek's 50th anniversary. Thanks, guys.

Like some other members of SoylentNews, I paid to see the first two StarTrek reboot films and I decided that I wouldn't be conned on a third occasion. Zachary Quinto is surprisingly good as Spock and Karl Urban is versatile as McCoy but that isn't enough to redeem it. I hear that the series: STD StarTrek: Disco Discovery is also awful, in part because it differs more from canon than many fan productions and, in part because of an overt progressive agenda. StarTrek is renowned for tolerance and harmony but occasionally pushed too far. A kiss between Kirk and Uhura was censored in Alabama, although that's a place more closely associated with incest than racial tolerance. The Original Series and The Next Generation also attempted to cover racism and homophobia more tactfully. However, The Next Generation had an unconscious undercurrent of casual racism where, for example, black actors portray a violent race. Also, good Klingons are portrayed by actors of Christian, West African descent and bad Klingons are portrayed by actors of Muslim, East African descent. Furthermore, the Ferengi have a remarkable similarity to the stereotype of a short, ugly, money obsessed, Hollywood Jew - complete with the sexual objectification of women. (StarFleet also has an acute shortage of gallium and therefore none of the LEDs are blue.)

Look further afield and other science fiction is just as bad. I hear that the StarWars triple trilogy is awful. Episode 4, Episode 5 and Episode 6 are swashbucklers in space with excellent three act structure - individually and as a trilogy. Repeating this on another scale was ambitious. Unfortunately, it failed. Technical problems were overcome, such as matching analog and digital cinematography. However, Episode 1 has no plot. (It also has Jar Jar Binks which some believe is a German, Italian and/or Latino immigrant stereotype.) Journalists were shown pre-release screenings where 80 minutes of the footage was shown in a random order. Ostensibly, this was to prevent the plot being published but this was soon discovered to hide the lack of plot. Episode 2 has been described as "attack of the cloned plot". Episode 3 has the unenviable job of tying two bags of shit to Episode 4. Given the circumstances, this was achieved competently. However, that's not a recommendation.

Disney StarWars was made in record time with an overlap of cast and crew with the StarTrek reboots. When their work on StarTrek scores zero out of 2 and previous work on StarWars Episodes 1-3 scores 1/2 out of 3, I'd be an idiot to pay to see Episodes 7-9 in a cinema. The film: Solo may also disappoint. Despite all of the advances in textiles and fur rendering, audiences complain that Chewbacca looks worse than 40 years ago. At best, this is a failure to meet raised expectations.

There is also the issue of affirmative action casting. After StarTrek cast a white male captain (William Shatner as Kirk) then another white male captain (Patrick Stewart as Jean-Luc Picard), StarTrek cast a black male commander (Avery Brooks as Sisko) than a white female captain (Kate Mulgrew as Mrs. Columbo in space Janeway). With far more clumsiness and the full wisdom of hindsight, StarWars Episodes 7-9 has a black male protagonist and a white female protagonist. What an original combination! Are they called Cisco and Janeway? And, apparently, a character from Episode 5 is now pansexual. I hope this gets viewed with equal clumsiness.

A much bigger issue is that big budget action films have converged on the hero's journey plot. Unfortunately, the film industry has been following the example "beat sheet" from the book: Save The Cat to the extent that the film: Pacific Rim, the film: Skyfall, the film: The Dark Knight Rises and many others can be played concurrently. Minute-by-minute, almost the same plot is followed, complete with blatently underlined plot developments for the protagonists' and antagonists' intrinsic and extrinsic motives. Among other matters, this fully explains Bruce Wayne's pointless flashbacks and the 58 second dojo scene in Pacific Rim. Film industry experts, such as Lord Putnam, thought that finances would fall apart by 2010 but inertia has carried it to 2018 and perhaps much further. How many times are you going to pay to see the same film made with different actors?

Meanwhile, an astounding amount of money is spent on television. Black Mirror costs more than US$1 million per episode. Westworld costs more than US$3 million per episode. The Crown costs more than US$5 million per episode. Adjusting for inflation, budgets are typically lower than The Next Generation. However, no-one is paying, for example, US$6000 per second to composite a star-field at warp. And, at a minimum, everyone shoots lossless digital at 3840×2160, 60FPS, 10 bit per channel. The money spent to subscribe to video on demand and the money spent on productions has lured multiple Oscar winners away from film and stage. Regardless, much of this big budget television can only be streamed from proprietary systems over the Internet. Although, even when Black Mirror was made for broadcast, it was never cut to length or with regular advert breaks.

CBS's reaction to a fan production reaching US$1.3 million was to sue and shut it down. It had reached professional quality and a professional budget. It was also produced with love and on its own schedule. That was too much competition. However, given that CBS is the center of an eco-system where it has sole discretion about the revenue model, CBS could have chosen many other options:-

1. Recruit the best talent.
2. Allow fan productions to continue.
3. Move to a licensee only revenue model.
4. Allow 2×15 minutes under fair use and charge licensing fees on anything else.

I envision a scenario where a credit card processor takes 3%, crowd-funding platforms take 2% and CBS takes 5% or so. A worked example for StarTrek: Axanar's US$1.3 million funding would be US$39000 for credit card processing, US$26000 for crowd-funding, US$65000 for licensing and the remaining 90% (US$1.17 million) for production. Licensing requires due diligence, signing a standard contract, approving a script and approving footage. This would be per episode or per film and budgets could grow by at least a factor of five per production over an unlimited number of teams.

Prospective teams, in their own time and at their own expense, would have incentive to pitch productions which are consistent, original, interesting and plausible. Fans would choose the best proposals with their own money prior to production. CBS would take fees while enforcing minimal regulations. It would be easy to trace the majority of money from the largest productions. In particular, it would be implausible to raise significant money on an obscure website without it being discovered by CBS.

Even without this quality control, the better fan productions have been consistent with each other. The Original Series and StarTrek Continues are rigorously consistent to the extent that Gene Roddenberry's son regards both as canon. I know a partially-sighted science fiction expert who is unable to distinguish any difference beyond a holodeck, a counsellor and Vic Magnogna's less stilted delivery as Kirk. And where it differs, it is preferable.

Where StarTrek Continues overlaps with StarTrek: Axanar, it is consistent. Likewise where StarTrek: New Voyages overlaps with StarTrek: Of Gods And Men and The Original Series. Anyone failing to meet this established standard will lose the respect of their peers. With fan efforts, the peer pressure is more important than turnover or licensing. However, with the three most recent StarTrek films and the seven most recent StarWars films, quality has been secondary. Goodwill has evaporated and it may not return.

I'm a fairly typical case where cinemas have lost at least US$60 of revenue. I assume that there are millions of similar cases. I'm willing to forego a big screen experience and put some of that shortfall into quality, small screen, fan productions. However, my favorite options are closed due to a licensor without vision. StarTrek could become a vast fan led franchise of impecable quality. I dare to suggest a commons with a shepard. But, on CBS's current path, I have taken my business elsewhere. The condition to bring it back isn't particularly high but I won't wait forever.

In the interim, I'll watch stuff that I've already seen or find the nearest alternatives. Many businesses compete with their previous work and some preference is due to familiarity. Microsoft is a great example. However, there are few businesses where neophile customers prefer the work from 30 years ago or 50 years ago.

Executives wonder why we prefer StarTrek Continues, StarTrek: Axanar or science fiction such as The Expanse. They have a simple, positive message without being preachy. There is plausible diversity without casual racism or sexism. (1960s style uniforms are the major exception to sexual equality.) For amateur productions, we can overlook a large amount of lopsided diversity because fans represent themselves; often at a financial loss. Meanwhile, official productions are decreasingly successful at casting people who embody Gene Roddenberry's vision of harmony. William Shatner and Leonard Nimoy were quietly Jewish without issue. George Takei has been as openly homosexual as possible while pursuing a career. I get the impression that Zachary Quinto would rather be omitted from discussion and perhaps I've already said too much. Where did it go wrong? Denise Crosby as Tasha Yar had a robosexual plot and then got written out of The Next Generation. Apparently, appearing in Playboy magazine did not prompt the studio to drop her. Whereas, the alleged actress, Alice Eve, and, to a lesser extent, the Scientologist, Kirstie Alley, have been notoriously transphobic. Alice Eve is best known for being filmed in underwear and next best known as the antagonist in Black Mirror, Season 3, Episode 1: Nosedive. (Charlie Brooker: You are an arch troll.)

Maybe the StarTrek canon has spread so far that it has all become doggerel. However, yet another hero's journey and a shutdown on homages isn't a long-term strategy. CBS is in a unique position that few brands can ever hope to achieve. But it is trashing cultural heritage with a random series and a few cheesy films. Follow the example of Lego. Unfortunately, Lego has at least a 1×16 up its ass about MiniFig licensing. Ignoring that, Lego has sold fan designs on a revenue share basis, encourages conventions and encourages the use of unofficial software in combination with official hardware. We probably like Lego more than StarTrek. We probably spend more on it too.