SoylentNews

Richard Nixon was not a crook!

SoylentNews is all politics all the time. Richard Nixon was POTUS when the dd command was introduced into Unix.

I write a lot of Bash code. I wrote my own web server in Bash. I wrote my own YouTube downloader in Bash.

I have need of a buffer to smooth the throughput of a pipe because wget will write as soon as data arrives which is usually the right thing to do but there are some rare times when YouTube produces data very unevenly.

Ken Thompson wrote dd back in June of 1974 when Nixon was in power and every Soylent political troll was definitely already alive.

Now dd is not a buffer but a buffer can be constructed using dd and a pipe.

Producer | dd ibs=4096 obs=65536 | dd ibs=4096 obs=65536 | Consumer

Yes there are two invocations of dd and both invocations are identical. Block sizes are uneven so dd will collect small blocks and output large blocks. Blocks flow from the first dd through the pipe to the second dd and together dd and pipe are the buffer.

The dd pipe buffer is an ingenious construction but its history is hard to find and it might have been reinvented many times since it is deceptively simple. So I will attribute the dd pipe buffer to Carl Dong.

Cue political trolls because SoylentNews.

So I see everybody hates QUIC. Bad news for me because all my everything runs QUIC.

I don't like big bloated web servers and instead I use php -S for HTTP and socat for SSL so I've been living on the menstruating edge of technology where I forked lsquic and added netcat style features to run an external script or connect to a TCP port.

I was hoping after RFC 9000 got published as an RFC and the QUIC protocol stopped changing then some people might be interested in wanting to develop some small command line tools for QUIC instead of big bloated web servers and huge content delivery networks.

I'd really like a netcat for QUIC but I guess I'm just going to have to keep maintaining my own fork of lsquic because everybody hates QUIC.

Mosh is a remote terminal over UDP but it relies on SSH for authentication and that's just silly.

Mosh without SSH is an old request.

Hello. I believe many are attracted to mosh like me because they
cannot run ssh, and sad to realize, although advertised as a
replacement of ssh, mosh depends on ssh.

The reasons users prefer not to use ssh could be because it is plain
out blocked. Like in Iran, or stemed, like in China?.

I agree. Some of us live behind firewalls, and sometimes the UDP holes are easier to access than the TCP holes.

The question was asked during a presentation about Mosh.

Do we support pluggable login providers, so we could do something instead of SSH? Yes and no. The simple answer is we don't support any login at all. So you can do what you like.

There are projects to do Mosh authentication over UDP: udpserv and Mosh-UDP are a couple of them.

I made my own: Mosh Mallet

https://gitlab.com/Zinnia_Zirconium/youtsuck

I wrote my own YouTube downloader in Bash and it's the fastest I've ever used.

nobody@localhost:/tmp$ time bash youtsuck.sh https://www.youtube.com/watch?v=z8yCIJPJP5I | grep -o '^.*playback.'
https://r1---sn-5uaezn6r.googlevideo.com/videoplayback?

real    0m0.125s
user    0m0.006s
sys     0m0.000s
nobody@localhost:/tmp$ time bash youtsuck.sh https://www.youtube.com/watch?v=z8yCIJPJP5I? | grep -o '^.*playback.'
https://r1---sn-5uaezn6r.googlevideo.com/videoplayback?

real    0m0.109s
user    0m0.008s
sys     0m0.002s

Oh yes and it does its own cipher translation in Bash.

nobody@localhost:/tmp$ cat /tmp/3f282887ab0d169e9b017a84a6be6073.json
[["sts",18613],["swap",9],["swap",4],["reverse"],["swap",54],["reverse"]]

Let's remember how slow youtube-dl is.

nobody@localhost:/tmp$ time python youtube-dl -4 -g -f 18 https://www.youtube.com/watch?v=z8yCIJPJP5I | grep -o '^.*playback.'
https://r1---sn-5uaezn6r.googlevideo.com/videoplayback?

real    0m2.527s
user    0m1.727s
sys     0m0.089s

youtube-dl can suck it.

YouTube famously uses a rolling cipher and effective downloader tools need to be able to decipher it to produce useful links to video files. The cipher changes every few days so downloader tools avoid the need for daily manual updates by automatically downloading the JavaScript implementation of the cipher from YouTube and caching the result.

I use three downloader tools that have some automated mechanism for dealing with cipher updates.

• Athlon1600/youtube-downloader caches the original JavaScript and parses it on every execution.
• youtube-dl interprets the original JavaScript on first execution and caches a JSON representation of the cipher.
• JWZ youtubedown parses JavaScript once then rewrites its own code to insert cipher instructions.

The cipher changes every few days except old ciphers still work for about a month. The get_video_info API accepts an "sts" parameter to request an old cipher. The get_video_info API is always faster than requesting an HTML page (~150ms versus ~400ms) but not every video is available from the get_video_info API.

As far as I can tell JWZ pioneered the trick of using "sts" to request an old cipher. Every other mention of the trick that I can find credits JWZ. Using an old cipher was the default strategy for youtubedown between March 2015 and August 2018.

YTCrack is a notable example for being a DOS program that reads an FLSTSKEY environment variable. JWZ is credited.

(And then there is this particularly ignorant blogger who claims "sts" is "some kind of session identifier" which is entertainingly wrong.)

I never gave up on get_video_info and "sts" and old ciphers. When JWZ abandoned the trick I still used it. I forked youtubedown and I manually updated the old cipher every month when it expired. For those hard to download videos I used youtube-dl to do HTML scraping as a fallback.

Since the get_video_info API is much faster when it works I want to avoid HTML scraping whenever possible. But I also want to reduce my use of youtube-dl.

So I reinstated HTML scraping in my fork of youtubedown and enabled self-modifying code in production. Probably it will corrupt itself eventually. Especially since I am running it on demand from a web server. Except I have zero users so maybe it will be fine.

My current strategy for answering a streaming video request is try the fastest method first.

1. [~150ms] athlon1600-youtube-downloader-fast (get_video_info API with "sts" and old cipher)
2. [~200ms] youtubedown.fast (get_video_info API with "sts" and old cipher)
3. [~400ms] athlon1600-youtube-downloader-fast (HTML scraping with new cipher caching)
4. [~500ms] youtubedown.fast (HTML scraping with self-modifying new cipher)
5. [~800ms] youtube-dl (IPv4 only HTML scraping first attempt)
6. [~800ms] youtube-dl (IPv4 only HTML scraping second attempt)
7. [~800ms] youtube-dl (IPv6 only HTML scraping third attempt)
8. [~800ms] youtube-dl (IPv6 only HTML scraping fourth attempt)

All of these tools I modified.

• athlon1600-youtube-downloader-fast
• youtubedown.fast
• youtube-dl only YoutubeIE build script

youtube-dl is still the slow desperate last resort even after I wrote a script that reduced its execution time from ~3000ms to ~800ms. Best case scenario is youtube-dl never runs.

I do a lot of work to provide low TTFB on 360p previews of music videos for zero users.

The world of YouTube downloaders is a kakistocracy and youtube-dl is the undisputed king.

I've been using JWZ youtubedown instead for the past five years even though youtubedown is designed only to download files and I want links. With a few modifications I can make youtubedown produce links and it's faster than youtube-dl. Everything is faster than youtube-dl.

I've known about Athlon1600/youtube-downloader since January and I've been thinking about getting around to making it even faster. So I started working on it last week.

How slow is youtube-dl? Well let's see.

First let's make the results completely fair and eliminate latency to YouTube by using Linode.

nobody@localhost:/tmp$ ping -c 5 www.youtube.com
PING www.youtube.com(yb-in-x88.1e100.net (2607:f8b0:4002:c09::88)) 56 data bytes
64 bytes from yb-in-x88.1e100.net (2607:f8b0:4002:c09::88): icmp_seq=1 ttl=111 time=1.27 ms
64 bytes from yb-in-x88.1e100.net (2607:f8b0:4002:c09::88): icmp_seq=2 ttl=111 time=1.34 ms
64 bytes from yb-in-x88.1e100.net (2607:f8b0:4002:c09::88): icmp_seq=3 ttl=111 time=1.39 ms
64 bytes from yb-in-x88.1e100.net (2607:f8b0:4002:c09::88): icmp_seq=4 ttl=111 time=1.41 ms
64 bytes from yb-in-x88.1e100.net (2607:f8b0:4002:c09::88): icmp_seq=5 ttl=111 time=1.48 ms

--- www.youtube.com ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 1.274/1.383/1.481/0.073 ms
nobody@localhost:/tmp$ ping -c 5 r1---sn-5uaezn6r.googlevideo.com
PING r1---sn-5uaezn6r.googlevideo.com(2607:f8b0:4002:14::7 (2607:f8b0:4002:14::7)) 56 data bytes
64 bytes from 2607:f8b0:4002:14::7 (2607:f8b0:4002:14::7): icmp_seq=1 ttl=123 time=0.769 ms
64 bytes from 2607:f8b0:4002:14::7 (2607:f8b0:4002:14::7): icmp_seq=2 ttl=123 time=0.863 ms
64 bytes from 2607:f8b0:4002:14::7 (2607:f8b0:4002:14::7): icmp_seq=3 ttl=123 time=0.840 ms
64 bytes from 2607:f8b0:4002:14::7 (2607:f8b0:4002:14::7): icmp_seq=4 ttl=123 time=0.850 ms
64 bytes from 2607:f8b0:4002:14::7 (2607:f8b0:4002:14::7): icmp_seq=5 ttl=123 time=0.839 ms

--- r1---sn-5uaezn6r.googlevideo.com ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 0.769/0.832/0.863/0.037 ms

OK we have low ping. Let's see how badly youtube-dl performs.

nobody@localhost:/tmp$ time python youtube-dl -g -f 18 https://www.youtube.com/watch?v=z8yCIJPJP5I | grep -o '.*playback.'
https://r1---sn-5uaezn6r.googlevideo.com/videoplayback?

real    0m3.136s
user    0m2.353s
sys     0m0.057s

3136 milliseconds. That's terrible.

I know! Let's compile youtube-dl into python bytecode! That'll make it fast. Right?

nobody@localhost:/tmp$ ( mkdir compile ; cd compile ; unzip ../youtube-dl ; python -OO -m compileall . ; zip -0 ../youtube-dl.zip $( find -name '*.pyo' ) )
nobody@localhost:/tmp$ time python youtube-dl.zip -g -f 18 https://www.youtube.com/watch?v=z8yCIJPJP5I | grep -o '.*playback.'
https://r1---sn-5uaezn6r.googlevideo.com/videoplayback?

real    0m2.257s
user    0m1.450s
sys     0m0.071s

2257 milliseconds. Really?

I know! Let's go get the lazy extractors edition of youtube-dl and compile it to python bytecode!

nobody@localhost:/tmp$ ( mkdir lazy ; cd lazy ; unzip ../youtubedl-lazy ; python -OO -m compileall . ; zip -0 ../youtubedl-lazy.zip $( find -name '*.pyo' ) )
nobody@localhost:/tmp$ time python youtubedl-lazy.zip -g -f 18 https://www.youtube.com/watch?v=z8yCIJPJP5I | grep -o '.*playback.'
https://r1---sn-5uaezn6r.googlevideo.com/videoplayback?

real    0m1.760s
user    0m1.087s
sys     0m0.034s

1760 milliseonds. Screw this.

How much faster can Athlon1600/youtube-downloader be? Well let's see.

nobody@localhost:/tmp$ time php athlon1600-youtube-downloader-fast/public/video_info.php   https://www.youtube.com/watch?v=z8yCIJPJP5I | grep -o '.*playback.'
https://r1---sn-5uaezn6r.googlevideo.com/videoplayback?

real    0m0.147s
user    0m0.035s
sys     0m0.005s

147 milliseconds. Huh.

Yeah. I think youtube-dl is too slow.

Edit: I made a couple of changes to the lazy extractors edition of youtube-dl

nobody@localhost:/tmp$ cat compile-youtubedl-lazy.sh
#!/bin/bash
temp="$(mktemp -d)" || exit
cp -pv youtubedl-lazy "$temp"
pushd "$temp"
unzip youtubedl-lazy
echo -e '\n\n_ALL_CLASSES = [YoutubeIE]\n' >> youtube_dl/extractor/lazy_extractors.py
find -name '*.py' | xargs sed -i 's/^import re\b/import regex as re/'
python -OO -m compileall .
zip -0 youtubedl-lazy.zip $( find -name '*.pyo' )
popd
cp -pv "${temp}/youtubedl-lazy.zip" .

nobody@localhost:/tmp$ bash compile-youtubedl-lazy.sh > /dev/null 2>&1
nobody@localhost:/tmp$ time python youtubedl-lazy.zip -g -f 18 https://youtu.be/R-fhIo0TaDs | grep -o '.*playback.'
https://r3---sn-5uaezn7e.googlevideo.com/videoplayback?

real    0m0.807s
user    0m0.349s
sys     0m0.025s

807 milliseconds. Now youtube-dl is tolerably slow.

Edit: I made a couple of changes to the vanilla version of youtube-dl

nobody@localhost:/tmp$ cat compile-youtubedl-only.sh
#!/bin/bash
temp="$(mktemp -d)" || exit
cp -pv youtube-dl "$temp"
pushd "$temp"
unzip youtube-dl
echo 'from .youtube import YoutubeIE' > youtube_dl/extractor/extractors.py
sed -i 's/GenericIE/YoutubeIE/g' youtube_dl/extractor/__init__.py
find -name '*.py' | xargs sed -i 's/^import re\b/import regex as re/'
python -OO -m compileall .
zip -0 youtube-dl.zip $( find -name '*.pyo' )
popd
cp -pv "${temp}/youtube-dl.zip" .
nobody@localhost:/tmp$
nobody@localhost:/tmp$ time python youtube-dl.zip -g -f 18 https://www.youtube.com/watch?v=R-fhIo0TaDs | grep -o '.*playback.'
https://r3---sn-5uaezn7e.googlevideo.com/videoplayback?

real    0m0.757s
user    0m0.300s
sys     0m0.042s

757 milliseonds. Go go slow youtube-dl

North American Scum

oh I don't know, I don't know, oh, where to begin
when we're north american
but in the end we make the same mistakes all over again
come on north americans

we are north american scum
we're from north america

Why does it seem like everybody everywhere else in the world finds ways to free internet except North Americans.

NETWORKS &BROWSING TIPS Archives » AndroidTechVilla

Brazil, Pakistan, Zimbabwe, Myanmar, Syria, Nigeria, Angola, South Africa ..........

Whut? South Africa?? Elon Musk isn't giving free internet to his native country? What a bum.

Ok so maybe there's like one guy who had the right idea.

Do better North Americans. I want free internet on every street corner and beaming out of the sky. I want people to want it.

QUIT ASKING

Step 1) Pay your bill.
Step 2) Get on Internet.

Yawn.

So I ran nmap -sU on Comcast xfinitywifi and it said every UDP port is open? Whut?

I'm like I know I read somewhere xfinitywifi blocks UDP.

It's true. Every UDP port is open after you do the silly knock. I modified AndIodine to pick a random port for raw mode. Of course the app needs the iodine server to listen on every port but that's one line of iptables.

Now pruneman can have automatic xfinitywifi penetration he always wanted. Sorry I didn't do Firefox. I wish Psiphon would though.

Residential Neighborhood Guest Network Road Warrior should be a thing. I wonder if it works on CBCI too.

No I'm not Captain Midnight but the recent merger of Dish Network and Boost Mobile last month reminds me of him.

John R MacDougall was that guy who jammed HBO in 1986 to protest the scrambling of satellite TV. Before scrambling anybody who owned a satellite dish could receive the HBO signal for free just by having the right equipment. Scrambling hurt his business of selling satellite dishes. Unfortunately the only thing Captain Midnight accomplished by jamming a satellite was criminalizing his method of protest.

Now I never had cable or satellite so I was never involved but we're kinda in a similar situation with mobile internet access.

A while back I bought a mobile Wi-Fi hotspot device from Boost Mobile. I've noticed only two differences since the Dish Network Boost Mobile merger. The Boost Mobile logo now has a Dish in it. And my device now connects at 1900MHz instead of 2500MHz.

After I bought and paid for my Boost Mobile device I activated it but I never paid for a plan because I realized I didn't need to. As seen in How to activate boost mobile there's a choice during activation to add money later. The device is immediately connected to the internet but any attempt to use it as a Wi-Fi hotspot produces a redirect to the Boost Mobile account page as a reminder to add money.

So basically just by having the right equipment I got heavily censored internet access. Enter every Belarusian's favorite censorship circumvention app: Psiphon. Now Psiphon alone won't get around the Boost Mobile redirect. But an HTTPS proxy will. I searched the web for "proxy port 80" and chose a proxy. I configured Psiphon to use a proxy with a custom HTTP header "Host: boostmobile.com"

And I got free uncensored internet. Well not exactly. Psiphon also has limits like for example port 22 is blocked. Running SSH on other ports isn't much trouble. There's the inherent reliability issue of using entirely third party servers. I'm perfectly capable of running my own proxy and VPN server if I need to but I don't need to. Free internet.

So I wonder if Captain Midnight is still active in getting free signal using equipment you already own. Because times have changed since free satellite HBO but I imagine there must be some demand somewhere for free internet access when Dish Boost Mobile is just giving it away to anyone who can install an app.

So I scraped song metadata from Jango Radio and made it searchable.

I've been looking for an appropriate forum to mention I did it. I don't think an appropriate forum exists. Coding doesn't matter. My accomplishment simply doesn't register as an accomplishment when nobody has heard of Jango Radio. I think I know why.

One Sound Promotion is the real business behind Jango Radio and Radio Airplay. Jango Radio is a free music streaming service. Radio Airplay is a paid music promotion service. Revenue from Radio Airplay obviously pays for Jango Radio. I knew this already.

Now I'm convinced One Sound Promotion doesn't have any customers.

71.3% of content on Jango Radio is music from independent artists who paid Radio Airplay for promotion. Only 28.7% of content is music from professional artists someone might expect to hear everywhere. I know the exact numbers because I have all the metadata.

Business looks good when Jango Radio receives uploads every hour from paying customers. Except I don't think independent artists really are Radio Airplay customers.

The default search results on Jango Index are the newest 100 songs uploaded to Jango Radio. Usually the new 100 are all independent artists from Radio Airplay unless there's an upload of professional new releases in progress. So I looked at where releases from independent artists are coming from.

YouTube is a popular destination for music. Independent artists from Radio Airplay who appear on Jango Radio also appear on YouTube. They have automatically generated channels which credit music distributors like CD Baby and DistroKid and ONErpm and TuneCore. Music distributors distribute to multiple streaming services like Spotify and Deezer and Pandora and YouTube and many more including also Radio Airplay which is Jango Radio.

So independent artists pay music distributors for package deals which happen to include Radio Airplay. Independent artists don't pay Radio Airplay directly. Radio Airplay receives revenue indirectly through music distributors. The independent artists whose music is uploaded to Jango Radio every hour don't know Radio Airplay exists.

Jango song_id 304662 ("Jango Thin Skin" by Split P) includes the lyric: "How many sales have you made on TuneCore?" and that should have been my first clue.

https://jango-index.ml/?song_id=304662 [explicit]

So! I'm convinced nobody has heard of Jango Radio because Radio Airplay is an unnoticed cash cow grazing at the edge of a vast music distribution network.