Microsoft's Windows Subsystem for Linux is coming to all Windows 10 users (archive):
You won't have to be a tester to try Windows 10's new, built-in Linux kernel in the near future. Microsoft has confirmed that Windows Subsystem for Linux 2 will be widely available when Windows 10 version 2004 arrives. You'll have to install it manually for a "few months" until an update adds automatic installs and updates, but that's a small price to pay if you want Linux and Windows to coexist in peace and harmony. It'll be easier to set up, at least -- the kernel will now be delivered through Windows Update instead of forcing you to install an entire Windows image.
Embrace, Extend... Excite!
Windows blog post.
Previously: Windows 10 Will Soon Ship with a Full, Open Source, GPLed Linux Kernel
I've been using that for more that 20 years. Initially, one of the big draws was a working, free X server, which let me run Bourne shells not only on my desktop Windows, but to ssh into Suns and other *NIX boxes at various employment. I have even run the X displays from HP logic analyzers back to it.
What does the new kernel shim give me, other than I can run binaries (non-X, I suspect) from some distribution other than Cygwin?
What do I lose, compared to Cygwin?
What does the new kernel shim give me
Choice is good, right ? Isn't that what you geeks keep repeating day in and day out ?
I CHOOSE NOT to run Windows. (Except at work, where someone else is responsible for maintaining it.)
I did some fairly serious Cygwin work in the 2003 timeframe, and I was really impressed with how "close to the metal" it was, I'd get 98%+ performance out of a Cygwin application as compared to the same code running on a dedicated Linux shell, and I suspect that last 1.x% of performance was just the Windows background tasks popping in to phone the mothership, etc.
Still, the whole exercise felt... dirty, somehow. It didn't work at all when I migrated to 64bit in 2005, for that I had to go to Gentoo. Many years later 64 bit Cygwin became possible, but, remarkably, it was a bigger PITA even than installing Gentoo on the bare metal.
For the things that work in Cygwin, it's a perfectly valid option. When things get funky, I still prefer to be working in something like Ubuntu where you can lean on the experiences of many other people who have tried (and succeeded) to do the same thing you're trying to do.
Lately, when presented with a "must provide a Windows environment" mandate, our preferred architecture is Ubuntu with Windows running in a Virtual Box - this way we can be sure that we've got Windows under control, and if Windows packs it in - at least the Ubuntu system is still up and running. One application plays real-time audio - in the Ubuntu layer, because Windows never did reach 100% smooth playback capability and we hate explaining to our customers "yeah, it just does that sometimes, nothing we can do about it."
> I did some fairly serious Cygwin work in the 2003 timeframe, and I was really impressed with how "close to the metal" it was, I'd get 98%+ performance out of a Cygwin application as compared to the same code running on a dedicated Linux shell
I'm going to assert that your work load did not involve forking. We had an application stack that ran on Solaris and Linux. A decision came down that there should be a windows port, and to get this done with minimal fuss, some stuff would be properly ported, but some of the supporting scripts and less run-time critical bits would run under cygwin. Our experience was cygwin was at least 100 times slower when forking processes (and this had a lot to do with windows being slower than shit to fork processes-- this was microsoft's motivation for the shit ton of git patches they dropped that allow a bunch of operations to be performed without a zillion forks (forks on linux are as fast as threads on windows-- forks on windows are as slow as swapping floppy disks [almost]).
Windows doesn't support fork() or exec() natively. In order to emulate the behavior of fork(), cygwin has to jump through all sorts of hoops. It is a surprisingly complex operation. The context switches, mutexes, etc. produces a lot of overhead. Then if you call exec(), you end up causing another massive behavior emulation routine.
Yep, no forks in my little Cygwin toys.
I did a similar exercise just last month - ported a C++/Qt app to Python/PyQt - the first part of the port went swimmingly, 96% performance. Then I got to the "core" of what the C++ was doing: image pixel value manipulations, and Python tied itself in a knot: something like a 700x slowdown.
Then I got to the "core" of what the C++ was doing: image pixel value manipulations, and Python tied itself in a knot: something like a 700x slowdown.
Did you try having Pillow, NumPy, or another image or signal processing library do the inner loops?
Or was the nature of the problem such that it would require so many calls in and out of the library that the overhead of marshaling arguments between the Python and C environments would dominate runtime? Because I ran into a Python-to-C call rate bottleneck a few years ago when I was trying to get Pillow and PyPy to work together. I wanted to take the sum of each 16x16-pixel area in a difference image, and in Pillow, that requires making a cropped copy of each 16x16-pixel image. In PyPy, calls to extensions that use the traditional Python C extension API [blogspot.com] or ctypes are slow, whereas calls to extensions that use CFFI [readthedocs.io] (the Python port of LuaJIT FFI) are fast. Guess what Pillow wasn't using.
The real trick is learning time. I have a half-dozen rather straightforward problem statements to implement, and they're all obvious (to me) how to implement in Qt/C++, but constitute a pretty steep learning curve for me in Pillow/NumPy/OpenCV/what-have-you. Convert RGB to HSV or HSL, run statistics on the color channel values at each pixel location (things like mean, median, mode), make a transform which maps one set of images to have an identical color channel histograms as another set of images at each pixel location, copy-paste arbitrary ellipses from one image into another with alpha-blend margins, etc. etc. When the library has a function pre-built (like RGB to HSL transform) then, brilliant, say the magic words and it just happens.
My greatest frustration is when a library doesn't do something, because then the documentation tends to be silent on that point (hard to confirm a negative....), and when a library/language can't be made to do something in a practical manner, the documentation tends to be doubly silent on that point.
I used Cygwin for quite a bit. It's really nice, except when things don't build/break in some inexplicable way (a lot of it was because no one tested it under anything but Linux and OS X, and often not even that latter).WSL sidesteps that problem by just providing a Linux environment.
There are three things to compare here:
Cygwin is a Windows DLL that provides a more-or-less complete set of POSIX APIs, with some Linux-compatibility things. It also provides a package manager and a few other things, but it's basically a way of compiling a *NIX program so that it will run on Windows. Because Cygwin programs are native Windows programs, they can load Windows DLLs, including the ones required for talking to the windowing system, and interop natively.
WSL was a compat layer akin to the Linux system call layer found in *BSD, Solaris, and so on. It is a bit more complex on Windows because Windows doesn't have quite such a strictly defined kernel interface. For Win32 (and UWP) apps, there are a lot of DLLs that are injected automatically, where some kernel component can directly manipulate the kernel-owned data structures within the process. WSL uses the picoprocess abstraction defined for Drawbridge (the Windows-based library OS). A picoprocess is even more cut down than a *NIX process: it's basically like a Mach task, with nothing inside and just a control handle. Current Windows kernels only provide one picoprocess handler (though it wouldn't be very difficult to fix this). This is used by WSL, so when a picoprocess is created it is then initialised to look like a Linux process on start up and has a Linux-compatible system call handler installed. As a side effect of WSL, Windows got a full pseudoterminal layer and some other reusable components. WSL processes are not normal Windows processes, but the layering is such that they sit on top of the same low-level services as Windows processes (unlike the old POSIX and OS/2 subsystems in Windows NT, which had completely different implementations for things that you'd quite like to be shared, such as most IPC). In particular, the filesystem is shared. This is the main performance pain point for WSL: there's a filter driver that sits on top of the NTFS driver and provides NT or POSIX semantics to the filesystem depending on how it's being accessed.
WSL2 is much simpler. It's a Hyper-V VM set up to use the WSL terminal layer as its system console (I believe that this is done by forwarding the serial interface through the Windows pseudoconsole, but I'm not sure, it my just be using SSH), with a root filesystem that's attached as a block device containing a Linux-native filesystem (so no translation via filter drivers), and with shared folders implemented as 9P over VMBus (hopefully FUSE over VMBus soon, but I don't know if that's actually on the Hyper-V team's roadmap). This has a native Linux kernel, so has a full network stack, IPC layer, and so on. This can communicate with the host kernel only via network IPC. In theory, that's a big limitation, but in practice that's pretty much all WSL1 programs ever do.
On the X11 side, there are several builds of X.org for Windows. I believe VcXsrv [chocolatey.org] is the best choice. There are also a load of expensive alternatives, though I've not seen any evidence that they're better. These all (as with any X server) communicate with network-transparent IPC, and so work fine with Cygwin applications, WSL applications, and anything that runs in a Hyper-V VM. I run a FreeBSD VM on Hyper-V on my work desktop and can quite happily display graphical apps forwarded over SSH.
One of the nice things about WSL1 is that you can run an X11 terminal, but still then run cmd.exe or powershell.exe to get a Windows command prompt (the new Windows terminal is coming along well, but I still don't like it quite as much as some more mature *NIX ones. Sorry Dustin!). I use this to run Konsole, edit code in vim, and then run the .bat file that Visual Studio installs to set up command-line paths correctly and then build with cmake + Ninja, all from within a terminal.
WSL and WSL2 both natively support fork, so process creation costs are low (the hoops Cygwin jumps through to make fork work are slightly terrifying and incredibly slow: if you've ever run a configure script in Cygwin then you'll have seen this). Cygwin is a POSIX compatibility layer, WSL is a Linux compatibility layer, so software written by muppets who don't think about portability (also known as 'Linux programmers') is more likely to work. WSL2 gives faster disk I/O and better Linux compatibility (it *is* Linux), so things like Docker that abuse weird corner cases of very Linux-specific APIs work. The main thing you lose relative to Cygwin is the ability to mix Win32 / UWP and POSIX code in the same process (which, it turns out, few people actually want to do). If you just want an X server, you don't need any of them.
I haven't tried WSL2 yet. I'm hopeful that the tight integration work on the Windows side will make it possible to have a FreeBSD version (some folks on the Hyper-V team tell me it should be relatively easy, but they don't have engineer time to devote to it). I miss control-t when I have to use Linux...
Those are the kind of trade-offs I was concerned about. The "shim" I mentioned was reallt WSL, but the WSL 2 is a different beast.
Although it was initially the X server, shell (and associated tools), and ssh, I usually use Cygwin now to do things Windows makes either more trouble than necessary or impossible, or just to use the same tool there as on other *NIX. Raw device access, for example checksumming an entire CD, is still easy on a Cygwin/bash command line.
I thought FreeBSD already ran in VMs. Is it "just" a matter of integrating with the new "free" setup or something else?
I thought FreeBSD already ran in VMs. Is it "just" a matter of integrating with the new "free" setup or something else?
It runs very well in Hyper-V (locally and in Azure), but WSL2 has a bunch of integration hooks. I believe FreeBSD has a 9p-over-VirtIO driver under review, but it's not landed yet. Once this is in, 9p-over-VMBus is a relatively simple change (same messages, different link-layer protocol, but FreeBSD already has VMBus support). That's needed for the transparent file sharing thing (the Windows filesystem is exposed in WSL as /mnt/c, /mnt/d, and so on. In WSL2 these are 9p-over-VMBus filesystems). I think there are a few other glue things that may be needed. I haven't tried it with WSL2, but I believe Linux supports the Hyper-V virtual sound interface, which means that command-line WSL2 applications should be able to open /dev/dsp and make sounds, whereas FreeBSD is missing that driver (I don't care too much about this one, but it would be nice to have).
Last time I spoke to anyone on the Hyper-V team working on this, they were interested in supporting other operating systems, but Linux is the thing customers care about so it's the highest priority.