Okay, maybe not everything you know about latency is wrong. But now that I have your attention, we can talk about why the tools and methodologies you use to measure and reason about latency are likely horribly flawed. In fact, they're not just flawed, they're probably lying to your face.
When I went to Strange Loop in September, I attended a workshop called "Understanding Latency and Application Responsiveness" by Gil Tene. Gil is the CTO of Azul Systems, which is most renowned for its C4 pauseless garbage collector and associated Zing Java runtime. While the workshop was four and a half hours long, Gil also gave a 40-minute talk called "How NOT to Measure Latency" which was basically an abbreviated, less interactive version of the workshop. If you ever get the opportunity to see Gil speak or attend his workshop, I recommend you do. At the very least, do yourself a favor and watch one of his recorded talks or find his slide decks online.
The remainder of this [linked] post is primarily a summarization of that talk. You may not get anything out of it that you wouldn't get out of the talk, but I think it can be helpful to absorb some of these ideas in written form. Plus, for my own benefit, writing about them helps solidify it in my head.
(Score: 2) by kaganar on Wednesday December 16 2015, @11:01PM
No, I didn't watch TFV (which seems concerned with layer-7 "latency", not layer 3 or below), but here's a common latency issue that people don't seem to think about but I encounter in different situations all the time at various locations:
A path through a network seems slow -- everything you do is slow, there's even some time outs for services you're using, but you see this:
C:\Windows\Sucks>ping -t xxx.xxx.xxx.xxx
Pinging someserver.blah [xxx.xxx.xxx.xxx] with 32 bytes of data:
Reply from xxx.xxx.xxx.xxx: bytes=32 time=16ms TTL=44
Reply from xxx.xxx.xxx.xxx: bytes=32 time=17ms TTL=44
Reply from xxx.xxx.xxx.xxx: bytes=32 time=17ms TTL=44
...
And then for shits you try this out (same thing, but a packet size with a more representative payload length):
C:\Windows\Sucks>ping -t -l 2048 xxx.xxx.xxx.xxx
Pinging someserver.blah [xxx.xxx.xxx.xxx] with 2048 bytes of data:
Reply from xxx.xxx.xxx.xxx: bytes=2048 time=1924ms TTL=44
Request timed out
Reply from xxx.xxx.xxx.xxx: bytes=2048 time=2319ms TTL=44
...
Just take one guess which one your ISP's technical support rep is going to do. I'm not an expert, but this seems to make some intuitive sense: if you're going to have bit-flipping problems because of line noise, what's more likely to get through unscathed without needing retransmission: long messages or short one? Once I explain that, I usually get a bit more cooperation other than "Huh, must be your computer. Anything else I can help you with?"