Stories
Slash Boxes
Comments

SoylentNews is people

SoylentNews is powered by your submissions, so send in your scoop. Only 16 submissions in the queue.
posted by LaminatorX on Friday January 16 2015, @04:08AM   Printer-friendly
from the mid-nite-theatre dept.

Quantity, gender, and case in internationalization. It's not easy. It's actually a great deal more difficult than it would appear at first glance. Sifting through SoylentNews' #rss-bot feed for 2015-01-15 I came upon a link to this gem of an article from 1999: A Localization Horror Story: It Could Happen To You.

The author provides a simple example of the challenges arising from trying to internationalize phrases detailing how many files and directories were found during a search. Just when it seemed like the problem was solved, another language presented yet another challenge that upended that erstwhile solution. The author identifies challenges unique to English [US and UK], Chinese, Italian, Russian, Arabic, and more.

Both entertaining in writing style and illuminating in content, this article gave me a whole new perspective on internationalization and the variations present in human languages. I'm so glad to have come upon it and hope others find it interesting, too!

This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 3, Funny) by Marand on Friday January 16 2015, @04:23AM

    by Marand (1081) on Friday January 16 2015, @04:23AM (#135282) Journal

    The author identifies challenges unique to English [US and UK], Chinese, Italian, Russian, Arabic, and more.

    Both entertaining in writing style and illuminating in content, this article gave me a while new perspective on internationalization and the variations present in human languages.

    A typo while talking about challenges in English; that's a hell of a time to make a mistake, isn't it? ;)

    • (Score: 3, Interesting) by Anonymous Coward on Friday January 16 2015, @04:42AM

      by Anonymous Coward on Friday January 16 2015, @04:42AM (#135284)

      Submitting articles is tough. There's a lot to get right - facts, interest, grammar and spelling, links (and make sure they're not paywalled, which is not always obvious), avoiding duplication with previous stories at SN and Slashdot, fairness and balance - and you aren't getting paid so you feel guilty about spending more than 20 minutes pulling it together. It's a LOT harder to do than writing even a long comment post in response to a summary.

      But a good submission usually takes more than 20 minutes. Even so, I've submitted lots of articles to Slashdot and SN and I've made a glaring mistake in almost every single one.

      • (Score: 3, Interesting) by Marand on Friday January 16 2015, @04:53AM

        by Marand (1081) on Friday January 16 2015, @04:53AM (#135287) Journal

        I agree, it's hard to make a good submission, and once you spend any length of time reading the submission your brain starts to actively fight you by filling in the blanks when proofreading, making it even harder to avoid mistakes. I spent far longer than I probably should have (and definitely more than twenty minutes) on the submissions I sent in before, double- and triple-checking everything, from writing to links, and I probably still screwed something up on them, too.

        That's why I try to point out mistakes with a joke. That way, they can be corrected by an editor without any hostility or hurt feelings. If somebody gets a laugh out of it that way, even better.

        • (Score: 1) by anubi on Friday January 16 2015, @05:31AM

          by anubi (2828) on Friday January 16 2015, @05:31AM (#135292) Journal

          I can certainly understand.

          I just spent a week making sure a hand-routed PCB was just perfect before sending it out.

          No sooner I submit the board, it comes back with 1/8 inch of trace missing I flat missed putting in. *snort*

          Absolute perfection seems like absolute zero, the closer you try to come, the more time and expense invested.

          Sometimes the perfect point isn't the optimal point.

          I have submitted a couple of stories here, however I am not much good at getting the wrapping on straight. The editors here have done a wonderful job of dressing up interesting tidbits I found while nosing around the web. If what I submitted went up as I had provided it, I would look like quite an ignoramus.

          Knowing how many times I have overlooked something, despite my very best intention, I will be damned to hold stuff like that over anybody else. Every time I point the finger, I have three of 'em pointing right back at me.

          --
          "Prove all things; hold fast that which is good." [KJV: I Thessalonians 5:21]
          • (Score: 1) by arulatas on Friday January 16 2015, @04:23PM

            by arulatas (3600) on Friday January 16 2015, @04:23PM (#135398)

            Wish I had mod points.

            --
            ----- 10 turns around
            • (Score: 2) by Joe Desertrat on Saturday January 17 2015, @02:50AM

              by Joe Desertrat (2454) on Saturday January 17 2015, @02:50AM (#135584)

              Wish I had mod points.

              I wish you had typed "I wish you had mood points".

              [I had a typo in that, but I caught it on preview.]
              [Dang, in that too!]

        • (Score: 0) by Anonymous Coward on Friday January 16 2015, @11:36AM

          by Anonymous Coward on Friday January 16 2015, @11:36AM (#135334)

          once you spend any length of time reading the submission your brain starts to actively fight you by filling in the blanks when proofreading, making it even harder to avoid mistakes.

          Read it out loud. Don't subvocalize, don't mumble, do it like you are reading it to another person. It engages a different part of your brain and it is like you are reading it for the first time.

      • (Score: 0) by Anonymous Coward on Friday January 16 2015, @04:48PM

        by Anonymous Coward on Friday January 16 2015, @04:48PM (#135408)

        What is SN? I'm on the lookout for good geek news sites and SN doesn't ring a bell. Thanks!

        • (Score: 0) by Anonymous Coward on Friday January 16 2015, @05:52PM

          by Anonymous Coward on Friday January 16 2015, @05:52PM (#135425)

          Never mind!! I just figured out that I'm already here at Soylent News *smacks forehead*! I'm new to SN so I managed to overlook it. However, I'm still on the hunt for great geek news sites, so your suggestions are welcome.

      • (Score: 2) by el_oscuro on Saturday January 17 2015, @05:35PM

        by el_oscuro (1711) on Saturday January 17 2015, @05:35PM (#135695)

        I once submitted an interesting story to slashdot about an Army project which was switched to Linux. After it was published, I noticed something about the story which appear on the home page of another tech site: It was over a year old. I didn't notice it, the editors didn't notice it, and only a few of 300 comments pointed it out.

        --
        SoylentNews is Bacon! [nueskes.com]
    • (Score: 2) by frojack on Friday January 16 2015, @05:26AM

      by frojack (1554) on Friday January 16 2015, @05:26AM (#135291) Journal

      A typo while talking about challenges in English; that's a hell of a time to make a mistake, isn't it? ;)

      No that's par for the course. Muphry's Law [wikipedia.org] I believe.

      --
      No, you are mistaken. I've always had this sig.
  • (Score: 2, Interesting) by Anonymous Coward on Friday January 16 2015, @04:30AM

    by Anonymous Coward on Friday January 16 2015, @04:30AM (#135283)

    This reminds me of the American auto industry, which basically didn't care about fit and finish (doors closing without rattling, inside handles staying attached for more than six months, etc) for several decades after the Second World War because they were breaking sales records almost every year. Oh, the Germans sold better cars, but those were expensive here, and everyone knew that getting them serviced in America was *really* expensive.

    Then the series of OPEC oil shocks hit in the '70s, and consumers became more interested in smaller cars, which the Japanese specialized in. After they started buying them, they noticed that the Japanese had spent a long time working on fit and finish - with help from the American statistician W. Edwards Deming and his peers, who were ignored in America but were very highly respected in Japan.

    I think we're at the cusp of an "OPEC" event with regards to localized UI and error messages, and correct handling of substitution parameters (even in English). Right now, customers might not care that much because they can still get their work done. But at some point, suddenly it's going to matter, and those who can't provide polished product will be left in the dust.

    • (Score: 2) by Common Joe on Friday January 16 2015, @06:16AM

      by Common Joe (33) <reversethis-{moc ... 1010.eoj.nommoc}> on Friday January 16 2015, @06:16AM (#135299) Journal

      I don't know. Living here in Germany, the majority of technical stuff is done in English, but content is for locals. I.e., the programmers can read and write English (even if they can't speak it) so they understand how to code, but the website content is in German.

      I've never done localization, but it seems very difficult for both the programmers and the translators. Difficult = $$$. (Or, difficult = €€€ in my case.) I think only the larger businesses will invest the time and money into this kind of stuff.

      In theory, localization should be straight forward, but how does one thoroughly test a complicated program (with tough to find error messages) in multiple languages to make sure it all came out ok? My local trains can't even get the English right over the loud speakers asking us to change trains. ("This trains ends at the next stop. All change please." My wife and I joke that we need to keep a spare set of clothes in our backpacks so we can change.) I can say that the quality of translations that my wife and I have seen in every day life have been extremely poor. Localization of programs is an order of magnitude harder.

      • (Score: 2) by tangomargarine on Friday January 16 2015, @08:18AM

        by tangomargarine (667) on Friday January 16 2015, @08:18AM (#135314)

        If that's the worst example I could find, I'd count myself extremely lucky.

        --
        "Is that really true?" "I just spent the last hour telling you to think for yourself! Didn't you hear anything I said?"
        • (Score: 2) by Common Joe on Friday January 16 2015, @10:18AM

          by Common Joe (33) <reversethis-{moc ... 1010.eoj.nommoc}> on Friday January 16 2015, @10:18AM (#135326) Journal

          [Chuckling] No, no... the train example is far from the worst. It's just something I've heard very often for over a year. It's so simple that it shouldn't be a problem, but it is still a problem even after they updated some of the voices. The idea was that if simple things can't be done right, the more complicated things will be that much more likely to be with error or cost a lot more. That was all I was trying to say.

      • (Score: 3, Interesting) by sudo rm -rf on Friday January 16 2015, @08:35AM

        by sudo rm -rf (2357) on Friday January 16 2015, @08:35AM (#135316) Journal

        As a sidenote, i think some projects / manufacturers take i18n too far, e.g. Excel and Libre Office Calc translate the function names to your locale settings by default[1]. My Linux machine with culture settings de_AT and my Windows machine with de_DE both translate

        =SUM to =SUMME

        AND to UND

        and so on. Because most online resources/tutorials/examples are in English, it can give you a hard time to search the internet for help. Luckily there are dictionaries like this [excelnova.org].

        [1] I know I can change the setting of how to display the function names, but it doesn't help when having a german speaking customer on the line discussing an xlsx.

      • (Score: 0) by Anonymous Coward on Friday January 16 2015, @12:28PM

        by Anonymous Coward on Friday January 16 2015, @12:28PM (#135343)

        Difficult = $$$. (Or, difficult = €€€ in my case.)

        Difficult = ¥€$

  • (Score: 3, Insightful) by Marand on Friday January 16 2015, @04:45AM

    by Marand (1081) on Friday January 16 2015, @04:45AM (#135285) Journal

    Maybe it's because of missing details, but it seems like the author could have redefined the problem and avoided the complex situation and solution of the story. He even touched on a way to do it at one point, when he mentioned that it's much easier to translate static phrases with unchanging elements, and that adding changing elements complicates the translation greatly. Instead of trying to make dynamic sentences, it could have been possible to reword the text in a way that separated the numbers from the translated strings. Example:

    Directories scanned: %g
    Matching directories: %g
    Matching files: %g

    (There's probably a better way to word it, but for the sake of example this is sufficient.)

    Sometimes you get so hung up on solving the problem in front of you, instead of redefining the problem to be more manageable, that you make more work for yourself.

    • (Score: 2) by CirclesInSand on Friday January 16 2015, @05:33AM

      by CirclesInSand (2899) on Friday January 16 2015, @05:33AM (#135293)

      Especially if you are using it for console output, it writing a quick regex that pipes in the stdout of the other program much easier this way. For a GUI, on the other hand, and for a program that is for sale, you might have a boss that expects prose.

      • (Score: 0) by Anonymous Coward on Friday January 16 2015, @10:29AM

        by Anonymous Coward on Friday January 16 2015, @10:29AM (#135328)

        Of course if you want to use it for regex matching, you don't want to apply any locale at all (or equivalently, apply the "C" locale), because even with that form, your regex /Maching (directories|files):/ will match neither Passende Verzeichnisse: nor Passende Dateien:.

        But you could make this "itemized form" the base form, and the "prose form" ("I scanned %g directories." + singular/plural form) the English locale. Which would have the advantage that you'd get less scripts which forget "LANG=C" before calling tools whose output shall be parsed, because people tend to forget that not everyone has a non-English locale, so your code already breaking with English locale is an advantage.

        In other words: Make the "C" locale equivalent to "machine readable English" and the "en" locale equivalent to "pretty English prose for humans".

        • (Score: 0) by Anonymous Coward on Friday January 16 2015, @10:33AM

          by Anonymous Coward on Friday January 16 2015, @10:33AM (#135329)

          because people tend to forget that not everyone has a non-English locale,

          This should, of course, have read: "… that not everyone has an English locale".

    • (Score: 2) by frojack on Friday January 16 2015, @05:33AM

      by frojack (1554) on Friday January 16 2015, @05:33AM (#135294) Journal

      Exactly. And most languages have such constructs.

      You can forget about the plural/singular forms when you use this construct, because it really doesn't matter. Unless you are trying to write conversational text, any time wasted on this sort of problem really isn't worth it.

      --
      No, you are mistaken. I've always had this sig.
    • (Score: 2) by MrNemesis on Friday January 16 2015, @05:08PM

      by MrNemesis (1582) on Friday January 16 2015, @05:08PM (#135411)

      I was going to chip in with the same thing - I do the same sort of stuff myself in shell scripts (because I'm a terrible coder and formulating sentences from variables is frequently hard even in a single language) but it's a perfectly valid approach.

      However, I get the impression that this was tended as an overly simplistic example just to demonstrate some of the hideous hidden complexity inherent in translation; "if translating two very simple statements and some numbers is hard, imagine trying to do this with a whole X".

      Fascinating article by the way and even though I don't code much perl and don't know what gettext and maketext are I actually read the whole thing. As a native and solely english speaker who didn't have any languages other than french to study at school it's a brilliant glimpse of another world.

      --
      "To paraphrase Nietzsche, I have looked into the abyss and been sick in it."
  • (Score: 2) by c0lo on Friday January 16 2015, @05:12AM

    by c0lo (156) Subscriber Badge on Friday January 16 2015, @05:12AM (#135288) Journal

    gettext has ways which handle the pluralisation for quite a while.
    Next time, RTFM [gnu.org] - as a translator, you may also want to read the tramslating plurals [gnu.org] section.

    For your hair-raising pleasurable experience (assuming you still have hair to raise):

    In Polish we use e.g. plik (file) this way:
    1 plik
    2,3,4 pliki
    5-21 pliko'w
    22-24 pliki
    25-31 pliko'w
    and so on (o’ means 8859-2 oacute which should be rather okreska, similar to aogonek).

    --
    https://www.youtube.com/@ProfSteveKeen https://soylentnews.org/~MichaelDavidCrawford
    • (Score: 1, Informative) by Anonymous Coward on Friday January 16 2015, @06:09AM

      by Anonymous Coward on Friday January 16 2015, @06:09AM (#135298)

      I haven't used gettext() before, but I have used the Qt toolkit, and it too has a fairly easy-to-use translation system (QtLinguist), which is also plural-aware [digia.com].

      Some interesting facts about plurals from the Qt documentation:

      • Latvian has a specific grammatical number, the nullar, for the "n = 0" case.
      • Dhivehi, Inuktitut, Irish, Maori, and a few other languages have a dual form for the "n = 2" case.
      • Czech, Slovak, Lithuanian, and Macedonian have a dual, but they use it according to more complex rules.
      • Slovenian has a trial in addition to the singular, dual, and plural forms.
      • Romanian handles the "n >= 20" case differently from the "n
      • Arabic has six different forms, depending on the value of n.
      • Chinese, Japanese, Korean, and many other languages don't distinguish between the singular and the plural.
      • (Score: 2) by Common Joe on Friday January 16 2015, @06:26AM

        by Common Joe (33) <reversethis-{moc ... 1010.eoj.nommoc}> on Friday January 16 2015, @06:26AM (#135303) Journal

        This is one of the best intros to localization I've ever read. It's actually easier to understand than just about every other localization doc I've seen to date. (From a beginner's point of view.) I wouldn't recommend the GNU articles suggested. That is difficult reading.

    • (Score: 2, Informative) by martyb on Friday January 16 2015, @06:44AM

      by martyb (76) on Friday January 16 2015, @06:44AM (#135304) Journal

      gettext has ways which handle the pluralisation for quite a while.
      Next time, RTFM [gnu.org] - as a translator, you may also want to read the tramslating plurals [gnu.org] section.

      Yes, traMslating is very challenging! =)

      From the linked story:

      The following article by Sean M. Burke and Jordan Lachler first appeared in The Perl Journal #13 and is copyright 1999 The Perl Journal.

      And, to my reading, provided the background and justification for their writing the Locale::Maketext [cpan.org] CPAN module. Nice to see that gettext has been updated since the linked story was published back in 1999.

      For your hair-raising pleasurable experience (assuming you still have hair to raise):

      In Polish we use e.g. plik (file) this way:
      1 plik
      2,3,4 pliki
      5-21 pliko'w
      22-24 pliki
      25-31 pliko'w
      and so on (o’ means 8859-2 oacute which should be rather okreska, similar to aogonek).

      My hair supply is rather limited and not up to the challenge (pun intended!) Your example agrees well with this part of the linked story (emphasis added):

      Then your Russian translator calls on the phone, to personally tell you the bad news about how really unpleasant your life is about to become:

      Russian, like German or Latin, is an inflectional language; that is, nouns and adjectives have to take endings that depend on their case (i.e., nominative, accusative, genitive, etc...) -- which is roughly a matter of what role they have in syntax of the sentence -- as well as on the grammatical gender (i.e., masculine, feminine, neuter) and number (i.e., singular or plural) of the noun, as well as on the declension class of the noun. But unlike with most other inflected languages, putting a number-phrase (like "ten" or "forty-three", or their Arabic numeral equivalents) in front of noun in Russian can change the case and number that noun is, and therefore the endings you have to put on it.

      He elaborates: In "I scanned %g directories", you'd expect "directories" to be in the accusative case (since it is the direct object in the sentence) and the plural number, except where $directory_count is 1, then you'd expect the singular, of course. Just like Latin or German. But! Where $directory_count % 10 is 1 ("%" for modulo, remember), assuming $directory count is an integer, and except where $directory_count % 100 is 11, "directories" is forced to become grammatically singular, which means it gets the ending for the accusative singular... You begin to visualize the code it'd take to test for the problem so far, and still work for Chinese and Arabic and Italian, and how many gettext items that'd take, but he keeps going... But where $directory_count % 10 is 2, 3, or 4 (except where $directory_count % 100 is 12, 13, or 14), the word for "directories" is forced to be genitive singular -- which means another ending... The room begins to spin around you, slowly at first... But with all other integer values, since "directory" is an inanimate noun, when preceded by a number and in the nominative or accusative cases (as it is here, just your luck!), it does stay plural, but it is forced into the genitive case -- yet another ending... And you never hear him get to the part about how you're going to run into similar (but maybe subtly different) problems with other Slavic languages like Polish, because the floor comes up to meet you, and you fade into unconsciousness.

      So, yes, that makes the English "directory" vs "directories" distinction look like child's play, by comparison; thanks for the illuminating example! (I hesitate to submit this as it is now 1:30 AM and I'm tempting Muphry's Law [wikipedia.org]; oh well!)

      • (Score: 2) by c0lo on Friday January 16 2015, @10:49AM

        by c0lo (156) Subscriber Badge on Friday January 16 2015, @10:49AM (#135332) Journal

        reading and writing can be challenging

        Indeed (especially after a sleepless night; I should refrain from commenting in such cases).

        --
        https://www.youtube.com/@ProfSteveKeen https://soylentnews.org/~MichaelDavidCrawford
    • (Score: 2) by dcollins on Friday January 16 2015, @07:15AM

      by dcollins (1168) on Friday January 16 2015, @07:15AM (#135307) Homepage

      That part of the article seemed much less shocking than was intended, to me, because it seems broadly analogous to the English issue of ending an ordinal number (1st, 2nd, etc.):

      x1 st
      x2 nd
      x3 rd
      x4-0 th

      except
      x11-13 th

    • (Score: 0) by Anonymous Coward on Friday January 16 2015, @11:42AM

      by Anonymous Coward on Friday January 16 2015, @11:42AM (#135335)

      o’ means 8859-2 oacute

      You mean ó?

    • (Score: 1) by fritsd on Friday January 16 2015, @01:52PM

      by fritsd (4586) on Friday January 16 2015, @01:52PM (#135364) Journal

      ISO latin-2 is sóó old-fashioned! ;-)

      (yes that was a jǫke)

  • (Score: 1) by cliffjumper222 on Friday January 16 2015, @07:06PM

    by cliffjumper222 (2628) on Friday January 16 2015, @07:06PM (#135469)

    I write plugins for Minecraft and volunteers have translated the locale.yml (and other) files into Chinese, Korean, Polish, French, German, etc. However, normally the admin of the server translates the file, or comes up with their own server-specific text. locale.yml is a lexicon file with all the strings of the whole game written out with variables marked by tokens, e.g., [playername] or [cost], etc. Those tokens are replaced by the values when the string is shown. Color codes are also available. The advantage of this is that the translator can put those values anywhere they like in their translation. Sometimes in a sentence, sometimes not. This works for simple, non-professional situations.

    What this article taught me was that in some languages (Russian sounds daunting!), you need to display different grammar depending on the data that is to be shown. In essence, you need a little scripting capability. This is actually not that difficult to do. I can add an option to add Javascript (or similar), into the locale.yml file for each element. This would be executed at run time if it exists. I'm quite excited about this because it could be really useful and extensible. Admins would be able to get really creative with the responses shown to commands based on the output rather than using the stock output!

    As with all the translations, if anything fails or is absent the default English translation is shown. This speaks to the "inheritance" part he was talking about. It also alerts the admin (who usually makes the locale.yml file themselves) to where they need to translate, which is not always done in one go.

    --
    He who dares wins, Rodney
  • (Score: 2) by darkfeline on Friday January 16 2015, @11:18PM

    by darkfeline (1030) on Friday January 16 2015, @11:18PM (#135550) Homepage

    You do it by writing the semantic tree of what you want to say instead of writing it in one language and then translating it to another language.

    For example, instead of

            gettext("Your query matched 10 files in 2 directories.")

    you do (in Lispy grammar)

            (gettext '(matched (your query) (10 files) (in 2 directories))

    The idea is you design a semantic grammar tree that can then be automatically compiled into whatever language you want. The semantics don't change, only how each language expresses each semantic construct. Note that the preferred semantic structure may vary between languages (e.g., language A says (eat me apple), language B (me (eat apple)), but these are logical transformations that can be done easily and reversibly.

    Here's a post that talks about a similar concept: http://felesatra.moe/posts/2015/01/01/the-babel-meta-language/ [felesatra.moe]

    --
    Join the SDF Public Access UNIX System today!