Stories
Slash Boxes
Comments

SoylentNews is people

posted by mrpg on Wednesday July 19 2023, @01:11AM   Printer-friendly
from the Sir-Robot-to-the-likes-of-you dept.

https://arstechnica.com/information-technology/2023/07/why-ai-detectors-think-the-us-constitution-was-written-by-ai/

If you feed America's most important legal document—the US Constitution—into a tool designed to detect text written by AI models like ChatGPT, it will tell you that the document was almost certainly written by AI. But unless James Madison was a time traveler, that can't be the case. Why do AI writing detection tools give false positives? We spoke to several experts—and the creator of AI writing detector GPTZero—to find out.

[...] In machine learning, perplexity is a measurement of how much a piece of text deviates from what an AI model has learned during its training. As Dr. Margaret Mitchell of AI company Hugging Face told Ars, "Perplexity is a function of 'how surprising is this language based on what I've seen?'"

So the thinking behind measuring perplexity is that when they're writing text, AI models like ChatGPT will naturally reach for what they know best, which comes from their training data. The closer the output is to the training data, the lower the perplexity rating. Humans are much more chaotic writers—or at least that's the theory—but humans can write with low perplexity, too, especially when imitating a formal style used in law or certain types of academic writing. Also, many of the phrases we use are surprisingly common.

Let's say we're guessing the next word in the phrase "I'd like a cup of _____." Most people would fill in the blank with "water," "coffee," or "tea." A language model trained on a lot of English text would do the same because those phrases occur frequently in English writing. The perplexity of any of those three results would be quite low because the prediction is fairly certain.


Original Submission

 
This discussion was created by mrpg (5708) for logged-in users only, but now has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 2, Funny) by Anonymous Coward on Wednesday July 19 2023, @06:17AM (2 children)

    by Anonymous Coward on Wednesday July 19 2023, @06:17AM (#1316796)

    ...and you must be postmenopausal.

    Starting Score:    0  points
    Moderation   +2  
       Offtopic=1, Insightful=1, Funny=2, Total=4
    Extra 'Funny' Modifier   0  

    Total Score:   2  
  • (Score: 1) by khallow on Wednesday July 19 2023, @12:34PM

    by khallow (3766) Subscriber Badge on Wednesday July 19 2023, @12:34PM (#1316840) Journal
    im angry angry about all these rules we should be following but arnet whycant you guys just not suck
  • (Score: 2, Touché) by Anonymous Coward on Wednesday July 19 2023, @04:03PM

    by Anonymous Coward on Wednesday July 19 2023, @04:03PM (#1316874)

    Parent was not off-topic. It was indicating that GP poster had lost his period.

    If you don't get the joke, don't think you are qualified to moderate it.