Back in May, writer Jun Wu told in her blog how Perl excels at text manipulation. She often uses it to tidy data sets, a necessity as data is often collected with variations and cleaning it up before use is a necessity. She goes through many one-liners which help make that easy.
Having old reliables is my key to success. Ever since I learned Perl during the dot com bubble, I knew that I was forever beholden to its powers to transform.
You heard me. Freedom is the word here with Perl.
When I'm coding freely at home on my fun data science project, I rely on it to clean up my data.
In the real world, data is often collected with loads of variations. Unless you are using someone's "clean" dataset, you better learn to clean that data real fast.
This discussion has been archived.
No new comments can be posted.
Perl Is Still The Goddess For Text Manipulation
|
Log In/Create an Account
| Top
| 60 comments
| Search Discussion
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
(1)
(1)
(Score: 4, Insightful) by The Mighty Buzzard on Thursday September 19 2019, @03:13PM (29 children)
Said it before and I'll say it again, there is absolutely nothing as good as perl for random text wrangling unless you're dealing with data sets so enormous that you absolutely have to have the fastest running code possible. For most applications you will never make up the extra time spent coding it in any other language.
My rights don't end where your fear begins.
(Score: 3, Interesting) by JoeMerchant on Thursday September 19 2019, @03:27PM (16 children)
I live mostly inside my C++ bubble, further subset into the Qt API. In here, QString is pretty damn good at wrangling string issues, and when it's not enough you can always bail out to RegEx (sounds like: retch, for a reason I think.)
So much is just whatever you are familiar with, Boost, stdlib, whatever... if you know how to use them they've got most of the tools you need pre-coded, and if you find yourself doing the same thing over and over that takes more than 2 lines to accomplish, that sounds like time for a personal library extension...
🌻🌻 [google.com]
(Score: 3, Insightful) by The Mighty Buzzard on Thursday September 19 2019, @03:37PM (5 children)
Kind of the point. Using perl you can do a whole hell of a lot in one line. I've used a lot of languages over the years and still pick new ones up for fun every year or two and I've never found anything even close to as versatile and efficient with text as perl. Python usually takes a minimum of three times as many lines to accomplish what perl can do legibly in one, five to ten is more common.
My rights don't end where your fear begins.
(Score: 2) by JoeMerchant on Thursday September 19 2019, @03:49PM (4 children)
Yeah, one-liners are good - I really appreciate having an easy GUI that I can just throw open a scrolling text box in, append text to it all day long with single line commands, HTML format that text if I feel like it, maybe toss on a few checkboxes to toggle boolean control variables (like command line switches, but changeable at runtime...)
Again, it's all in what you're used to. Today, I'm appreciating the verbose log files that make it relatively easy to spot what went weird when the testers come up with their 1/10,000 behaviors.
🌻🌻 [google.com]
(Score: 3, Interesting) by The Mighty Buzzard on Thursday September 19 2019, @03:57PM (3 children)
Honestly, I mostly use grep, awk, and sed for most one-liner type stuff. Perl is overkill for the extremely simple stuff. I mostly use it when I need something at least slightly more complex. The ability to get way more work done per readable line is just as useful in a script as on a command line though.
My rights don't end where your fear begins.
(Score: 0) by Anonymous Coward on Friday September 20 2019, @05:53AM (2 children)
I find grep and sed incredibly useful and intuitive, while never quite grokking awk. Don't know why.
(Score: 0) by Anonymous Coward on Friday September 20 2019, @08:57AM (1 child)
Because awk is for parsing of stuff, especially column oriented documents. If you want 5th column of something, for example. But if you don't deal with column data, then you probably would never need awk.
(Score: 0) by Anonymous Coward on Friday September 20 2019, @11:00AM
Awk is really good at stuff you would normally have to pipe sed and grep for, you can use one simple awk statement. It also has some formatting capabilities so I like to use it when writing shell functions.
(Score: 0) by Anonymous Coward on Thursday September 19 2019, @03:48PM (4 children)
Your bubble will burst when touched by Unicode multilanguage data mixed with funny math/geometry/engineering symbols and true emoji.
(Score: 2) by The Mighty Buzzard on Thursday September 19 2019, @03:52PM (2 children)
Perl's kind of shitty at that too unless you know the few simple tricks to make it not shitty at it.
My rights don't end where your fear begins.
(Score: 0) by Anonymous Coward on Friday September 20 2019, @03:20AM (1 child)
Really? Perl was the first language to manage that properly for me. Yeah, sometimes you need to be explicit in unusual ways, but at least you _can_ without jumping through hoops. I think this gets thrown in the "tricks" category just because it so rarely comes up that you're probably going to need to look it up when it does, cuz otherwise you just pick an encoding and forget about it.
(Score: 2) by The Mighty Buzzard on Friday September 20 2019, @03:26AM
Yup. There are even changes involving unicode jumping hoops in perl since I bitchslapped SN into more or less supporting it.
My rights don't end where your fear begins.
(Score: 2) by JoeMerchant on Thursday September 19 2019, @08:55PM
Funny thing, QString handles Unicode, UTF-8, etc. conversions pretty much seamlessly, as do the other modern string classes. Thank God for that, because the last thing I want to fool with is conversion between Unicode and UTF-8.
🌻🌻 [google.com]
(Score: 3, Insightful) by legont on Thursday September 19 2019, @07:09PM (4 children)
Yeah, but once one finishes building it, one discovers that he rebuild Perl.
"Wealth is the relentless enemy of understanding" - John Kenneth Galbraith.
(Score: 2) by JoeMerchant on Thursday September 19 2019, @08:59PM (3 children)
If you're that heavy into what Perl does, by all means, use it. For me, it's a bigger PITA to "shell out" to get access to Perl than it is to recode in C++, on the rare occasions it is necessary. If I really loved Perl so much but still needed to be in C++, I'd make a dedicated wrapper for Perl and get full access that way.
BTW: anyone considering PyQt for anything bigger than a toy project, my recommendation is: don't. But, then, that's pretty much my recommendation for Python all over - sure, there are some pretty impressive "large" things out there mostly based in Python - like trac, which I have happily used for over 10 years now, but... for the most part, unless the coding team is super disciplined, Python degenerates into a ball of snakes much faster than any of the C derivatives I have ever worked with.
🌻🌻 [google.com]
(Score: 2) by legont on Friday September 20 2019, @01:01AM (2 children)
My main choice for a long time was C (plain, without ++). For quick and dirty things I'd use AWK and I am talking here not about one liners, but full blown software of a few hundreds or even thousands lines of code. There was even an AWK compiler that one guy wrote and was selling for $99 that did a very good job.
At some point I discovered Perl, gave it a try, and it replaced AWK, even compiled version of it, for me. Time passing, I realized that I pretty much stopped using C except in rare special occasions and that Perl would cover everything for me.
Management would be forcing at different times Java, dotnet, Python and so on, but so far at the end Perl could not be replaced. There is another attempt going right now and this time they may succeed, but let's see...
I appreciate your comment about Python, but if you were asked to replace a huge Perl project with something modern that fresh college kids would like, what would you recommend?
"Wealth is the relentless enemy of understanding" - John Kenneth Galbraith.
(Score: 4, Insightful) by JoeMerchant on Friday September 20 2019, @01:11PM (1 child)
Perl.
I worked for almost a decade converting fresh college kids' code (Matlab, Python, and strangely: a fair bit of Fortran) to C++ so that their broken toys could be sold to real customers.
🌻🌻 [google.com]
(Score: 2) by legont on Friday September 20 2019, @05:54PM
Yes, my thoughts exactly.
"Wealth is the relentless enemy of understanding" - John Kenneth Galbraith.
(Score: 2) by Thexalon on Thursday September 19 2019, @04:18PM (8 children)
Perl's great for this ... unless there's a simple known sed/awk approach to the problem, because those are both a bit faster and simpler.
The only thing that stops a bad guy with a compiler is a good guy with a compiler.
(Score: 2) by The Mighty Buzzard on Thursday September 19 2019, @05:00PM (7 children)
Simpler, yes, faster, maybe. After three to five pipes it starts making more sense to break out the perl.
My rights don't end where your fear begins.
(Score: 2) by Thexalon on Thursday September 19 2019, @05:06PM
Yeah, I'm talking about the near-1-liners that are sometimes available. I agree that if you start getting into complexity, then yes, bust out the Perl.
The only thing that stops a bad guy with a compiler is a good guy with a compiler.
(Score: 0) by Anonymous Coward on Friday September 20 2019, @12:22AM (5 children)
Though there's the extra advantage from shell pipelines: free smp support. Each process is (obviously) a separate process so each stage of your pipeline can get its own core without writing multi-threaded code.
(Score: 2) by The Mighty Buzzard on Friday September 20 2019, @01:32AM (4 children)
Not a huge help when they're run sequentially, as piped commands are.
My rights don't end where your fear begins.
(Score: 1, Interesting) by Anonymous Coward on Friday September 20 2019, @04:06AM (3 children)
In MS-DOS they ran sequentially, in Unix/Linux the programs run in parallel and consume each other's output as soon as it becomes available. Run 'find | cat' on a large directory tree. The output in the terminal starts appearing immedeately, which can only happen if cat runs parallel to find. Or run 'find | rev' if you want to see the second command make a visible change.
(Score: 2) by The Mighty Buzzard on Friday September 20 2019, @07:24PM (2 children)
In Linux/*BSD they effectively run sequentially as well in most cases as well. There's a pretty small subset of commands that output useful information as they run.
My rights don't end where your fear begins.
(Score: 0) by Anonymous Coward on Friday September 20 2019, @08:41PM (1 child)
Don't they run in reverse sequence for all shell redirection? If you run 'echo "test" | less' , less starts first, opening the fd that echo then prints out to.
(Score: 2) by The Mighty Buzzard on Friday September 20 2019, @08:57PM
For a value of "run" meaning "start up and then do nothing at all until the other thing we're about to start sends you some data" you mean?
My rights don't end where your fear begins.
(Score: 0) by Anonymous Coward on Thursday September 19 2019, @09:30PM (2 children)
We use Perl for data sets in the TB range at work and it's fine.
(Score: 2) by The Mighty Buzzard on Friday September 20 2019, @01:35AM (1 child)
Depending on the use case, it can be, sure. If you're trying to serve up thousands of pages a second off that dataset or some such, I'd go with something a little more close to the metal so you can use an optimized function that does only what you need and does it in as few cycles as possible.
My rights don't end where your fear begins.
(Score: 0) by Anonymous Coward on Friday September 20 2019, @08:03PM
We're using Vertica (a column-oriented SQL database) for the high performance queries. But for loading the data into Vertica, we use Perl.
(Score: 1, Insightful) by Anonymous Coward on Thursday September 19 2019, @03:14PM (10 children)
R would be better than this. Probably python too.
(Score: 4, Insightful) by Anonymous Coward on Thursday September 19 2019, @03:32PM (8 children)
Python is better than being fucked up the ass, but not much else. It's a terrible language designed by people who don't understand programming very well. Syntactic white space, uninformative error messages and dynamic typing being a few transgressions.
(Score: 3, Insightful) by Anonymous Coward on Thursday September 19 2019, @07:17PM (7 children)
There. FTFY.
(Score: 0) by Anonymous Coward on Thursday September 19 2019, @09:37PM (6 children)
First an eyeroll, but then...yeah. Actually kind of funny! Guess trying to avoid this kind of accidental entendre is an alternative reason to adhere to the "code of conduct"...
(Score: 0) by Anonymous Coward on Thursday September 19 2019, @09:46PM (5 children)
And to which "code of conduct" might you be referring, friend?
(Score: 2) by The Mighty Buzzard on Friday September 20 2019, @01:38AM (4 children)
Certainly not ours [github.com]...
My rights don't end where your fear begins.
(Score: 2) by edIII on Friday September 20 2019, @03:03AM (2 children)
Why is this bad? I convert tabs to spaces because of inconsistent treatment of tabs amongst programs while spaces plus a monospaced font are.
Technically, lunchtime is at any moment. It's just a wave function.
(Score: 4, Funny) by The Mighty Buzzard on Friday September 20 2019, @03:29AM
That you ask has outed you as a witch. Gentlemen, bring the firewood.
My rights don't end where your fear begins.
(Score: 1, Informative) by Anonymous Coward on Friday September 20 2019, @09:19AM
Tabs are for indentation. Space is for *formatting*. So, if you have 1 indent, you have 1 tab. If you want more because it's line continuation or something, then you add spaces. That's why it's nice to have editors with visible whitespace. QtCreator is a nice editor for that. VS Code is nice for that too.
Why? If you think, you wouldn't bother to ask. But since you didn't ... this allows one to adjust *preferred* whitespace to every developer simply by changing the tab width setting in any sensible editor. If some dev wants 5 spaces for tab, that's fine. If they want 2, that's fine. IF they like 8 or whatever, it all works. And it works without changing any line of code and fucking up history.
(Score: 2) by NotSanguine on Friday September 20 2019, @07:06AM
There will be no ed/vi/nano in my house. Emacs is the only OS I use. Now if I only had a text editor...
No, no, you're not thinking; you're just being logical. --Niels Bohr
(Score: 2) by gringer on Friday September 20 2019, @03:04AM
R is terrible for text processing. Extracting matches from regular expressions involves compiling the expressions and parsing a list. Some of the newest tidyverse packages are greatly improving the syntax, but the speed for text manipulation still remains an issue.
Ask me about Sequencing DNA in front of Linus Torvalds [youtube.com]
(Score: 3, Interesting) by Freeman on Thursday September 19 2019, @03:18PM (3 children)
Open Refine can do quite a bit with regard to text manipulation as well. I would say Open Refine is also much easier to see what you've done and what you're doing. There's also cool features like, being able to save a set of procedures, so you can replicate the process on another data set.
http://openrefine.org/ [openrefine.org]
Joshua 1:9 "Be strong and of a good courage; be not afraid, neither be thou dismayed: for the Lord thy God is with thee"
(Score: 2, Interesting) by Anonymous Coward on Thursday September 19 2019, @04:45PM (2 children)
When you do not know what you are doing, you should not be touching any datasets.
When you do know, you have plenty of time-tested tools that you can use, without having to learn a Google-invented bastard language not fit for any other purpose.
(Score: 2) by Freeman on Thursday September 19 2019, @05:19PM (1 child)
Or, just use the built-in tools and ignore their bastardized language? This tool isn't an online Google service. It seems like your major complaint is Google touched it at one point, so it must be tainted demon spawn?
Joshua 1:9 "Be strong and of a good courage; be not afraid, neither be thou dismayed: for the Lord thy God is with thee"
(Score: 1, Touché) by Anonymous Coward on Thursday September 19 2019, @09:56PM
The reason to play point-and-click game every time anew, instead of run a long-ago written oneliner, is ???
(Score: 1) by fustakrakich on Thursday September 19 2019, @04:37PM (1 child)
Sounds like evidence tampering to me
La politica e i criminali sono la stessa cosa..
(Score: 3, Funny) by The Mighty Buzzard on Thursday September 19 2019, @05:03PM
Like phallic manipulation, it all depends on context. The frozen foods aisle in WalMart or someone else's patient records will both get you in trouble.
My rights don't end where your fear begins.
(Score: 1, Insightful) by Anonymous Coward on Thursday September 19 2019, @06:10PM (4 children)
But is Perl 6 used for /anything/ yet?
(Score: 1, Funny) by Anonymous Coward on Thursday September 19 2019, @07:03PM
You brave! I was too afraid to ask that. No. Just, no.
(Score: 1, Interesting) by Anonymous Coward on Thursday September 19 2019, @09:35PM
I use it for fun, it's a hell of a language. But I can't find anyone to pay me to work on it and I'm not aware of anything with tens of thousands users that is written in it. And of course tens of thousands of users isn't many, to be really notable it would need millions.
I've written a script that wraps ffmpeg and uses it to reencode all of my video files to something compatible with my Kodi gadget. Plus a bunch of other miscellaneous scripts.
So give it a look if the mood strikes. (And I realize that for most people the mood will never strike.)
(Score: 2) by The Mighty Buzzard on Friday September 20 2019, @01:41AM
It was used for my IRC bot, MrPlow for maybe a month. Then it got too painful to use anymore and I rewrote him in C. He's currently written in Rust just because I felt like learning the language.
My rights don't end where your fear begins.
(Score: 2) by edIII on Friday September 20 2019, @03:09AM
Perl 6 is a whole new language, and not a successor to Perl 5. Everyone is still using Perl 5 while trying to suss out 6. Repos are not going to drop 5 anytime soon for this reason.
Technically, lunchtime is at any moment. It's just a wave function.
(Score: 4, Insightful) by Anonymous Coward on Thursday September 19 2019, @08:44PM (7 children)
Devops was done in Perl before Python was a thing.
Python does not handle inheritance loops automatically, Perl does. Polymorphism in Python is an intolerable mess. It is easy as cheese in Perl. These two things are the whole reason you use OO. Sorry but there it is. Perl has better OO than Python does.
Perl was free and well documented and not OO a long time ago, and so a lot of people learned on it and released wickedly evil code into the wild. So you've got this really bad non-OO code on sites like hotscripts and you look at in and go Eeeew! But if you write Perl in OO it is quite clean and fast and easy to maintain and it has been for more than a decade now.
The problem with Perl is that badly written code works. The great thing about Perl is that badly written code works.
There are some distro maintainers in the Linux world that seem to hate on Perl. Debians default Perl distro was busted for years. Probably still is. I guess there are some guys over there that like Python, though they also like SystemD, so there is no accounting for taste.
Incidentally SN is written in Perl. It is based on slash, which is FOSS. I've looked at the source for slash. Eeeeew! But it has lasted decades on now, so there you go.
Python looks clean when you first see it. But then you discover that there is no such thing as an undef in Python, and that all the builtin types do not have compatible existence tests. Then you discover that every object is actually two objects (the class itself is a separate object with distinct isolated functions) and that there are methods to handle inheritance instead of doing it implicitly at constructor time. And then you have all the invented words in the documentation that try to explain that all the non-OO-isms are a feature, instead of just saying Python has some bad internal architecture, and this is how we worked around it to give a sortof OO environment.
I'm sorry but the Python zealots are fucking irritating. "Everything is an object!" Then why the fuck do you Python guys NEVER use multiple inheritance? Oh, and Pythons maintainer has abandoned the project. Maybe he discovered Perl?
(Score: 1, Touché) by Anonymous Coward on Thursday September 19 2019, @09:07PM (1 child)
Just sayin'
(Score: 0) by Anonymous Coward on Thursday September 19 2019, @09:22PM
Larry Wall was Moses in a previous life. Everything was written in Perl at some point.
(Score: 5, Interesting) by legont on Friday September 20 2019, @01:22AM
From what I see the biggest problem with Perl is that it just works. I have mission critical half a million lines of code that was not maintained since developers were fired in 2008 crash and it still works. Nobody at the office realizes the miracle because the thing works and does not bother anybody with outages. The result? We lost Perl in-house expertise.
Java shit was broken numerous times during this time and management had to invest in Java developers, hire them, train them, budget total rewrites multiple times. Hence, there are now thousands of java "experts" in the office and they have to be because otherwise it would simply die. Meantime only a few Perl ones remain and there is not enough man power to upgrade the code for the new environment by two orders of magnitude. They actually want me to rewrite mentioned 500,000 lines of code with help of two off-shores, while it was originally developed by a dozen of guys each one smarter than I.
Kids, if you are reading it, make sure your product breaks often. Don't repeat our mistake. Never develop maintenance free one.
"Wealth is the relentless enemy of understanding" - John Kenneth Galbraith.
(Score: 0) by Anonymous Coward on Friday September 20 2019, @03:14AM (1 child)
"Sorry but there it is. Perl has better OO than Python does."
Perl 5's OO is surreal. You used to have to roll your own objects via 'bless', but now libraries abound for all kinds of crazy object systems. Sounds like a nightmare, but they all work together. If that's not having your cake and eating it too, I don't know what is.
(Score: 3, Insightful) by The Mighty Buzzard on Friday September 20 2019, @03:32AM
I'm on keto, you insensitive clod!
Okay, not really. Any diet that doesn't mandate biscuits and gravy regularly can fuck right off.
My rights don't end where your fear begins.
(Score: 0) by Anonymous Coward on Friday September 20 2019, @04:55AM (1 child)
Somewhere during the late 1990's I got a nice compliment from someone with the relevant experience for making it: my Perl code was among the cleanest and best structured he'd ever seen.
Nowadays I strongly prefer Python. Why? Because it's cleaner in my perception. I remember reading somewhere that Larry Wall basically wrote Perl to fit his own way of thinking. Python is a better fit for my way of thinking. To me that means that one language probably isn't objectively better than the other, they are simply optimized for different brain types.
The way you write about Python suggests you look at it as if it's a different kind of Perl that doesn't behave the way Perl should behave. It isn't, it's a different programming language, not a different Perl. Someone who looks at Perl expecting it to be like Python will have similar criticism and be just as wrong.
I'm not asking you to like Python, I'm perfectly happy if you don't. I'm just pointing out that you're looking at it from a specific perspective which isn't necessarily superior to other perspectives. It's just a preference. You're complaining about Python zealots but your perspective seems to have similar limitations, at least in this post.
(Score: 1, Interesting) by Anonymous Coward on Friday September 20 2019, @03:20PM
I've used Perl Python Ruby and Java, and Python. Python looks clean. But the top down constraints built into it make it bloody murder to refactor. If you code yourself into a corner in Python you have to refactor from the top down. If you code yourself into a corner in Perl, you can can cut the other end off the toothpaste tube. It isn't pretty and you shouldn't do it that way, but you can in a pinch.
In Perl you can pay now or pay later. In Python you pay up front and hope to god you didn't make any mistakes in the initial design. Because it does not tolerate design mistakes. This means that you have to have somebody who really understands software design well to make Python scale. And if they understood it that well, chances are they wouldn't have picked Python.