do one thing and do it well
March 18, 2020 11:30 AM Subscribe
The growth of command line options, 1979-Present (danluu.com): "Someone might argue that all these extra options create a burden for users. That's not exactly wrong, but that complexity burden was always going to be there, it's just a question of where the burden was going to lie. If you think of the set of command line tools along with a shell as forming a language, a language where anyone can write a new method and it effectively gets added to the standard library if it becomes popular, where standards are defined by dicta like "write programs to handle text streams, because that is a universal interface", the language was always going to turn into a write-only incoherent mess when taken as a whole. At least with tools that bundle up more functionality and options than is UNIX-y users can replace a gigantic set of wildly inconsistent tools with a merely large set of tools that, while inconsistent with each other, may have some internal consistency."
To be clear, I'm not saying that I or anyone else could have done better with the knowledge available in the 70s in terms of making a system that was practically useful at the time that would be elegant today. It's easy to look back and find issues with the benefit of hindsight. What I disagree with are comments from Unix mavens speaking today; comments like McIlroy's, which imply that we just forgot or don't understand the value of simplicity, or Ken Thompson saying that C is as safe a language as any and if we don't want bugs we should just write bug-free code. These kinds of comments imply that there's not much to learn from hindsight; in the 70s, we were building systems as effectively as anyone can today; five decades of collective experience, tens of millions of person-years, have taught us nothing; if we just go back to building systems like the original Unix mavens did, all will be well. I respectfully disagree.
If structured data or objects were passed around, formatting could be left to a final formatting pass.
Sure, but I think one trick is to agree on a format for this structured data or objects, which need to be re-formatted before formatting (ultimately) to something textual or at least human-parsable.
One kind of update on the CLI was proposed back in 2011 that uses MIME types to identify data being passed along a pipeline of modular commands, to be able to replace or augment text with other data types (images, sound, etc.). I thought it was quite clever, but recall that this idea was not received well here and elsewhere.
posted by They sucked his brains out! at 11:45 AM on March 18, 2020 [1 favorite]
Sure, but I think one trick is to agree on a format for this structured data or objects, which need to be re-formatted before formatting (ultimately) to something textual or at least human-parsable.
One kind of update on the CLI was proposed back in 2011 that uses MIME types to identify data being passed along a pipeline of modular commands, to be able to replace or augment text with other data types (images, sound, etc.). I thought it was quite clever, but recall that this idea was not received well here and elsewhere.
posted by They sucked his brains out! at 11:45 AM on March 18, 2020 [1 favorite]
If structured data or objects were passed around, formatting could be left to a final formatting pass.
This is a feature of Powershell, IIRC.
posted by mikelieman at 12:04 PM on March 18, 2020 [10 favorites]
This is a feature of Powershell, IIRC.
posted by mikelieman at 12:04 PM on March 18, 2020 [10 favorites]
Coming from a bash/perl background I've found Powershell has grown on me, despite trying several different creams and home remedies. I don't have actual systems automation at work because Reasons, and so I've found that some clever uses of Powershell and locally scheduled cronjobs help me manage both my Windows and Linux environments. (Starting with v6.0 AKA PS Core, can be run on Windows, Mac, and Linux: https://github.com/powershell/powershell )
posted by endotoxin at 12:13 PM on March 18, 2020 [4 favorites]
posted by endotoxin at 12:13 PM on March 18, 2020 [4 favorites]
My experience was that shell scripts were fine as far as they went (I even wrote a passable Eliza program in Bourne shell once) but at some point you need to bring out the powertools. Woe unto you if you want to do anything stateful! It's just barely possible using awk (pronounced "awkkk," I believe), but now you have another problem.
You can sometimes solve a problem by writing a shell loop that reads and processes a line at a time. But if you're honest with yourself, you've already stepped outside of the pipeline paradigm and are now using a clumsy procedural language for the problem.
So into this mess stepped a new and powerful language (perl) that could do everything all in one consistent, though very idiosyncratic, language. There things stood for ages while perl scripts got larger and more impenetrable. Now I find that python can handle most of my scripting needs and I can't remember the last time I used perl. I'm pretty happy using python for my everyday scripting needs and it's cross-platform to boot (as is perl).
When using shell scripts are you sure all the commands you're using are Unicode-aware? UTF-8 lets you get away with a lot of things that seem to work but in reality have bad edge cases that won't bite you until they do. And the thing that bit you might have been written years ago by someone no longer available to help, not that they would remember enough to be of much help anyway. Which brings me to my next point.
There are levels of integration to consider. It's one thing to cobble together commands to solve one-off problems. It's another to develop a little private toolbox of your own common shortcuts. And it's a whole different thing when these scripts end up in build systems or as part of production processes. At that point you really have to worry about maintenance and how much reverse-engineering you'll be foisting on the maintainer. If you aren't using python (or something roughly equivalent like ruby, Groovy, or node.js) you're doing it wrong.
Beyond a certain level of complexity you're going to need a debugger. Best not to mix paradigms either because you really don't want to debug a node script invoked from a python script.
On Windows, PowerShell takes an interesting approach that I, frankly, never quite got my head around or ever found even slightly sympathetic. But it's probably just right for problems I don't come across much.
It seems to me that problems that can be solved well using shell scripts were mostly solved a long time ago. It was good while it lasted.
posted by sjswitzer at 12:15 PM on March 18, 2020 [5 favorites]
You can sometimes solve a problem by writing a shell loop that reads and processes a line at a time. But if you're honest with yourself, you've already stepped outside of the pipeline paradigm and are now using a clumsy procedural language for the problem.
So into this mess stepped a new and powerful language (perl) that could do everything all in one consistent, though very idiosyncratic, language. There things stood for ages while perl scripts got larger and more impenetrable. Now I find that python can handle most of my scripting needs and I can't remember the last time I used perl. I'm pretty happy using python for my everyday scripting needs and it's cross-platform to boot (as is perl).
When using shell scripts are you sure all the commands you're using are Unicode-aware? UTF-8 lets you get away with a lot of things that seem to work but in reality have bad edge cases that won't bite you until they do. And the thing that bit you might have been written years ago by someone no longer available to help, not that they would remember enough to be of much help anyway. Which brings me to my next point.
There are levels of integration to consider. It's one thing to cobble together commands to solve one-off problems. It's another to develop a little private toolbox of your own common shortcuts. And it's a whole different thing when these scripts end up in build systems or as part of production processes. At that point you really have to worry about maintenance and how much reverse-engineering you'll be foisting on the maintainer. If you aren't using python (or something roughly equivalent like ruby, Groovy, or node.js) you're doing it wrong.
Beyond a certain level of complexity you're going to need a debugger. Best not to mix paradigms either because you really don't want to debug a node script invoked from a python script.
On Windows, PowerShell takes an interesting approach that I, frankly, never quite got my head around or ever found even slightly sympathetic. But it's probably just right for problems I don't come across much.
It seems to me that problems that can be solved well using shell scripts were mostly solved a long time ago. It was good while it lasted.
posted by sjswitzer at 12:15 PM on March 18, 2020 [5 favorites]
Interesting article. I don't think I agree with any of the conclusions - it sounds like a recipe for making tools more opaque and harder to debug for the average user yet no simpler - but the facts are worth thinking about. One could quibble about some definitions, e.g., I'd argue that "kill -l" is actually at least 64 different command line options that you have to read some other document to look up. But, it's clear that some command line options are far from simple and inconsistent.
My personal bugaboo is a wish that everyone would allow for long keywords and long escaped commands in terminal software and then use them by default in examples and answers to questions. If you use the tool daily, "-xvzkpf" is handy. But, if you're reading it as an example while trying to do something slightly different with software you've never used before, "--extract --verbose --gzip --keep-old-files --preserve-permissions --file" is a hell of a lot more useful as a starting place. Likewise, in emacs, I'm happy that I don't have to type "M-x save-buffer" every time. But, I'm really glad that I can type "M-x toggle-indicate-empty-lines," since I only do it once every eight years and can guess the name despite having absolutely no idea what the short command is. I don't know if it counts as simplicity in the sense the author means, but it requires one less level of translation to be useful to humans.
posted by eotvos at 12:17 PM on March 18, 2020 [13 favorites]
My personal bugaboo is a wish that everyone would allow for long keywords and long escaped commands in terminal software and then use them by default in examples and answers to questions. If you use the tool daily, "-xvzkpf" is handy. But, if you're reading it as an example while trying to do something slightly different with software you've never used before, "--extract --verbose --gzip --keep-old-files --preserve-permissions --file" is a hell of a lot more useful as a starting place. Likewise, in emacs, I'm happy that I don't have to type "M-x save-buffer" every time. But, I'm really glad that I can type "M-x toggle-indicate-empty-lines," since I only do it once every eight years and can guess the name despite having absolutely no idea what the short command is. I don't know if it counts as simplicity in the sense the author means, but it requires one less level of translation to be useful to humans.
posted by eotvos at 12:17 PM on March 18, 2020 [13 favorites]
As a programmer rather than an administrator, I'm not terribly familiar with these utilities. One could imagine, instead of insisting on THE structure being a sequence of symbols, there could be a set of expressive structures- leaning heavily on graphs, trees, and lists.
Say you have a simple program that just knows how to operate on some simple structure- but the data you actually have embeds that simple data inside some more complex container. Instead of one program that knows how to ingest the data, operate on it, and re-constitute it, you could have a supervising program that knows how to walk over the data, feed it to the simple program, and then reconstitute the complex structure from what the simple program outputs.
(I just snuck the "bind" half of a monad tutorial into this thread)
posted by a snickering nuthatch at 12:21 PM on March 18, 2020
Say you have a simple program that just knows how to operate on some simple structure- but the data you actually have embeds that simple data inside some more complex container. Instead of one program that knows how to ingest the data, operate on it, and re-constitute it, you could have a supervising program that knows how to walk over the data, feed it to the simple program, and then reconstitute the complex structure from what the simple program outputs.
(I just snuck the "bind" half of a monad tutorial into this thread)
posted by a snickering nuthatch at 12:21 PM on March 18, 2020
Generic data structures plus lambda injection? Do I pass?
posted by j_curiouser at 12:28 PM on March 18, 2020
posted by j_curiouser at 12:28 PM on March 18, 2020
TBH, the lion's share of commands I type begin "git", "npm" or suchlike. :)
posted by sjswitzer at 12:28 PM on March 18, 2020
posted by sjswitzer at 12:28 PM on March 18, 2020
The self-documenting sub-command pattern that's well-established these days (see: git, most package managers) sure is better than the ten-million-flag pattern that you still see in a lot of GNU utilities. I'm also hoping Microsoft's cross-platform PowerShell implementation will normalise the idea of commands returning structured data, so we can finally embrace the future that Lisp machines promised 40 years ago.
posted by tobascodagama at 12:41 PM on March 18, 2020 [5 favorites]
posted by tobascodagama at 12:41 PM on March 18, 2020 [5 favorites]
I don't know if it counts as simplicity in the sense the author means, but it requires one less level of translation to be useful to humans.
If you want "simplicity" and "useful to humans" why in god's name are you using Emacs?
*ducks*
posted by axiom at 12:48 PM on March 18, 2020 [10 favorites]
If you want "simplicity" and "useful to humans" why in god's name are you using Emacs?
*ducks*
posted by axiom at 12:48 PM on March 18, 2020 [10 favorites]
Emacs is more your swan than your duck. I mean, ducks can be bad tempered but they won't break your arm.
posted by flabdablet at 12:57 PM on March 18, 2020 [6 favorites]
posted by flabdablet at 12:57 PM on March 18, 2020 [6 favorites]
Commands that accept -h only to emit use --help for help and not outputting the help text anyway can go away.
Commands that emit --help output to stderr can go away.
Commands without manpages, or worse contain a manpage that's a link to a website or PDF, can go away.
Complex commands that accept long command line options but don't have a dry-run/pretty-print option to emit the same command with all options expanded can also go away. Especially if I've used the command in a script and am expecting to remember what it does.
Gnu tar should probably have gone away years ago. No-one needs that many options.
posted by scruss at 1:01 PM on March 18, 2020 [13 favorites]
Commands that emit --help output to stderr can go away.
Commands without manpages, or worse contain a manpage that's a link to a website or PDF, can go away.
Complex commands that accept long command line options but don't have a dry-run/pretty-print option to emit the same command with all options expanded can also go away. Especially if I've used the command in a script and am expecting to remember what it does.
Gnu tar should probably have gone away years ago. No-one needs that many options.
posted by scruss at 1:01 PM on March 18, 2020 [13 favorites]
For years the solution has been apparent and yet nobody seems to want to expand the alphabet so that we can continue having single-character command line flags without running out of letters. Hmmmph.
posted by Nerd of the North at 1:29 PM on March 18, 2020 [10 favorites]
posted by Nerd of the North at 1:29 PM on March 18, 2020 [10 favorites]
The Basic Multilingual Plane of Unicode would provide enough real estate to last for quite some time, flagwise. Obviously you'd want to be careful not to type -о instead of -o or -ᴏ.
posted by Not A Thing at 1:42 PM on March 18, 2020 [5 favorites]
posted by Not A Thing at 1:42 PM on March 18, 2020 [5 favorites]
Mod note: One deleted; sorry, please don't do giant textdumps of eg commands; they're hard for people on mobile and people using screen-readers. Thanks.
posted by LobsterMitten (staff) at 1:57 PM on March 18, 2020 [3 favorites]
posted by LobsterMitten (staff) at 1:57 PM on March 18, 2020 [3 favorites]
"-xvzkpf" is handy.
tar I can do, but the one that gets me is ps. It has *two different sets* of command line parameters and there isn't a mapping from all of them in one to the other.
Obviously you'd want to be careful not to type -о instead of -o or -ᴏ.
I once gave a customer (and our customers are IT professionals) a command to type. He helpfully converted hyphens to some funky dash, and "corrected" the spelling of things like sestatus to setstatus.
posted by Pruitt-Igoe at 2:03 PM on March 18, 2020 [4 favorites]
tar I can do, but the one that gets me is ps. It has *two different sets* of command line parameters and there isn't a mapping from all of them in one to the other.
Obviously you'd want to be careful not to type -о instead of -o or -ᴏ.
I once gave a customer (and our customers are IT professionals) a command to type. He helpfully converted hyphens to some funky dash, and "corrected" the spelling of things like sestatus to setstatus.
posted by Pruitt-Igoe at 2:03 PM on March 18, 2020 [4 favorites]
Commands that accept -h only to emit use --help for help and not outputting the help text anyway can go away.
Oh, man. Software which knows what you want do but will stand on its philosophical objection to you doing it that way is one of my biggest peeves. Some time ago, gimp (which I know is not command-line, so I guess this is a derail) suddenly decided it didn't like people using the File>Save (As) menu option to write to anything but GIMP's internal format, and to use File>Export (As) instead. The program knew what you meant when you asked it to Save As "mypicture.png", say, but instead of actually doing it would chide you to use "Export" instead.
posted by jackbishop at 2:04 PM on March 18, 2020 [18 favorites]
Oh, man. Software which knows what you want do but will stand on its philosophical objection to you doing it that way is one of my biggest peeves. Some time ago, gimp (which I know is not command-line, so I guess this is a derail) suddenly decided it didn't like people using the File>Save (As) menu option to write to anything but GIMP's internal format, and to use File>Export (As) instead. The program knew what you meant when you asked it to Save As "mypicture.png", say, but instead of actually doing it would chide you to use "Export" instead.
posted by jackbishop at 2:04 PM on March 18, 2020 [18 favorites]
Some time ago, gimp (which I know is not command-line, so I guess this is a derail) suddenly decided it didn't like people using the File>Save (As) menu option to write to anything but GIMP's internal format,
Stuff in the Gnome project makes weird choices about How Things Ought To Work all the time.
I used to use gnome-terminal for all my bashing until they removed the "Rename Tab" feature that I always used to bind to a hotkey. I'd have a nice named tab for launching, a named tab for each logfile I had to parse, a named tab for the python source I'm vi'ing ... and yeah there are shell escape sequences that will do the same thing but the labeling of window elements on the desktop is not a shell concern, that's a windowing subsystem concern, so don't tell me to use a shell escape sequence.
The Gnome video player is evolving into some kind of unrecognizable future mutant, every release it gets weirder and weirder. They deleted playlists awhile ago and replaced it with some kind of ... history browser?.. that I can click a back button to access.
(for all I complain, I love the Gnome desktop, I love the Activity Overview or whatever it's called. Lets me bounce back and forth between two very tight modes that both use the whole screen: 1. doing a thing and 2. choosing the next thing to do.)
posted by Sauce Trough at 2:32 PM on March 18, 2020 [3 favorites]
Stuff in the Gnome project makes weird choices about How Things Ought To Work all the time.
I used to use gnome-terminal for all my bashing until they removed the "Rename Tab" feature that I always used to bind to a hotkey. I'd have a nice named tab for launching, a named tab for each logfile I had to parse, a named tab for the python source I'm vi'ing ... and yeah there are shell escape sequences that will do the same thing but the labeling of window elements on the desktop is not a shell concern, that's a windowing subsystem concern, so don't tell me to use a shell escape sequence.
The Gnome video player is evolving into some kind of unrecognizable future mutant, every release it gets weirder and weirder. They deleted playlists awhile ago and replaced it with some kind of ... history browser?.. that I can click a back button to access.
(for all I complain, I love the Gnome desktop, I love the Activity Overview or whatever it's called. Lets me bounce back and forth between two very tight modes that both use the whole screen: 1. doing a thing and 2. choosing the next thing to do.)
posted by Sauce Trough at 2:32 PM on March 18, 2020 [3 favorites]
I can see the thinking with that. You don't expect to be able to use an exported document the same way as a native document. Saving as a jpg means you would lose layers and other information, much like how saving a docs file to a PDF would not result in a fully editable document.
posted by Pruitt-Igoe at 2:32 PM on March 18, 2020 [1 favorite]
posted by Pruitt-Igoe at 2:32 PM on March 18, 2020 [1 favorite]
and the shell is awesome. you need a histogram count of log file events, bucketed by hour and you don't have an ELK install built out? Gimme five minutes. It's a more brutal weapon for a less developed time.
posted by Sauce Trough at 2:34 PM on March 18, 2020 [4 favorites]
posted by Sauce Trough at 2:34 PM on March 18, 2020 [4 favorites]
→ Stuff in the Gnome project makes weird choices about How Things Ought To Work all the time.
Nautilus, aka Gnome Files. No longer puts files on the Desktop because they think no-one would do that. There are numerous crap plugins for Gnome Shell that sort-of replicate the desktop, but none of them do useful things.
posted by scruss at 2:54 PM on March 18, 2020 [3 favorites]
Nautilus, aka Gnome Files. No longer puts files on the Desktop because they think no-one would do that. There are numerous crap plugins for Gnome Shell that sort-of replicate the desktop, but none of them do useful things.
posted by scruss at 2:54 PM on March 18, 2020 [3 favorites]
Software which knows what you want do but will stand on its philosophical objection to you doing it that way is one of my biggest peeves.
Gaaaahhhh. It's a tangent, but you've just reminded me that the self-service checkouts in Sainsbury's will do this thing where if you insert your debit card to pay without selecting the debit card option, it'll yell at you to press the option and then pay. And...just...you can see that I've inserted my card, and you're smart enough to tell what I'm trying to do, so why don't you just select the option for me? ARGH.
posted by Mr. Bad Example at 4:02 PM on March 18, 2020 [8 favorites]
Gaaaahhhh. It's a tangent, but you've just reminded me that the self-service checkouts in Sainsbury's will do this thing where if you insert your debit card to pay without selecting the debit card option, it'll yell at you to press the option and then pay. And...just...you can see that I've inserted my card, and you're smart enough to tell what I'm trying to do, so why don't you just select the option for me? ARGH.
posted by Mr. Bad Example at 4:02 PM on March 18, 2020 [8 favorites]
It's likely that for the save-as and debit card cases it's that they want positive indication of what you intend to do in order to prevent unfortunate errors. Surely we've all occasionally inserted the wrong card. I know I have.
For instance if I type "git rn" it asks if I meant "git rm". Maybe? Probably! But I sure don't want it to just go ahead.
posted by sjswitzer at 4:24 PM on March 18, 2020 [3 favorites]
For instance if I type "git rn" it asks if I meant "git rm". Maybe? Probably! But I sure don't want it to just go ahead.
posted by sjswitzer at 4:24 PM on March 18, 2020 [3 favorites]
The Gnome video player is evolving into some kind of unrecognizable future mutant, every release it gets weirder and weirder.
For the most part, I've stopped compiling stuff for myself, but I still have a ~/software/mplayer, that I'll fire off my build script for every month or so. Nothing ever changes and I like it.
posted by mikelieman at 5:12 PM on March 18, 2020
For the most part, I've stopped compiling stuff for myself, but I still have a ~/software/mplayer, that I'll fire off my build script for every month or so. Nothing ever changes and I like it.
posted by mikelieman at 5:12 PM on March 18, 2020
Do one thing and do it well broke down when things moved from static to dynamic linking. The shared library model with the code sharing that it allowed meant that tools could just be drivers for collections of routines. That combined with the Shell's old resource limited parsing capabilities (and it's shite quoting ability), and the need for starting many processes to get things done (not that bad on *NIX, PITA on some other OSs) eventually led to Perl. Sane quoting, most functionality in single process, complex parser (only 'perl' the program can parse 'Perl' the language). And the famous quote "it's easier to port a Shell than a shell script". This leads to things like 'ffmpeg' with so many command line options that it borders on insanity but it will do almost everything, and ImageMagick's 'convert' which basically has a whole language for it's command line. Still doing the one thing and doing it well, it's just that the one thing is a huge thing.
posted by zengargoyle at 7:51 PM on March 18, 2020 [1 favorite]
posted by zengargoyle at 7:51 PM on March 18, 2020 [1 favorite]
My personal bugaboo is a wish that everyone would allow for long keywords and long escaped commands in terminal software and then use them by default in examples and answers to questions. If you use the tool daily, "-xvzkpf" is handy. But, if you're reading it as an example while trying to do something slightly different with software you've never used before, "--extract --verbose --gzip --keep-old-files --preserve-permissions --file" is a hell of a lot more useful as a starting place. Likewise, in emacs, I'm happy that I don't have to type "M-x save-buffer" every time. But, I'm really glad that I can type "M-x toggle-indicate-empty-lines," since I only do it once every eight years and can guess the name despite having absolutely no idea what the short command is. I don't know if it counts as simplicity in the sense the author means, but it requires one less level of translation to be useful to humans.
The PowerShell community has stepped up in this regard. When PowerShell launched it was super common for people to post examples that relied on aliases of the full commandlets, the magic variables that were inspired by perl and lots of positional parameters. Now when writing scripts in a good editor like Visual Studio Code will start nagging you to avoid the aliases and use the script parameter names and the like. It is a much more welcoming and easier to read/understand environment at the expense of being more verbose.
posted by mmascolino at 7:54 PM on March 18, 2020 [3 favorites]
The PowerShell community has stepped up in this regard. When PowerShell launched it was super common for people to post examples that relied on aliases of the full commandlets, the magic variables that were inspired by perl and lots of positional parameters. Now when writing scripts in a good editor like Visual Studio Code will start nagging you to avoid the aliases and use the script parameter names and the like. It is a much more welcoming and easier to read/understand environment at the expense of being more verbose.
posted by mmascolino at 7:54 PM on March 18, 2020 [3 favorites]
Gnu tar should probably have gone away years ago. No-one needs that many options.
Never mind tar. Had a look at rsync lately?
I enjoy wondering how McIlroy feels about busybox. It provides a suite of commands each of which has fewer options than the mainstream one it's replacing, and it uses very little RAM, but far from doing one thing it literally does everything.
And what zengargoyle says about imagemagick and to an even greater extent ffmpeg is absolutely on point. When the "one thing" in "do one thing and do it well" has grown to the extent that modern hardware makes feasible, there is no longer any way of sweeping the complexity of all that astonishing potential under any conceptual rug. Piling up more rugs, which is what the PowerShell approach boils down to, just makes it harder to work out which layer the painful Lego lump is concealed in.
If I am going to get to grips with doing the kinds of things I like to do with video and audio streams then I just need to understand what I'm doing and why, and the subtask granularity and option syntax of the available tools is easily the least important part of that.
That a given set of scripting-oriented tools can generally be relied upon to do what is required matters way more than any lower bound on the tidiness of the syntax that persuades them to do it; as long as that stuff is documented somewhere, preferably with examples, it's all good.
posted by flabdablet at 1:15 AM on March 19, 2020 [1 favorite]
Never mind tar. Had a look at rsync lately?
Here is a short summary of the options available in rsync. Please refer to the detailed description below for a complete description.My new favourite rsync option, found only a few months ago though I'm sure it's been there for a very long time, is -s, --protect-args. Almost but not always enough to make rsync not break everything if you forget how many layers of quoting to use and/or why. Beware of filenames containing square brackets.
(130-line "short summary" elided on the basis of being too huge for mobile devices and screen readers, per lobstermitten's request)
I enjoy wondering how McIlroy feels about busybox. It provides a suite of commands each of which has fewer options than the mainstream one it's replacing, and it uses very little RAM, but far from doing one thing it literally does everything.
And what zengargoyle says about imagemagick and to an even greater extent ffmpeg is absolutely on point. When the "one thing" in "do one thing and do it well" has grown to the extent that modern hardware makes feasible, there is no longer any way of sweeping the complexity of all that astonishing potential under any conceptual rug. Piling up more rugs, which is what the PowerShell approach boils down to, just makes it harder to work out which layer the painful Lego lump is concealed in.
If I am going to get to grips with doing the kinds of things I like to do with video and audio streams then I just need to understand what I'm doing and why, and the subtask granularity and option syntax of the available tools is easily the least important part of that.
That a given set of scripting-oriented tools can generally be relied upon to do what is required matters way more than any lower bound on the tidiness of the syntax that persuades them to do it; as long as that stuff is documented somewhere, preferably with examples, it's all good.
posted by flabdablet at 1:15 AM on March 19, 2020 [1 favorite]
There are levels of integration to consider. It's one thing to cobble together commands to solve one-off problems. It's another to develop a little private toolbox of your own common shortcuts. And it's a whole different thing when these scripts end up in build systems or as part of production processes. At that point you really have to worry about maintenance and how much reverse-engineering you'll be foisting on the maintainer. If you aren't using python (or something roughly equivalent like ruby, Groovy, or node.js) you're doing it wrong.
I think there's a strong argument to be made that the first two things you mention are what shell scripting was originally designed for and remains good for, and that being used as anything more than glue within production processes just isn't. Shell script is a user tool, and the fact that most IT users now use almost no text-based syntax to control most of what their devices do doesn't change that.
Once a production process involves anything significantly more complicated than invoking a compiler with some standard set of options, then maintainability and extensibility rapidly become important as you correctly note. Letting the scope of a build script creep to the point where these are lost will almost always turn out to be a consequential mistake. But I'd argue that the choice of scripting language associated with your build and deployment processes matters rather less than careful choice of the tools invoked by those scripts and the overall design of the processes themselves.
The stone-age example of this principle in action is the creation of Unix make, a tool designed to capture a recurring build process requirement and make handling it with scripts unnecessary. Make was implemented in the same language as the compilers and lexers and other text processing tools of the day, not in a scripting language.
Javascript is a maintainability nightmare. Given a choice of failing build systems to fix, I would much rather tackle one that used bash to glue competent build tools together in some semi-sane fashion than to have to dive into the murky depths of Node.
posted by flabdablet at 1:57 AM on March 19, 2020 [2 favorites]
I think there's a strong argument to be made that the first two things you mention are what shell scripting was originally designed for and remains good for, and that being used as anything more than glue within production processes just isn't. Shell script is a user tool, and the fact that most IT users now use almost no text-based syntax to control most of what their devices do doesn't change that.
Once a production process involves anything significantly more complicated than invoking a compiler with some standard set of options, then maintainability and extensibility rapidly become important as you correctly note. Letting the scope of a build script creep to the point where these are lost will almost always turn out to be a consequential mistake. But I'd argue that the choice of scripting language associated with your build and deployment processes matters rather less than careful choice of the tools invoked by those scripts and the overall design of the processes themselves.
The stone-age example of this principle in action is the creation of Unix make, a tool designed to capture a recurring build process requirement and make handling it with scripts unnecessary. Make was implemented in the same language as the compilers and lexers and other text processing tools of the day, not in a scripting language.
Javascript is a maintainability nightmare. Given a choice of failing build systems to fix, I would much rather tackle one that used bash to glue competent build tools together in some semi-sane fashion than to have to dive into the murky depths of Node.
posted by flabdablet at 1:57 AM on March 19, 2020 [2 favorites]
If you think of the set of command line tools along with a shell as forming a language, a language where anyone can write a new method and it effectively gets added to the standard library if it becomes popular, where standards are defined by dicta like "write programs to handle text streams, because that is a universal interface", the language was always going to turn into a write-only incoherent mess when taken as a whole.
I would further argue that with the best will in the world the IT industry as a whole was always fated to turn into a write-only incoherent mess purely on the basis of increased uptake, and that this totally unavoidable outcome was pretty much fully realized by about 1995.
posted by flabdablet at 4:09 AM on March 19, 2020
I would further argue that with the best will in the world the IT industry as a whole was always fated to turn into a write-only incoherent mess purely on the basis of increased uptake, and that this totally unavoidable outcome was pretty much fully realized by about 1995.
posted by flabdablet at 4:09 AM on March 19, 2020
This leads to things like 'ffmpeg' with so many command line options that it borders on insanity but it will do almost everything
FFmpeg is a media transformation tool. The plethora of options reflects the diversity possible in inputs and outputs, as well as the various transforms available. The alternative would be to produce a bespoke tool for each permutation or atleast the major classes i.e. one tool for AVI file to MP4 file, another for MKV file to MP4 file, another for MKV web stream to MP4 file..etc ad nauseam.
That said, the volume of options shown by
e.g.
posted by Gyan at 6:43 AM on March 19, 2020 [3 favorites]
FFmpeg is a media transformation tool. The plethora of options reflects the diversity possible in inputs and outputs, as well as the various transforms available. The alternative would be to produce a bespoke tool for each permutation or atleast the major classes i.e. one tool for AVI file to MP4 file, another for MKV file to MP4 file, another for MKV web stream to MP4 file..etc ad nauseam.
That said, the volume of options shown by
ffmpeg -h gives a false impression. For a given input and desired output, most of them are inapplicable. You almost always only want to look at the components involved i.e.ffmpeg -h component_type=component_namee.g.
ffmpeg -h encoder=aac orffmpeg -h filter=scaleposted by Gyan at 6:43 AM on March 19, 2020 [3 favorites]
"write programs to handle text streams, because that is a universal interface"
When using shell scripts are you sure all the commands you're using are Unicode-aware? UTF-8 lets you get away with a lot of things that seem to work but in reality have bad edge cases that won't bite you until they do.
This is what's always bugged me about the "simplicity" of the bash/Unix approach. "Just text" is anything but -- what kind of text? Is it ASCII, ShiftJIS, UCS-2, UTF-8, UTF-16? Can a character be more than one byte? What about line endings -- is it LF, CRLF, CR-only, or (god forbid) LFCR? Is it escaped? Is it RTL text direction?
Blithely ignoring these issues is something the Unix community got away with for years until it caught up with them. It also kept interest in Unix very Western-European-centric for a long time.
Powershell is much better about these things, even if the syntax is an obtuse Microsoft nightmare.
posted by neckro23 at 9:51 AM on March 19, 2020 [5 favorites]
When using shell scripts are you sure all the commands you're using are Unicode-aware? UTF-8 lets you get away with a lot of things that seem to work but in reality have bad edge cases that won't bite you until they do.
This is what's always bugged me about the "simplicity" of the bash/Unix approach. "Just text" is anything but -- what kind of text? Is it ASCII, ShiftJIS, UCS-2, UTF-8, UTF-16? Can a character be more than one byte? What about line endings -- is it LF, CRLF, CR-only, or (god forbid) LFCR? Is it escaped? Is it RTL text direction?
Blithely ignoring these issues is something the Unix community got away with for years until it caught up with them. It also kept interest in Unix very Western-European-centric for a long time.
Powershell is much better about these things, even if the syntax is an obtuse Microsoft nightmare.
posted by neckro23 at 9:51 AM on March 19, 2020 [5 favorites]
I once gave a customer (and our customers are IT professionals) a command to type. He helpfully converted hyphens to some funky dash, and "corrected" the spelling of things like sestatus to setstatus.
If they pasted it into Outlook or Word, that would have mostly been automated for them.
posted by sjswitzer at 10:14 AM on March 19, 2020 [3 favorites]
If they pasted it into Outlook or Word, that would have mostly been automated for them.
posted by sjswitzer at 10:14 AM on March 19, 2020 [3 favorites]
That said, the volume of options shown by ffmpeg -h gives a false impression.
ffmpeg -h, or the ffmpeg man page, or man pages in general, are references, pure enumeration of functionality, and not great learning materials IME. Those sources are my second stop during my annual "oh I gotta do something with ffmpeg" google spasm.
(❤️s to the dorks out there blogging "today I did this thing with ffmpeg!" you have helped me a great deal over the years)
posted by Sauce Trough at 2:17 PM on March 19, 2020 [2 favorites]
ffmpeg -h, or the ffmpeg man page, or man pages in general, are references, pure enumeration of functionality, and not great learning materials IME. Those sources are my second stop during my annual "oh I gotta do something with ffmpeg" google spasm.
(❤️s to the dorks out there blogging "today I did this thing with ffmpeg!" you have helped me a great deal over the years)
posted by Sauce Trough at 2:17 PM on March 19, 2020 [2 favorites]
If they pasted it into Outlook or Word, that would have mostly been automated for them.
Apologies for the derail, I'm quarantined.
So last year I worked with a major accounting software vendor on a database migration project for my company. they had to write a lot of SQL for us that would bring our database from Old Shitty State into New Clean Sparkling State.
Their source control was a word file that they stored all of the commands in.
The commands were interleaved with narrative and were generally bolded to distinguish them from the narrative.
They managed versioning by changing the filename, and not in a consistent way, one of them used incrementing NATO phonetics (Alpha, Bravo, Charlie, Dog) to mark versions. Another used major.minor.patchlevel.
They distributed / shared the source via email attachment.
Sometimes someone would edit it with Word's "smart quotes" feature toggled on. the resulting email chains were hilarious.
posted by Sauce Trough at 5:07 PM on March 19, 2020 [6 favorites]
Apologies for the derail, I'm quarantined.
So last year I worked with a major accounting software vendor on a database migration project for my company. they had to write a lot of SQL for us that would bring our database from Old Shitty State into New Clean Sparkling State.
Their source control was a word file that they stored all of the commands in.
The commands were interleaved with narrative and were generally bolded to distinguish them from the narrative.
They managed versioning by changing the filename, and not in a consistent way, one of them used incrementing NATO phonetics (Alpha, Bravo, Charlie, Dog) to mark versions. Another used major.minor.patchlevel.
They distributed / shared the source via email attachment.
Sometimes someone would edit it with Word's "smart quotes" feature toggled on. the resulting email chains were hilarious.
posted by Sauce Trough at 5:07 PM on March 19, 2020 [6 favorites]
Their source control was a word file that they stored all of the commands in.
Reading that sentence literally literally made my scalp prickle with fear.
It's still doing it, a bit.
posted by flabdablet at 11:45 PM on March 19, 2020 [3 favorites]
Reading that sentence literally literally made my scalp prickle with fear.
It's still doing it, a bit.
posted by flabdablet at 11:45 PM on March 19, 2020 [3 favorites]
sometimes I get the feeling that my career's sole purpose is to enable me to deliver the stupid version of the Rutger Hauer monologue from Blade Runner on my deathbed.
"I've seen things you people wouldn't believe ... scp as a production deploy tool ... Word used as a version control system ... all those moments will be lost like a number two flush. Time to die."
posted by Sauce Trough at 4:59 PM on March 20, 2020 [5 favorites]
"I've seen things you people wouldn't believe ... scp as a production deploy tool ... Word used as a version control system ... all those moments will be lost like a number two flush. Time to die."
posted by Sauce Trough at 4:59 PM on March 20, 2020 [5 favorites]
NATO phonetics (Alpha, Bravo, Charlie, Dog)
That's... delta? Unless they switched from NATO to "sounds like" midway through.
posted by axiom at 1:31 AM on March 23, 2020
That's... delta? Unless they switched from NATO to "sounds like" midway through.
posted by axiom at 1:31 AM on March 23, 2020
That's... delta? Unless they switched from NATO to "sounds like" midway through."As someone who grew up with the ham radio (ITU) conventions, I always find alternatives weird. But, I'm not sure they're the best. Using Greek letters as western alphabet "first letters" is incredibly strange. The first letter of alpha and beta are . . . alpha and beta. And even I don't have any idea what "oscar" means. But, at least they tried to be sorta cosmopolitan and inoffensive. And, standards are useful, even if they're imperfect.
posted by eotvos at 1:39 PM on March 23, 2020
it's possible that they were just as good at phonetics as they were at database programming.
There was definitely a Dog release. The owner recalled the email containing it a couple hours after he sent it, and then a few hours later he sent out a replacement emailwith a corrected version of the file, but with the exact same filename as the recalled version. He didn't want to skip Dog, he explained.
posted by Sauce Trough at 1:32 AM on April 1, 2020
There was definitely a Dog release. The owner recalled the email containing it a couple hours after he sent it, and then a few hours later he sent out a replacement emailwith a corrected version of the file, but with the exact same filename as the recalled version. He didn't want to skip Dog, he explained.
posted by Sauce Trough at 1:32 AM on April 1, 2020
The owner recalled the email containing it a couple hours after he sent it, and then a few hours later he sent out a replacement email with a corrected version of the file, but with the exact same filename as the recalled version.
Microsoft Windows patches have been released with similar version number fuckery on more than one occasion. I have spent quite a lot of time manually removing KB updates from automatically updated Windows boxes to force them to "re-acquire" the "same" update but without the show-stopper bug.
posted by flabdablet at 1:56 AM on April 1, 2020 [1 favorite]
Microsoft Windows patches have been released with similar version number fuckery on more than one occasion. I have spent quite a lot of time manually removing KB updates from automatically updated Windows boxes to force them to "re-acquire" the "same" update but without the show-stopper bug.
posted by flabdablet at 1:56 AM on April 1, 2020 [1 favorite]
« Older Covid-19 and the LGBTQ community | while it's there, it's invincible on behalf of the... Newer »
This thread has been archived and is closed to new comments
posted by grumpybear69 at 11:40 AM on March 18, 2020 [7 favorites]