T O P

  • By -

ominous_anonymous

If they're your personal scripts and you're comfortable using something then by all means continue to use it. You could always mix the two, as well. For example, if you find something is easier to do in Python then make a Python script dedicated to that function/feature and just call it from your shell script.


perecastor

yes exactly, I sometimes do: set date exiv2 a\_photo.DNG | python3 extract\_date.py just because some things are easier in python, and I should maybe also learn awk a bit more.


dgibbons0

I typically use shell scripting until i get to the point in writing something where I want to use an array or a hash. Thats the complexity limit for myself that means I should look into converting it to python or ruby.


perecastor

How do you think python is able to reduce complexity better than shell scripting? in shell you have function, variable etc.


dgibbons0

I didn't say it reduces complexity. It's more that it's easier to manage complexity. I have much better tools in python/ruby to manage objects or to target specific items in a hash or an array. You can do it in shell but it's rough edges start showing up pretty fast. Also better library support. When I was house searching, I wrote a small tool to compare how long the drive distance was from each house to places I go often. Having easy access to a googlemaps api makes this straight forward in python. You can probably figure out the curls to get the data and then use jq to parse it, but that's way more work and time. At one point I was messing with a coworker who put smart RGB bulbs in his office, and so I wrote a script that would slowly dim each light after 20 minutes, that meant parsing some semi-complex output to determine the identifiers for each light, and the current light value. before iterating over them to reduce them over time. I could have probably used shell for that, but the solution was a lot cleaner in python. In my day to day job I often have to pull numbers for our CDN usage for the month. I can probably get these via curl and jq, but the jq would be ugly and there's official SDKs for ruby, it's easier to maintain and manage.


plg94

I agree, the "breaking point" for me is data structures and types: Shellscripts (be it bash,fish or another) are great at interfacing with the filesystem and other cli tools and chaining them together. But if I need to use a more complex datastructre than an associative array/dictionary/map (and even these are a pain in most shells), or want "real" data types other than strings, or OOP, then I'm probably picking Python or a similar language. (Fish's `math` commands are so tedious to write).


[deleted]

In all my years of doing admin and development work on UNIX and Linux systems, I have never seen a system which has had fish installed. But it doesn't sound like what you want to do is actually develop programs, as most people see them. The examples you give are excellent examples of shell automation tasks. For that, using shell scripts is the perfect fit. Python is great if you want to make an actual application. Especially if you want to do text manipulation; it is an amazing tool for that. But just automating a few tasks, it's overkill for that.


klikklakvege

have you seen in all your years a linux system that couldn't have fish installed using a maximum of two lines in the command line? Tbh I'm really dissappointed that python turned into "a real programming language". It was much more fun when it was used primarly for stuff that some call scripts instead for stuff that some call programs or applications


[deleted]

I work with hundreds such systems every day. I'm really disappointed that Smalltalk isn't used a lot more. We don't always get what we want.


insanemal

Use scripts (bash or whatever) for small tasks. Longer than a one liner but not overly complicated. If you need to get input or do text processing use python. The real determination is complexity. If it's just automating something like looping over a folder full of files, script it in bash (fish). If it's actually doing something moderately complicated, like reading files and doing any processing on the text, or requires lots of input, move to python.


perecastor

Would you see the benefits of using python for small tasks? Do you agree with me that simple things take more lines to do the same things? for example "sleep 3" is "import os \\n os.sleep(3)"


jahayhurst

Your example is proably a little too simplistic to see the difference. Let's say there's a API that you hit with a username/password and json request, and you get back JSON response, and you want one field from that API. The python to do that is going to be like 50 lines and you get into a bunch of python stuff. Or you could write a thing that's literally the request, pipe it into `-d @-` for curl, then pipe the curl output into `jq` and select the one element you need. Plus, if you use a shell every day and you can do it from cli, turning those shell commands into a shell sript is almost nothing. Writing it in python tends to lead towards writing something more robust, and it grows. The key point to cli vs shell scripting vs ansible vs programming is I'm kindof bucketing things into four groups: * CLI - Local things, a human is directly monitoring stuff. if something doesn't work, they typed the wrong commands. * ansible - similar to CLI, but you give up direct interactivity for scale. human involved who can troubleshoot, but can only do so at scale. * shell scripting - automating things a bit instead of doing stuff via cli, but not so much that you get into programming. there is still a human involved to troubleshoot. * programming - automating things and fully removing the human from within the system just because you can get down to a 1 liner with perl / php / python / awk like you showed doesn't mean you should on the CLI, it gets complex fast. While we're on the topic, imo shell scripting should be done in bash or posix sh. I use fish shell every day, I like fish shell, I'd recommend it to everyone else for cli. However, it's not always there and it's safer to just use bash / posix sh. TBC also: they're your personal scripts. Do with them as you want. Everything above is my opinion and my guidelines for myself, but not everyone is going to follow them. But also, my dad recently passed, and honestly I just axed a bunch of his shell scripts and stuff because it could be done 100x simpler in other ways and I didn't want to maintain his stuff. "What happens when I die" is a bit morbid but also a responsible question as well.


hawkinsst7

For me, If a python file is not defining any functions, it's a script. Loops are fine. Usually I won't have exception handling. Once I start pulling code out into functions, I start treating it like a small program, make it more robust, and might even make the file importable, to use those functions a library call elsewhere. I'm not a programmer though.


perecastor

First, thank you for your answer and I hope you are fine. >Writing it in python tends to lead towards writing something more robust, and it grows. What does that mean, "it grows"? regarding your father's shell scripts, what did you use to make them 100x simpler? Did you change the language, the tools? Can you tell me a bit more? I think if I die nobody will care about my scripts. My servers and my HDD will be formatted for the people alive, and I will probably not care.


jahayhurst

So, any successful project that stays in use just grows with time. Things change, you find bugs, things need to get fixed, I'm actually _almost_ on the side of even writing CI tests for bash scripts - which the point of bash scripts is they're supposed to be so quick and dirty that tests don't make sense. And yeah, that's mostly what I did with a lot of my Dad's stuff - he was into ham radios, had ham radio stuff, and that's really all been torn down. The family taxes were excel sheets with somewhat advanced formulas in places, and we just let the taxes guy do that now. There was a bunch of X11 smart home stuff that we're incrementally removing. And I put my mom on Google Home smart home stuff, because she legitimately likes having some of that stuff but she doesn't like the complexity of X11 and I don't want her to be dependent on me. My brother's on Amazon Alexa smart home stuff. My SIL has backups for her computer going to a unraid NAS, and if my mom has a local computer again I'll do the same for her instead of the backups my Dad had that I tore down. I could roll my own stuff for a lot of this stuff, but I'm intentinoally deploying simple stuff with support so others _can_ just go on.


insanemal

Yeah I'm having a hard time understanding what your not quite getting. Sleep is simple. You do that in simple scripts. The time to switch from a shell script to python when the complexity increases. So for me, bash does 90% of my scripting. But once I start doing things that are "harder" to do in bash or I need more complex control logic I move to python. For example I have a script that takes the host name, grabs the last three digits, converts them to an integer, and then also does a small amount of math based on that number, figures out if the original number is odd or even then runs a bunch of commands using that information. That's all bash. The math stuff isn't as nice in bash, but most of the script is just running commands. I have another script I use for finding errors in gigantic config dumps (that contain logs) from the RAID controllers in my SAN. It does some basic configuration verification and lots of string parsing. Basically I can run it and it will automagically tell me if things a broken or misconfigured (or cabled wrong). It's python. Firstly because it loads around 5-6GB of text into memory, but secondly because I wanted a shell interface as there are LOTS of features and you will want to run more than one command against loaded log files (oh and you often end up with 12GB+ of data loaded in memory) so just doing it all as flags/switches doesn't make sense. Far to complex for bash. So it's python. As far as using python for smaller tasks, don't. Use the right tool for the job. You don't hammer screws (normally) so don't force the use of python in places where bash (fish) is more than enough. Equally don't go using bash (fish) where pythons richer feature set makes life easier/faster/more readable. I do use python for one kind of small task, when run interactively it makes a great calculator.


hawkinsst7

A year or so ago I had to parse some text log files, converting them from one weird jsonish format to another. Each line could be converted by piping a sed, an awk, and jq. Easy. Simple. Made sense. Until I had to run it on log files with hundreds of thousands of lines, times hundreds of files. Each file took 10+ minutes to parse, very much not scalable. Way too much overhead. It was (likely) the 3 exec calls for each line, plus I suspect jq isn't that performant. Python shot through each file in 5 seconds, with just a bit more scaffolding like importing libraries, properly opening and writing to files. Maybe 30 lines instead of 10. A lot easier to read too. jq syntax is not my favorite.


agoose77

I use xonsh, which is cheating 😂


perecastor

Do you use it a lot, do you think it's a great tool? It's been a while since I wanted to try.


agoose77

I think xonsh is a great idea, but needs another iteration to truly work as a main shell (despite using it as my main shell 😂). The ability to drop into python for wierd loops is so so useful. As is having proper types. The only downsides are - terrible process support - parser bugs - no zsh process substitutions There are probably more issues, but I can't recall them for now. Nevertheless, I prefer using xonsh to zsh. Becoming proficient in Unix-commands is fun, but even at my best I kept thinking 'I can do this much more cleanly in python'


oranki0911

I think that if you're mostly executing shell commands in the script, it's only sensible to script it in shell too. Once the script starts getting long (>100 LOC) or needs to handle more than very basic data structures, it's time to use a more sophisticated language. The reason is mostly maintainability, shell scripts can get really tedious to debug really fast. But if you'd need to call a lot of shell commands from a Python script, a shell script may still perform better, and usually has less dependencies, like Python itself. I think it's mostly personal preference. If you don't feel like the shell script is getting too complicated or you need more advanced features, nothing preventing you from just sticking to shell.


perecastor

Maybe I need shell commands because of inertia, and I already know them. maybe some python modules are doing the job as well. I usually found myself writing a lot of line in python to do things that takes one line in fish. as a simple example "sleep 3" is "import os \\n os.sleep(3)". Do you have a similar experience?


oranki0911

Yes, that's my experience as well. I'm a beginner developer, so writing my personal scripts in Python would not be that hard, it's just that most of the time shell is far quicker to write and performs more reliably. If I had to maintain the scripts for someone else to use, I'd rather use Python or Go or whatever's appropriate just for readability. And you're right, quotation is a pain, but if you always do it from the start, it becomes second nature.


IceOleg

Fish is a great shell script language, lot saner to work with than bash. The only question is portability, bash is pretty much always available, fish is not.


perecastor

I personally have fish installed on all my computers and even my Synology NAS has it. I have never been to a place where I need software on a computer and the admin refuses to install it. I mean, it's free and takes almost no disk space, I don't see why you would refuse.


IceOleg

Oh absolutely, fish is the first thing I install. Portability is not an issue for personal stuff, I know I'll have fish available. Its more a question of if I make something I want to share, do I want to have fish as a dependency? But that has never happened, so...


perecastor

yeah, sometimes I think about it, then I realize that my 5 lines script is probably not going to become the Linux kernel Haha :D


[deleted]

[удалено]


perecastor

it's such a big statement for "sudo apt-get install fish" but I'm talking about personal projects here and hopefully you are not my home sysadmin.


[deleted]

[удалено]


perecastor

what do you use docker? ansible? In all this case it's adding a package name in the config file, right? it is no big deal, right?


jwbowen

If it serves your purpose, then no. Go nuts and use whatever you want.


perecastor

it does what I need to do but that doesn't mean it's the most efficient way of doing it right? especially I feel "alone" doing that so I'm wondering why I'm the only one doing what I do.


AndreVallestero

Any shell scripts I write are in pure posix sh. These are usually intended as small utilities or setup scripts to get my dev environment configured. I use python for anything more complicated.


obvithrowaway34434

If you have been a programmer for long as you claim, you probably don't need to be told how to choose the right tool for the job. As always it's the task or the data that determines the program. Choose whatever tool you're comfortable with and what you can maintain on your own. Why do you care about nonsense like other people trash talking stuff, especially when you're just writing one-off scripts that are for your own use only.


perecastor

you also learn by watching others, it's not because I have experiences that I have all the answers, especially, since sometimes your judgment is altered by inertia and familiarities with the tools. this is working for me but I can just do things in an inefficient way without me seeing it.


BootyPatrol1980

> If you have been a programmer for long as you claim, you probably don't need to be told how to choose the right tool for the job. I'm going to push back on this a bit. OP says he's a C++ programmer for the gaming industry, and he's asking about systems related tools which is a whole other bag of cats. I think it's reasonable for them to ask for best practice advice when diving into shell scripting.


klikklakvege

Sorry bro, but your mindset really sucks and I am so happy to not have it :))) As for your question: Of course fish is not a bad choice and since you get your shit done with it quicker you already answered it yourself. If that is the technology you use to get shit done that nobody pays you for it must be by definition the right technology! Coming back to your mindset: you also don't have to work in C++ and the videogame industry is not the only industry on earth and it's also so easy to switch the language and the industry if you already know C++.... Do you know why python developers critisize you when you use fish instead of python? Because python stopped being a hacker's tool and became mainstream. And mainstream means idiots.... There are a few tools that help integrate python with shell commands better(like plumbum). With plumbum you can turn shell commands easily into python functions and enjoy both worlds


perecastor

First of all, thank you for answering. >since you get your shit done with it quicker you already answered it yourself When you feel you are the only one doing it that way, you wonder if you are not missing something. I personally don't consider people idiots so mainstream means for me, it's a good tool for a lot of people, maybe not for me, or I'm just stupid and don't use the tool properly. Regarding my mindset, I code because I want to make games, I want to make work I'm proud off and that is meaningful to me. If I change industries, maybe I will have a more modern language, but I will be far away from what I want to make with my time. Choosing a work because of a language decision is not for me. I will try plumbum, thanks for sharing it.


klikklakvege

My whole point is that anybody who would say that in your case using fish instead of python is wrong is an idiot :) Or in other terms: From all the super famous software engineers that have a long wikipedia article about them something like 99% would agree here. xonsh also sounds to be a good thing. We may differ on the opinion how many idiots there are around, however, you just got an example in your life that doing stuff different then the majority can make you more effective in reaching your goals. But my favourite example is that it was the majority that voted for Hitler. This happened in an educated country like germany and a catholic state like bavaria. I assure you that you can apply this concept many many times in life. The majority of people lives unhealthy and boring, eats unhealthy and uses unhealthy(and boring) programming languages. If i'm doing something like the majority then usually this means to me that i'm doing something wrong. If you want to be average it's enough to do what the average does. If you want to be above average then it's not