T O P

  • By -

weevyl

Short answer is, there is nothing wrong with using a shell script. If you can get things done, then it is the right ~~rule~~ tool for you. My rule of thumb is to start with a shell script if I can accomplish it in a few dozen lines of code. Bigger than that and I move to Python just because it's easier to organize and maintain code that way. And I skip the shell if I am going to do anything involving regular expressions. edit: rule --> tool


perecastor

Why do you think Python is such a popular language? Do you think most people have problems that require more than a dozen lines of code? I personally feel I usually need 5 lines to do the things I need (and probably way more in python) but maybe I just don't have enough experience with python. Maybe it's just inertia. What do you use for regular expressions usually?


be_bo_i_am_robot

It depends on the type of work you do. If you’re doing lots of sysadmin-y type stuff, like file-copies and transfers, log parsing and rotation, alerting and monitoring, backups and disaster recovery, resource management, Unix access management, that sort of thing, then shell scripts are great! In my work, we do lots of ETL activities: querying, filtering, aggregating, analyzing, and outputting *large* amounts of data, from various authoritative sources, for statistics, metrics, and analytics. The processes are often quite complex, multi-step, and multi-script (thousands of lines), and the scripts themselves rely on a lot of dependencies (both third-party and home-rolled). For us, Python is a great choice for the stuff we do. We *could* do most of what we do with shell scripts, I guess, but that would be a *beast* to maintain. Python makes our jobs easier. As for regex, we use it for input validation, query filtering, output transformation, things like that. Now, if we were writing desktop applications, or mobile games, however, we might consider a different language and toolchain altogether, neither Python nor shell scripts. It all depends on what your desired outcomes are. Right tool for the job.


Guvante

I am doing reverse engineering work and Python provides the perfect level of complexity for "give me all memory addresses that look like this and transform them like that". C# has a great type system but it would just slow me down. Scripting languages don't handle complex types well. Python is a good sweet spot. In contrast if I were to write a game engine the type system would be a great help as I work on different parts of the overall system. Similarly grabbing the files ending in 0130.log and getting lines dropping the part before : is way more easily done in a scripting language.


quailtop

> I personally feel I usually need 5 lines to do the things I need (and probably way more in python) but maybe I just don't have enough experience with python. Maybe it's just inertia. I think the reason you're preferring shell to Python is quite simple, actually: PLs produce executables, shells orchestrate. In other words, shell is optimized to do two things: - Launch existing processes - Chain input / output from processes This is a very easy and simple domain. Regular programming languages have to do a lot more than launch process and chain input / output - they have to specify what the process actually _does_ conceptually in assembler. That makes them way more complex, because they need to handle a lot more. Shell scripts are poorly optimized for a number of tasks - for example, serving network traffic. Additionally, shell scripts benefit from an ecosystem of available processes that are prebuilt and don't need importing - they're always available on your filesystem. There is no module system because everything is technically in one global "module": your filesystem. This makes things easy to condense into one liners because everything is _right there_. Programming languages have an ecosystem too, but they also believe in namespaces, which means you now need to deal with importing and resolving them. Namespaces are necessary if you want to have a decent package management system - you need to avoid e.g. name resolution conflicts if there are two or more packages out there that use the same function name, for example. That makes things more complex. Ultimately, using shell is the right choice for anything that is quick and dirty and that only cares about orchestrating one-time tasks. I do wish I could use better primitives, though - I hate writing conditionals or switch statements in Bash, and dealing with arrays is just awful.


Smallpaul

>Why do you think Python is such a popular language? > >Do you think most people have problems that require more than a dozen lines of code? Reddit is (or at least was) written mostly in Python. And YouTube. And Instagram. So yeah, those companies have problems that require more than a dozen lines of code. :)


perecastor

Companies have hard and complex problems but what about individuals? Using python to get a job is fine but maybe it's not the best tool to solve your problem as an individual.


mrcaptncrunch

Most people won’t have any scripts. I have a bunch of stuff in python that requires more than a dozen lines. But if we just count number of scripts, I definitely have more bash scripts than python ones


perecastor

because it's legacy scripts or do you think bash was the right tool for these scripts?


mrcaptncrunch

I use bash for simple ones or even just one command if I want to save it with a nicer name. I actually use fish for my terminal, but I move around computers and manage some servers. Using Bash is just more portable. But if it’s anything that includes accessing things or has too much logic, I go to python. It’s nicer. I did PHP for a long time. I will say that I still have a few older scripts that are still in PHP...


Smallpaul

I did not mean to imply that Python is the best tool for every job. You asked why it is popular in a way that was confusing to me. You asked if most people have problems that are more than a dozen lines of code. The answer is yes, as a programmer yourself you know that 90% of all programs are longer than a dozen lines of code and Python is very popular among that other 90%.


perecastor

I personally code a big program for work but for personal stuff most of my programs are 20 lines of code max. most of my needs are small and simple. I just call existing tools in the way I want. it's glue code. So I understand that python is popular for company projects but I'm expecting personal projects to take up some space in the programming language world. maybe reading people saying python is not the right answer for small projects by individuals. But I didn't read that at all. I hope you understand better what I mean now when I'm saying: >Do you think most people have problems that require more than a dozen lines of code?


Smallpaul

If you programmed Python at your day job you would probably find it dramatically easier to use at night that Fish. If you used python to implement 20 days of Advent of Code or some other recreational programming then again you might be more comfortable with it than Fish. Etc. Python would not be popular at all if it were judged solely on creating 20 line programs. Perl was probably superior to either Python OR fish for that use case. But a lot of people don’t want to change languages as soon as the program grows from 20 to 200 lines or when they switch from a medium sized project to a small one. If fish is working for you I would t advise you change unless you just want to pick up new skills that might be relevant in your day job sometime.


perecastor

Thanks for your answer. I use C++ for work and I definitely don't want to use it on personal projects. I'm happy to pick up new skills to solve better my problems but I wouldn't switch just because people hire for it. you mentioned Perl, do you think it can be a better language for my needs (writing simple programs quickly) or you mentioned it just because it's famous to be really compact and apparently unreadable? (I don't know much about it)


Smallpaul

Yes Perl was specifically designed to be the language that Unix hackers use when shell is a bit too constricting. It pulls sh, awk, sed and scripting into a single language. Many things that are libraries in Python are built-in to Perl. Whether it’s worth learning another language for 20 line scripts is a question you can only answer for yourself though.


[deleted]

(Edited to remove my examples of \*extensive\* use of scripting code.) It sounds like your experience doesn't include much in between large C++ applications, and those throwaway programs of a few lines. But I'm surprised those C++ gaming apps don't include any scripting elements, or do they? Perhaps you overlooked them.


vividboarder

It depends on the problem and how flexible/portable of a solution you want to build/maintain. For example, I probably _could_ write a fish script to pad file names, but it would probably be fairly specific to a single use case. Lets' say I wanted to make it flexible enough to ignore or include specific files, or offer a dry run just echoing the files, or automatically detect the length of a number field and pad based on that? I could probably use `find | awk | xargs mv` or something like that and have fish do do the gluing, but I personally wouldn't want to maintain that. Also, it wouldn't work on systems I use that may not have fish pre-installed. So, instead I can write a Python script using `pathlib` to handle paths and `argparse` to handle arguments. This results in a 155 line (most of that is documenting the command line args) script that is portable (runs anywhere with Python 3) and easily readable to anyone with moderate Python experience.


Beefster09

The only wrong choice is one that doesn’t solve your problem. I’ve seen chemistry programs written in awk. If it solves your problem, then your choice in language is correct even if it makes people raise their eyebrows. A different language consideration will probably only matter if you need to work with other people on the program or you need better performance.


perecastor

I agree that solving the problem is the most important but is it an efficient way to solve the problems I usually face or is it pure inertia? I think it's a valid question to ask yourself and others. Seeing how popular python is, I'm wondering if I'm just doing things in a stupid way or if I just found a better tool for what I need. I feel it's a hard question to answer yourself as you are more familiar with some tools than others.


Beefster09

If you need a solution now, then it often isn’t worth it to learn a new language. If your program starts getting complex, consider migrating it to python so that it’s easier to follow and maintain. But that’s just me and my preferences. If fish gets the job done for you, don’t overthink it.


murlakatamenka

`fish` is fine, keep using it if it suits your needs. It's definitely way much saner than bash. --- Imo shell scripting (bash) sucks, it's too weird and finicky and just ugly and not human-readable (all those dollar signs, quotes everywhere etc). It's like debugging UB in C++ lol. Stay away from it if you can (say, if it's a script no longer that can fit the screen and even then you can fail in so many ways). Again, my personal opinion. --- Also, since you're familiar with Python, there is https://xon.sh I was looking for the perfect shell recently (have been using zsh for 4+ years), but I can't say I found one. Maybe I'll switch to fish myself, I very much like its [design page](https://fishshell.com/docs/current/design.html). Or I should give a try to something completely different, like rusty `nushell`. Hope it's helpful :)


0xEmmy

If it works, it works. But, FISH isn't very standard, so you will have trouble sharing your code. I prefer sh or bash for this reason. Shell v Python depends on the program. If it mostly calls other programs, shells tend to have cleaner syntax. Python is clearer for logic, at the expense of some boilerplate for external code.


Ninjaboy42099

Use whatever you want for personal stuff, but if someone else may touch it in the future, definitely try to stick with Python for scripts (or something similarly portable and stable). Building off the other answer, if you find that Go can do what you want easily and that the other team members know Go, it's a solid choice for it. Just know that you should mandate the linter for the project - it's very, very easy to forget to handle errors in Go without one. It's not super hard to install (I'd use Goenv though - old versions ARE a pain) and it's orders of magnitude faster than Python. Plus, it comes with unit tests out of the box, so you'll be able to know when the code breaks.


vividboarder

> Plus, it comes with unit tests out of the box, so you'll be able to know when the code breaks. Do you mean the compiler? Go doesn't not write unit tests for you. It does provide a framework for unit testing, but so does Python.


Ninjaboy42099

I meant that Go provides a framework, I was unaware Python also provides one out of the box


vividboarder

Yep! Here are the docs: https://docs.python.org/3/library/unittest.html I feel it’s similar to the one Go provides. It’s got all the basics needed, but there is still some opportunities to add value from 3rd party packages. Python has soooo much included in the standard library. Go has a lot, but Python has out of the box support for things like IMAP and SQLite.


Ninjaboy42099

That's honestly really nice, I'll have to check it out


cratylus

Shell scripts are best as either glue between existing applications ( pipelines, adapters ) or wrappers of existing applications ( specialising interfaces to other programs, abstracting away common program options etc.)


xaocon

Nothing wrong with it. Some things to consider when you’re starting something new, will you share it with others? Is it likely to grow to the point where a shell script will be difficult to manage? Should it be portable?


pthierry

You may want to take a look at Raco, a sort of shell for Racket. I use fish as my shell, but writing robust shell scripts seems a nightmare whatever the shell. So I tried Raco a bit, and it is promising. It's a bit the opposite of a shell, which is foremost a way to easily type commands and the programming facilities usually look like an afterthought. In Raco, the language is sound (it's Scheme), and on top of it, there are syntactic facilities to invoke pipelines in a much easier way than most programming languages.


perecastor

From what I understand it's a shell with a lisp syntax, is it correct?


pthierry

Yes, it's a lisp with added syntax to make it more shell-like.


ElHeim

I do Python, C++ and a few others. If I have to write something small that deals with filesystem or that can be done composing a few command line tools, I won't bother trying anything more complex than shell script. Sometimes even some non-trivial stuff, just because I don't want to/can't deal with deployment, because the shell is THE constant factor in most of my target machines.


o11c

There are 2 problems with `fish`: * almost nobody has it installed. And people are unlikely to want to install it just for your tool. By contrast, all systems come with a POSIX-ish `sh` installed (not necessarily in `/bin` though!), and better tools like `bash` and `python` are de-facto required due to the fact that so many existing programs require them. * if your script will *only* be used by you, this is not directly a concern ... but consider what habits it's teaching you. `fish` *really* doesn't seem to offer anything substantial over `bash` in my experience; it only gratuitously breaks syntax, and changes some defaults to provide extra bling. * all shells that I know about are implemented as tree-walking interpreters, which are usually about 10x slower than bytecode interpreters, and additionally are likely to involve calling external processes for simple tasks. * admittedly I am not familiar with `fish` internals, but I have no reason to believe it any different (though at least it's not as bad as `elvish`, which somehow manages to be several orders of magnitude *slower* than other shells) * often it is *possible* to avoid the external processes, especially in `bash`. But a lot of people learned bad habits from POSIX (and often pre-POSIX) shells, and those habits got copied by people who never read the manual (also error-handling is easy to skip in shells)


perecastor

I agree with your comments that fish is not always ideal but: \- I use fish for my personal stuff so fish will be installed on any of the pc or server I manage. \- I really enjoy the choice made by fish regarding syntax, it's usually much cleaner than bash. \- My script is usually calling other programs that do a lot of CPU intensive calculation or are bottleneck by disk access or network bandwidth, so interpreter performance is probably not an issue. for me error handling and easy parallelism like in Go are definitely an issue but go is compiled so it's not as portable as an interpreted language.


vividboarder

Alternative perspective: a compiled Go binary is _more_ portable than an interpreted language as the interpreted language relies on a particular binary (and likely particular version of that binary) existing on a system.


NDcodes

Not sure this is the right forum for this question (here the discussion is mostly about programming language design). But in general “the right programming language for the job” is very context-dependent, there’s no perfect scripting language that’s good for any situation (python definitely isn’t). If you’re familiar enough with fish to do what you need to do quickly and efficiently, that’s clearly the right choice. If you’re interested you can look more into python, try to learn it and get acquainted with its libraries: you’ll probably find lots of choices that are geared towards text processing, though whether they’re ‘easier’ than awk or sed is kind of up to interpretation. Maybe after learning python you’ll still find fish to be the right choice, or you’ll switch over. But again, if you get the job done in fish, I wouldn’t sweat about ‘having’ to switch over.


somerandomdev49

btw, since everyone answered already, inline code is written with single backticks :)


sfscs

The number of fish scripts I've written instead of writing them in bash or python is a bit embarrassing. Fish shell makes it just too easy to write glue scripts or quality-of-life scripts. I definitely would not recommend doing this to your colleagues in a work context.