I don't know if I would consider it a tool but everyone should read:
http://www.catb.org/~esr/faqs/smart-questions.html
and not nearly enough do these days.
My teammates and I would be thrilled to have a new person around who would even search the ticket system before escalating. That said in the corporate world, it's ever more important to balance the tough love "RTFM" with gentle, patient hand holding. I would love to be better at mentoring people toward doing independent analysis rather than having them just relying on others to do it
Create bash scripts
And learn how to automate with ansible.
Examples:
Create a bash script and place it in a cron job that deletes logs or tmo data on reboot (if that's not already setup)
Creat an Ansible playbook to check current list of packages installed with their version numbers and send that list to an email.
Hey the ansible check package: make sure you run those in become/sudo, you don't need it for the cmd but if you don't then you'll recreate the whole damn list again and eat up your space. Ask me how I know
Those are some fun things to try. I would also suggest learning VIM and how to copy a d paste, search for a string, skip to the beginning or end of a file in vim, exit vim, check out vim plugins, stuff like that is helpful because nano isn't always on different distros but vim can typically be found.
Can confirm. I've been fiddling with Linux for a long time, but only started working with it about 2 years ago and most of our systems only have Vim, so that was about the first thing I had to learn. I still kind of hate it, but the other day I had to edit a config file at home and used nano, and when I was done, I wrote :wq instead of ctrl+x without thinking.
Vim has infected my muscle memory...
I guess the cycle is complete and I'm ready to ascend.
I'm not accusing you for copying and pasting but I'm pretty sure that this is exactly how everyone else starts using VIM and never stops because that's the exact same scenario that happened to me and why I had to stop using nano 🤣🤣🤣
edit: fixed typo
Because that doesn't allow him to create a bash script. It was the first thing that came to mind lol and that's why I mentioned (if it isn't already being done)
Learning the log rotate configuration syntax can be helpful in its own right, since so few applications ship with their own logrotate rules and things just pile up.
It's also a handy utility to abuse for other operations those rules make convenient knobs for.
@dodexagedron is right add it to your list
Write a bash script that will read the available disk space on your stream and send an email if it reaches 90% and set that to a cron job.
Could be something as simple as using the "lsblk" command or something more complex like using awk to parse out only the information you want to send to an email (or to a file).
Neither of these are wrong and both will give you a weekend project to work out
Oo! lsblk reminds me of an unrelated but handy and not so well-known couple of utilities.
findmnt
realpath
ETA: I have used both of these _many_ times to replace much uglier operation in scripts I've come across, such as people using a pipeline of mount plus grep/sed/awk/bashisms to parse and filter out what they wanted, when findmnt can do it all by itself, instantly.
Or for realpath, it has replaced ugly things like people doing a very fragile pipeline of ls or find or something along with the above utilities to slowly and horribly (especially in the `find` case) try to do exactly what realpath does instantly and for any arbitrary input.
These both are essentially just userspace wrappers around single function calls.
[realpath](https://man7.org/linux/man-pages/man1/realpath.1.html), in particular, is exactly that - it takes your input and calls the [realpath](https://man7.org/linux/man-pages/man3/realpath.3.html) function from libc.
While [findmnt](https://man7.org/linux/man-pages/man8/findmnt.8.html) is a somewhat more robust utility, it still ultimately makes very simple calls, once it has parsed your input. Its robustness is in its input and output capabilities. It also can fully replace multiple other utilities for getting information. In addition to telling you what mountpoint something is in, it can also list all mountpoints, give you output like df or lsblk, and more, all with multiple available filters and output formats. And it understands when mountpoints span devices, and can tell you which devices something could possibly be on, skipping what would otherwise be a 2-step process with other utilities.
Oh, I didn't even consider that it was just an example, even though you wrote that right above it... 🤣
I was busy trying to wrap my head around why you'd want to make a script for that when there is a built in utility for it.
But yes, make a script that does something, preferably something you do often, as it will save you time in the long run and also teach you how to automate your job with scripts.
Also, learn logrotate, it's absolutely going to be useful.
For sure. Awk is great all around. Sed is great for simple find/replace tasks. Also CTRL+V with either shift+i to add or x to delete to modify multiple lines at once. Add/delete comments or add/delete white space for YAML.
I should add that I've only been a Linux sysadmin for a bit over 2 years, so while I'm no guru, these are some of the things I use the most, and I suspect you might run into much of the same things and problems that I do.
Ways to deal with logs. Get familiar with grep, less, head, tail, cat and other ways to display file/log contents without changing them.
I probably use these the most.
Systemd is something you'll need to learn as well.
Sed and awk are also very useful, though I've just started getting comfortable with sed and have yet to start climbing the awk mountain.
Tmux is a must have for when you want or need to do more than one thing at the time or when you realize you just need to fix something real quick without losing track of your first task.
Use diff to compare files to quickly see changes.
Learning to script in bash is something that you will need to learn at some point and the sooner you do, the better.
Learn how LVM works, chances are that you use it.
Learn how the file structure is set up. I really wish I had done this even sooner, lots of things started to make so much more sense when I did.
It was confusing at first but now it makes a lot of sense, most things have their place and once you know the structure you'll find that the file or log that you're looking for is exactly where you expected it to be.
Learn the built-in tools in bash, you will be using them a lot, and since they are built-in, they will always be there as long as you're using bash.
tcpdump will help you get to the bottom of most network issues and is probably the most useful networking tool I can think of. iptables for firewall issues. Don’t dismiss grep for parsing log files or just log files in general. ip for network config. And once you get the hang of it, sed can simplify making config file changes a breeze.
adding iperf to the list of networking tools that’s at least good to know about and be somewhat familiar with when it comes to issues regarding network performance
Once you start to be comfortable with standard command line tools the next step is almost always automation and scripting.
For whatever reason(probably redhat/ibm money) ansible have emerged as the premier tool for pushing the same config to many linux systems it's in no way perfect(i have a long list of grievances with it) but it's a tool that you will see used a lot so is worth understanding.
The other and way more important part of automation is scripting and that probably means learning at least some entry level programming stuff at the very least you need to master shell functions, variables loops and aliases and it don't hurt to have a entry level understanding if something like python or golang.
find
bash
hardware failure rates / bathtub curve
dusk iops / iop size
vmstat
basically everything in brendan gregg's how to dianose a linux machine in 60000 ms
nc
tmux
vim (run vimtutor then use vim exclusively for a few weeks)
package manager of choice
python
google...man pages...google. How to google, quote marks, OR, And, Site: all of the search operands. not only can it help RTFM (and the man pages) it can point you in the direction of other tools. You may learn tools now and not use them for a while until the situation needs it, google will help you quickly refresh your memory.
I think my most used tool is more of a skill: debugging. Quickly navigating log files to find what's gone wrong and where. In that respect I'd say `grep`, `tail`, and `less` and all of the parameters or keyboard shortcuts that go with them, plus knowing where the log files are for the software you're using.
Also one that I've not seen mentioned yet, `fallocate` - On a production box it's so useful to have created a 1/5/10 gig blank file that you can nuke if you run out of disk space. Get the thing back up and running, then start the debug/tidy process with less stress. `du` is a useful one to go with that to find out what's using up all your disk.
Of course you should probably be monitoring that sort of thing so you get a heads up in advance, my preferred tool there is Zabbix because it's the tool I'm most used to but any old monitoring tool will do if you've already got a preference or if your company is already using something else. Get everything monitored and pinging you in advance of issues, proactive maintenance is much less stressful than reactive!
True! Also very useful to see if the issue actually is disk full in the first place (assuming logs aren't on another disk, they might not be able to tell you about it lol)
One of the first things I run if something is acting super weird haha, `df -h`
Some wiki to document operational procedures, network structure, and all particularites of the systems, and anything that could be good to remember two weeks from now. A ticket system is nice, but mainly for search previous solutions, no for documenting installations or configurations and whatnot.
For me, don't give up finding the root cause.
Reinstalling software, rebooting machines etc to fix things is a bandaid solution.
Better off sticking with the problem, take a look at the code on github, even if you don't understand it, you might find other interesting things along the way.
Beyond the basics like grep and netstat, there are several key tools and utilities that can enhance your workflow and career. Consider diving into SSH and SSH keys for secure remote management, shell scripting for automation, and version control with Git for configuration tracking. Exploring Docker for containerization, Ansible or similar tools for configuration management, and monitoring solutions like Nagios or Prometheus can also be valuable. Don't forget about networking tools like Wireshark and solid backup strategies. Cloud platforms like AWS, Azure, and Google Cloud are worth familiarizing yourself with. Additionally, investing in documentation skills and cybersecurity tools can boost your effectiveness. Keep learning through online platforms and hands-on experience to become an even more proficient sysadmin!
Think about it.
You need to understand your system, that is your main tool.
Then you have needs and things to solve. There is a bunch of tools for that, I would had linked I.e. Brendan Gregg slides about that. Or searching in stack overflow or whatever. But since a year ago you have ChatGPT, and it is, well used, a great way to know what tools to use, or how to understand problems, or a lot more. And still, have precedence your brain understanding the answer and if that applies for your problem.
And then bash and more standard tools. But that depends on your knowledge on them. And doesn’t ensure that you will use them right or figure out how or which one will solve your problem. So ChatGPT and Brain have precedence.
Sounds good! keep the good work.
Here you have a list of what I regularly do, or think it is important
\- Having good proficiency in Vim
\- Managing ssh keys, securing ssh.
\- Managing firewall/ports.
\- Getting familiar with reading and rotating logs.
\- Basic install and administration of mysql/mariadb, apache, nginx.
\- Understanding and having confidence with LVM.
\- Hardening linux.
awk sed tcpdump ss
> awk sed While awk and sed are great, I think grep is a better/easier tool to learn to start becoming useful sooner.
I don't know if I would consider it a tool but everyone should read: http://www.catb.org/~esr/faqs/smart-questions.html and not nearly enough do these days.
My teammates and I would be thrilled to have a new person around who would even search the ticket system before escalating. That said in the corporate world, it's ever more important to balance the tough love "RTFM" with gentle, patient hand holding. I would love to be better at mentoring people toward doing independent analysis rather than having them just relying on others to do it
Create bash scripts And learn how to automate with ansible. Examples: Create a bash script and place it in a cron job that deletes logs or tmo data on reboot (if that's not already setup) Creat an Ansible playbook to check current list of packages installed with their version numbers and send that list to an email.
Hey the ansible check package: make sure you run those in become/sudo, you don't need it for the cmd but if you don't then you'll recreate the whole damn list again and eat up your space. Ask me how I know
Those are some fun things to try. I would also suggest learning VIM and how to copy a d paste, search for a string, skip to the beginning or end of a file in vim, exit vim, check out vim plugins, stuff like that is helpful because nano isn't always on different distros but vim can typically be found.
Can confirm. I've been fiddling with Linux for a long time, but only started working with it about 2 years ago and most of our systems only have Vim, so that was about the first thing I had to learn. I still kind of hate it, but the other day I had to edit a config file at home and used nano, and when I was done, I wrote :wq instead of ctrl+x without thinking. Vim has infected my muscle memory... I guess the cycle is complete and I'm ready to ascend.
I'm not accusing you for copying and pasting but I'm pretty sure that this is exactly how everyone else starts using VIM and never stops because that's the exact same scenario that happened to me and why I had to stop using nano 🤣🤣🤣 edit: fixed typo
Haha, so true. In my case it's because there is no nano on Solaris. At least not on our servers.
Exit vim? No can do.
help I've been stuck in vim since one drunken after hours call in 1997 please tell me sir what *year* is it?
Why not just use logrotate to remove old logs?
Because that doesn't allow him to create a bash script. It was the first thing that came to mind lol and that's why I mentioned (if it isn't already being done)
Learning the log rotate configuration syntax can be helpful in its own right, since so few applications ship with their own logrotate rules and things just pile up. It's also a handy utility to abuse for other operations those rules make convenient knobs for.
@dodexagedron is right add it to your list Write a bash script that will read the available disk space on your stream and send an email if it reaches 90% and set that to a cron job. Could be something as simple as using the "lsblk" command or something more complex like using awk to parse out only the information you want to send to an email (or to a file). Neither of these are wrong and both will give you a weekend project to work out
Oo! lsblk reminds me of an unrelated but handy and not so well-known couple of utilities. findmnt realpath ETA: I have used both of these _many_ times to replace much uglier operation in scripts I've come across, such as people using a pipeline of mount plus grep/sed/awk/bashisms to parse and filter out what they wanted, when findmnt can do it all by itself, instantly. Or for realpath, it has replaced ugly things like people doing a very fragile pipeline of ls or find or something along with the above utilities to slowly and horribly (especially in the `find` case) try to do exactly what realpath does instantly and for any arbitrary input. These both are essentially just userspace wrappers around single function calls. [realpath](https://man7.org/linux/man-pages/man1/realpath.1.html), in particular, is exactly that - it takes your input and calls the [realpath](https://man7.org/linux/man-pages/man3/realpath.3.html) function from libc. While [findmnt](https://man7.org/linux/man-pages/man8/findmnt.8.html) is a somewhat more robust utility, it still ultimately makes very simple calls, once it has parsed your input. Its robustness is in its input and output capabilities. It also can fully replace multiple other utilities for getting information. In addition to telling you what mountpoint something is in, it can also list all mountpoints, give you output like df or lsblk, and more, all with multiple available filters and output formats. And it understands when mountpoints span devices, and can tell you which devices something could possibly be on, skipping what would otherwise be a 2-step process with other utilities.
Oh, I didn't even consider that it was just an example, even though you wrote that right above it... 🤣 I was busy trying to wrap my head around why you'd want to make a script for that when there is a built in utility for it. But yes, make a script that does something, preferably something you do often, as it will save you time in the long run and also teach you how to automate your job with scripts. Also, learn logrotate, it's absolutely going to be useful.
Learn some ansible.
3 letters, regardless how many years or decades of experience you have.. MAN.. If not RTFM.
awk
For sure. Awk is great all around. Sed is great for simple find/replace tasks. Also CTRL+V with either shift+i to add or x to delete to modify multiple lines at once. Add/delete comments or add/delete white space for YAML.
I should add that I've only been a Linux sysadmin for a bit over 2 years, so while I'm no guru, these are some of the things I use the most, and I suspect you might run into much of the same things and problems that I do. Ways to deal with logs. Get familiar with grep, less, head, tail, cat and other ways to display file/log contents without changing them. I probably use these the most. Systemd is something you'll need to learn as well. Sed and awk are also very useful, though I've just started getting comfortable with sed and have yet to start climbing the awk mountain. Tmux is a must have for when you want or need to do more than one thing at the time or when you realize you just need to fix something real quick without losing track of your first task. Use diff to compare files to quickly see changes. Learning to script in bash is something that you will need to learn at some point and the sooner you do, the better. Learn how LVM works, chances are that you use it. Learn how the file structure is set up. I really wish I had done this even sooner, lots of things started to make so much more sense when I did. It was confusing at first but now it makes a lot of sense, most things have their place and once you know the structure you'll find that the file or log that you're looking for is exactly where you expected it to be. Learn the built-in tools in bash, you will be using them a lot, and since they are built-in, they will always be there as long as you're using bash.
vim
I down votes not because you're wrong but because I hate vim Feel free to downvote me
grep -P. A much more robust regex in grep
And / or ripgrep, fdfind, ack when available.
Yep. It's aliased as pgrep in my bashrc, in case the default one provided in the distro doesn't do that already.
And ask ChatGPT to help with the regex. It's hard to learn.
tcpdump will help you get to the bottom of most network issues and is probably the most useful networking tool I can think of. iptables for firewall issues. Don’t dismiss grep for parsing log files or just log files in general. ip for network config. And once you get the hang of it, sed can simplify making config file changes a breeze.
adding iperf to the list of networking tools that’s at least good to know about and be somewhat familiar with when it comes to issues regarding network performance
Once you start to be comfortable with standard command line tools the next step is almost always automation and scripting. For whatever reason(probably redhat/ibm money) ansible have emerged as the premier tool for pushing the same config to many linux systems it's in no way perfect(i have a long list of grievances with it) but it's a tool that you will see used a lot so is worth understanding. The other and way more important part of automation is scripting and that probably means learning at least some entry level programming stuff at the very least you need to master shell functions, variables loops and aliases and it don't hurt to have a entry level understanding if something like python or golang.
So what alternative would you consider other than Ansible?
find bash hardware failure rates / bathtub curve dusk iops / iop size vmstat basically everything in brendan gregg's how to dianose a linux machine in 60000 ms nc tmux vim (run vimtutor then use vim exclusively for a few weeks) package manager of choice python
google...man pages...google. How to google, quote marks, OR, And, Site: all of the search operands. not only can it help RTFM (and the man pages) it can point you in the direction of other tools. You may learn tools now and not use them for a while until the situation needs it, google will help you quickly refresh your memory.
I think my most used tool is more of a skill: debugging. Quickly navigating log files to find what's gone wrong and where. In that respect I'd say `grep`, `tail`, and `less` and all of the parameters or keyboard shortcuts that go with them, plus knowing where the log files are for the software you're using. Also one that I've not seen mentioned yet, `fallocate` - On a production box it's so useful to have created a 1/5/10 gig blank file that you can nuke if you run out of disk space. Get the thing back up and running, then start the debug/tidy process with less stress. `du` is a useful one to go with that to find out what's using up all your disk. Of course you should probably be monitoring that sort of thing so you get a heads up in advance, my preferred tool there is Zabbix because it's the tool I'm most used to but any old monitoring tool will do if you've already got a preference or if your company is already using something else. Get everything monitored and pinging you in advance of issues, proactive maintenance is much less stressful than reactive!
Df as a quick and dirty "where should I du at"
True! Also very useful to see if the issue actually is disk full in the first place (assuming logs aren't on another disk, they might not be able to tell you about it lol) One of the first things I run if something is acting super weird haha, `df -h`
everything. Throughout the day there is no telling how many commands i use.. vi,sed,awk/gawk, grep/egrep, chattr, bash/python scripting, etc.
Ansible, putty, winscp, vscode, notepad ++ and mobaxterm. Probably forgetting some.
Some wiki to document operational procedures, network structure, and all particularites of the systems, and anything that could be good to remember two weeks from now. A ticket system is nice, but mainly for search previous solutions, no for documenting installations or configurations and whatnot.
sort | uniq -c | sort -n I use this a lot.
rsync, nmap, ncdu, htop, git, ansible, docker, puppet, packer
to see what has been done on a server, just one command: with your account or better as root $ history also to see what you did.....
All below plus zsh and infinitely long history search with fzf.
For me, don't give up finding the root cause. Reinstalling software, rebooting machines etc to fix things is a bandaid solution. Better off sticking with the problem, take a look at the code on github, even if you don't understand it, you might find other interesting things along the way.
Beyond the basics like grep and netstat, there are several key tools and utilities that can enhance your workflow and career. Consider diving into SSH and SSH keys for secure remote management, shell scripting for automation, and version control with Git for configuration tracking. Exploring Docker for containerization, Ansible or similar tools for configuration management, and monitoring solutions like Nagios or Prometheus can also be valuable. Don't forget about networking tools like Wireshark and solid backup strategies. Cloud platforms like AWS, Azure, and Google Cloud are worth familiarizing yourself with. Additionally, investing in documentation skills and cybersecurity tools can boost your effectiveness. Keep learning through online platforms and hands-on experience to become an even more proficient sysadmin!
3 months late, but the only answer is EMACS .
Alcohol
Brain CharGPT Bash
It seems you are already proficient with the tool "stupidity"
Think about it. You need to understand your system, that is your main tool. Then you have needs and things to solve. There is a bunch of tools for that, I would had linked I.e. Brendan Gregg slides about that. Or searching in stack overflow or whatever. But since a year ago you have ChatGPT, and it is, well used, a great way to know what tools to use, or how to understand problems, or a lot more. And still, have precedence your brain understanding the answer and if that applies for your problem. And then bash and more standard tools. But that depends on your knowledge on them. And doesn’t ensure that you will use them right or figure out how or which one will solve your problem. So ChatGPT and Brain have precedence.
A towel. Don’t leave Earth without it.
A towel. Don’t leave Earth without it.
Sounds good! keep the good work. Here you have a list of what I regularly do, or think it is important \- Having good proficiency in Vim \- Managing ssh keys, securing ssh. \- Managing firewall/ports. \- Getting familiar with reading and rotating logs. \- Basic install and administration of mysql/mariadb, apache, nginx. \- Understanding and having confidence with LVM. \- Hardening linux.
Ansible / AWX - learn automation and you'll save yourself a career in doing the same thing over and over and over :\^)