Jon Rumsey

An online markdown blog and knowledge repository.

Project maintained by nojronatron Hosted on GitHub Pages — Theme by mattgraham

Linux and Terminal Learnings and Reference Page

This page was created to store random notes, throughts, and snippets regarding:

The primary resource for these notes was through reading Ryans Tutorials.

NOTE: I am no longer making any effort to verify any of these work in Windows PowerShell, as I have moved to a pure-Linux workstation.

Table of Contents

Linux Facts and Figures

Many of these concepts will also apply to Unix and other unix-based operating systems.

Many Terminal Options

The command line, or Shell, is the text-only I/O mechanism for humans telling Linux what to do, and for Linux to tell human operators what it is doing or has done (or nothing at all, just a prompt awaiting input).

On my system I am using Oh My ZSH! which is a bash knock-off with lots of interesting themes that make bashing-around a little easier (for example: using Git) and fun (colors, layouts, fonts, markers, etc).

Despite having an "add on shell" like ZSH, I can still "drop to a Bash" shell with the command bash (Bourne Again SHell).

I could also use sh but since ZSH is installed, it really just loads ZSH again.
If I was in a remote terminal session I could force-load ZSH with sh or zsh at the prompt and it would work.

In Windows, I stick with PowerShell because it has much greater capability than 'cmd.exe', and can be well-integrated into the OS and to apps like Visual Studio.

Everything Is A File

Linux Is Extensionless

Linux Is Case SENSITIVE

Linux is Space-Sensitive Too

Quote Around Spaces or get frustrated (your choice).

Linux Escape Characters

Use a backslash \ to escape (nullify) the special meaning of reserved characters like spaces e.g.:

user@bash: pwd
/home/toor
user@bash: cd My\ Music
user@bash: pwd
/home/toor/My Music

Linux MANuals

Invoke man command followed by the command name you want information about to get an on-screen instruction MANual.

Exit MAN pages by pressing q on the keyboard.

To search all MAN pages from Terminal: man -k <search_term>

To search terms WITHIN a MAN page display: / <search_term> then press n for Next page.

MANual Tricks

man clears the screen on exit. You can keep the output in the terminal scrollback using less. [Bellingham Codes participant: Remington]

For example, I want to see a section of MAN pages about SSH:

man -P "less -X" ssh

Screenshot of man command that keeps man pages in terminal scrollback output

Using SHELLS

To execute a script in a different directory: sh> (cd /dir/where/script/is/ ; sh script-name.sh [args])

Doing this will contain the script execution within the child SH process, and if the script forces closure at the end, you won't lose your existing SH session.

File Operations in Linux

List Files

Remember, EVERYTHING is a file so <pathspec> actually means filename or foldername or foldername/filename etc.

List files in current directory: ls [options] <filespec>

List files in a list with file details (long listing) within the '/etc/ directory: ls -l /etc

List hidden files: ls -a

There are MANY options, see the LS MAN pages for more.

Create and View Hidden Files

Just prefix the file or folder name with a dot:

Use ls -a to list hidden files in a directory.

Manipulating Files

Create a directory: mkdir <name>

Create a directory tree: mkdir <parent_name>/<child_name>

Remove files and empty directories: rm [options] <filespec>

Recursively remove files and directories: rm -r <filespec>

Remove an empty directory: rmdir <name>

Create a new blank file: touch <path>/<filename>[.ext]

Copy a file or directory: cp [options] <source_path> <dest_path>

Recurse over directories: cp -r <source_path> <dest_path>

Move/Rename files or directories: mv [options] <source_path> <dest_path>

You can move directories without using recuse option.

Note: Moving directories will rename if paths are the same except:

Bulk-copy files and rename them, but file extension: basename -s .jpg -a *.jpg | xargs -n1 -i cp {}.jpg {}_original.jpg

About Relative and Absolute Paths

PATHs can be absolute or relative.

Linux file strucuture is hierarchical.

Folders ARE files, but with a 'd' flag set, so folders and files are part of the path.

Absolute paths start with a forward slash / e.g.: /etch

Relative paths identify a location 'relative' to the pwd and will not begin with a forward slash i.e. ../project/code401 is one folder 'back' from pwd and one folder 'forward' from the 'project' folder.

Home path is ~ which is equivalent to \home\$username

Use Wildcards

Zero or more characters: *

Single character: ?

A range of characters: []

List files in the pwd whose 1st character is a 'Q' or 'q', 2nd character is within the range 0-9, 3rd character is within the range 0-3, and any number of other character following: ls [Qq][0-9][0-3]*

Not: ^

Note about Not: As the first character within a range wildcard causes the filter to eliminate files that match that first character range wildcard characters.

Remember: Wildcards affect the entire pathspec, which includes directories and filenames and file extensions.

Text Editing in Linux

Nano Text Editor

My favorite when in Bash/ZSH.

Simple, to the point, plain-text editor.

Search exists but is limited.

Vi Text Editor

A much more capable text editor.

Should be kept in mind for driving around text files in a native Linux viewer/editor.
All CLI, no UI.

Two modes:

  1. Insert/Input Mode: Enter data into a file: Tap i to change to INSERT mode and --INSERT-- will be displayed at bottom of screen.
  2. Edit Mode: Move around the file, add, delete, or copy data, search for data and replace data. Tap [esc] to change to EDIT mode.

Launch VI: vi <filename>

Command usage:

Commands starting with a colon require pressing [Enter] to execute them.

Save and Close (multple options):

Advice:

Display File Contents On Screen

Display all contents of a file: cat <filespec> Displays file contents.

View portion of a file: less <filespec> Use arrow keys to scroll up and down, b to go "back a page", and q to quit.

Use tail <filespec> to get a quick 'end-of-the-file' view. tail -n NUM to show a specific number of lines at the end of the file.

Grep is used to filter data for viewing: grep

Note: See more grep usage in Grepping

Use Piping and Redirection

Redirect Standard Out (STDOUT) to a file: >

Append STDOUT to the end of a file: >>

Redirect Standard Error Out (STDERR) to a file: 2>

Pass contents of a file to a program as Standard Input (STDIN): <

Feed STDOUT of the program on the left, as STDIN to the program on the right: |

When multiple piped commands all require 'sudo', add it to the commands at the start of each pipe (see example below).

# the following will fail due to lack of permissions
wget -O - https://packages.adoptium.net/artifactory/api/gpg/key/public | tee /usr/share/keyrings/adoptium.asc
# the following will succeed
sudo wget -O - https://packages.adoptium.net/artifactory/api/gpg/key/public | sudo tee /usr/share/keyrings/adoptium.asc

Chaining Commands with Semicolons

Use semicolons to chain commands together.
Essentially, it will run commands, in order, immediately after each previous command returns.

Does not conditionally manage any previous command's succees (or failure) code e.g. 0 (pass) or non-0 (fail).

See this StackOverflow question and its responses for more

Get All The Things

Grepping

Run grep with extended regex: egrep or grep -E

Ignore case: -i

Return NON MATCHING lines: -v

Select only whole-word matches: -w

Print count of matching lines: -c

Print name of each file containing the match: -l (normally used when grep is invoked with wildcards in file arg)

Print the number before each line that matches: -n

Recursive (all files in given pathspec): -r

BEFORE context and AFTER context: -B 4 -A 4 prints 4 lines prior-to and following the match.

Regex: Follow standard regex rules, including Multipliers and Anchors.

Note: Check out regularexpressions101 for an easy tool to test RegEx before you 'buy' results. :-)

Use Filters

When displaying or searching file contents, it can be helpful to filter the results to minimize clutter.

Show the first n lines: head -n number

Show the last n lines: tail -n number

Sort lines in a given way: sort

Word Count, including characters and lines: wc

Search for a given pattern: grep

Server Services

CUPS Printing System

Developed by Apple Inc. for macOS and *nix OSes.

Cannonical includes it with Ubuntu 18.x and above.

Check out the Web GUI for administration.

Check version and architecture: dpkg -l cups-browsed

View device info: lpstat -v

Note: At midnight the CUPS Logging mechanism 'rolls over' and a CUPS reset is initiated, which causes a popup notification every night at midnight. There is a way to work around this (basically stop forced CUPS daemon restart) but there is no harm in allowing the restart to be executed.

System Stuff

Permissions

Linux permissions place rules on what can be done with a file:

Read: r
Write: w
Exectute: x

Linux defines three groups that permissions can be applied to:

Show permissions with ls -l <pathspec>

Result example: -rwxr----x 1 pi owner 1.2K Jan 1 00:00 /home/pi/file.txt

Dissected, from left to right:

Change Permissions on a file (meaning everything): chmod [permissions] [filespec]

Interrogating Linux

Get version (long): uname

Get specific Linux kernel version and type: uname -sr

Environment Variables

Environment variables like PATH= are stored in /etc/environment, as well as .zshrc and .config files, often in the user profile area (but there could be others).

Update Bash Profile By Hand

.bashrc: Contains bash configuration and environment variables, command history, and prompt customization like colors and content.

To include the current Git branch in the prompt:

  1. Build a function with the the following code: git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/ (\1)/' (no return statement or semicolon necessary).
  2. Call the function within the PS1 definition line like: ...\[033[1;33m\]$(branch_function)\[....

Environment Variable Hierarchy

Loading variables in the current context becomes the 'parent context' set of variables.

If the parent (e.g. ZSH) launches another shell, its variables are then pushed to the child process.

To force specific variables into the child process, bash for example: zsh> VAR_ONE="Foo" VAR_TWO="BAR" sh would load child process 'sh' with those two additional enironemnt variables preceeding it.

Aliases

Configure aliases to common commands, scripts, and executables to make your Linux SH (ZSH etc) life simpler.

To see a list of existing aliases in shell: alias

ZSH Aliasing:

Add an alias for IntelliJ IDEA:

  1. Use find to locate file 'idea.sh' (or check JetBrains Toolbox for info on where it might be).
  2. Add the following line to .zshrc before final 'export' declarations: alias idea="~/.local/share/JetBrains/Toolbox/apps/IDEA-C/ch-0/nnn.mmmm.vv/bin/idea.sh
  3. Restart your shell.
  4. Traverse to the Java project directory and type: idea . and your project will open.

Add an alias for AndroidStudio:

  1. Locate file "studio.sh" probably off your home directory (or see JetBrains ToolBox if you installed Android Studio that way).
  2. Add the path to an alias and name it something like: androidstudio alias androidstudio=$STEP_1_PATH`
  3. Restart your shell.
  4. Traverse to the Android project directory and type: androidstudio . and your project will open.

IntelliJ Recommendation

Installing JetBrains Toolbox before installing Android Studio or IDEA and use that to install those (and any other IDEs), then follow these steps to manual set up idea . or studio . to launch from current directory:

  1. Launch JetBrains Toolbox
  2. Click on 'Toolbox App Menu'
  3. Click 'Settings'
  4. Expand the 'Tools' subsection
  5. Copy the user profile location listed in Shell scripts location
  6. Open PowerShell (sh or zsh for Linux) and you'll see command (script) files that launch IntelliJ apps
  7. Add an Alias for each command (script) and name it accordingly

The Alias for IDEA would look like:

Interrogate Files and Disk

Beyond creating, copying, moving, and deleting files, use these commands to manage the file system and disks.

Find the size of directories in pwd: du -sh ./*
Disk space utilization report: df -h
Find files modified within last 24 hours in a specified directory: find $directory -mtime -1

More About Finding Files

Which: Finds where a command lives which code WhereIs: Like 'which' but includes lib and man file locations: whereis code

Find

Search for files based on many properties: Modified, permissions, owners, symlinks, directory, filename, and rwx permissions.

find . -name 'my_file.txt' -type f # find 'my_file.txt' file starting from here
find /home/username/Downloads -name '*tar*' -type f # find files containing 'tar' in name starting in Downloads
find . -name "jdk" -type d # find jdk directory starting from here

Interrogating Hardware

List PCI Hardware/Driver info: lspci or lspci -k or lspci -nnk
Check LSB Module State: lsb_release -a
Report on hardware driver info: sudo lshw

Interrogate Network Interfaces

Get IP info from an interface: ip a
Get network driver info: inxi -n

View Internal Logs

Many system program logs are stored in /var/log
Device Message Logs: sudo dmesg
Messages in Logs related to network(s): sudo dmesg | grep iwl
Network Manager Log: sudo journalctl -b 0 -u NetworkManager

Manage Processes

Cancel a running operation: [CTRL] + C
Cancel a process by its ID: kill $process_id
Force canceling a 'stubborn' process: kill $process_id -9
Show list of running processes and IDs: ps
Put currently running process onto a background (paused) thread: [CTRL] + z
List current background processes: jobs
Move background process to foreground: fg $job_number
List ALL services, running AND not running: service --status-all
Use grep to find specific not/running: service --status-all | grep '\[+\]'
List services using systemctl: systemctl list-units
Check a specific service state using ps: ps -ef | grep postgres

Install Software

Config files are generally stored in /etc
Commonly used program binaries are stored in /bin
Other program binaries, perhaps related to users (rather than system) are stored in /usr/bin
Reinstall Ubuntu package 'linux-firmware': sudo apt-get install --reinstall linux-firmware
Apt is the primary software manager utility: apt get install $package
DPKG is the underlying package management system: dpkg -i $deb_package
As stated at AskUbuntu.com: dpkg => apt-get, aptitude => Synaptic, Software Center

Updating The Local Apt Repo

The Local Apt Repo is usually stored in /etc/apt/sources.list.d/pgdg.list
An example repo update will look something like this (for postgresQL):

sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'

Common steps to update Apt and Install from new sources:

  1. Update local apt repo with latest info from the source
  2. Optionally add an asc key
  3. 'sudo apt-get update'
  4. 'sudo apt-get -y install $package_name'

Manage Software Packages

Installed packages will probably have an entry in $PATH that should be interrogated.
Listing and Removing packages might leave behind $PATH entries.
List existing packages: dpkg -l $package_name_in_single_quotes
Remove a package: dpkg -r $package_name
Purge remaining package artifacts: dpkg -P $package_name

Specialized Tools

Yarn

Check out yarnpkg.com

Also Linuxize.com

Ensure you are in the project root.

Install globally: npm install --global yarn

Check version: yarn --version (currently 1.22.19)

New Project: yarn init [project-name]

Add dependency:

yarn add [package] > yarn add [package]@[version] > yarn add [package]@[tag]

Upgrade: yarn upgrade [package]

Remove dependency: yarn remove [package]

Install/upgrade all dependencies: yarn install

Image Converter

Help file usage template: convert [options ...] file [[options ...] file ...] [options ...] file

More information can be found at legacy.imagemagick.org/Usage/.

Convert Quality: convert $input_file_path.img -quality nn $output_file_path.img changes compression level on jpeg/miff/png.

Resize: convert -adaptive-resize 200x200 adjusts width and height using 'mesh interpolation' (limit to less than 50% larger or smaller otherwise use -resize).

Scale an image: convert -scale ${percentage}%. A simplified version of -resize.

Compress Image: convert -compress type

Add a Caption: Consider creating a label instead. See the Usage link above for details.

Vertical or horizontal flip: convert -flip or convert -flop, respectively.

Compress, Zip, Tar Files for Archiving

NOTE: This Section Is Under Development

Tar

Tempate (from tar --usage): `tar [options] -C DIR -f ARCHIVE [--exclude=PATTERN]

Template: tar -c [-f ARCHIVE] [OPTIONS] [File...]

Example to create a tar from target directory to named directory: tar -c directory.tar.gz -zvf file_list

EXCLUDE FILES while TARing: Insert --exclude=PATTERN before source directory.

RECURSION: Automatic. It is the default behavior of TAR. Use --norecursion to turn it off.

TAR References:

GZIP Manual at Gnu.org

Ubuntu Tar Reference

Primer: How Linux Works

This section contains notes from a Microsoft Reactor session about the very basics of Linux.

Host Gwyneth Pena-Siguenza, MSFT Cloud Developer Advocate.

Agenda:

High Level Components

Hardware: RAM, CPU, USB devices, etc. Kernel: In-memory; Assigns tasks to CPU; Manages hardware; Inteface between running programs and hardware. User Processes: All programs that are running and managed by the Kernel e.g. Email, Code, word processing, browser, etc. User Mode/Space: Access to only subset of memory and "safe CPU operations"; cannot crash entire system. Kernel Mode/Space: Unrestricted access to processor and RAM; Can crash entire system.

Main Memory

Storage area for bits and bytes. All running processes are made up of bits - instructions for the CPU. The state of every process resides in RAM.

State: A collection of bits in RAM that represent the current action of a running process.

CPU and Kernel

CPU is in charge of operating on memory.

The Kernel is the collection of non-tangible operations and state within the CPU and RAM.

Kernel Responsibility and Actions

Process Mangement:

Memory Management:

Device Management:

System Calls:

Summary:

User Space

Main Memory allocated by kernel for user processes.

Every process running lives in user space.

Organization:

Components use each other: Usually on the same level or below.

User Space talks to the Kernel: SysCalls used to comms between User Space and Kernel Space.

Virtualization and Containerization

Virtualization:

Containerization:

Users Space

Users: Entity associated with username and ID. Permissions and boundaries are applied to allow (or prevent) access to files, services, etc.

Superuser:

Memory Helper: MMU

MMU: Memory Management Unit.

File Hierarchy Standard

Root: '/' Dev: '/dev' Device files Etc: '/etc' Core system config directory, user passwords, boot, device, networking, and other setup files. Library: '/lib' Binaries: '/bin' System Bin: '/sbin' System-based binaries. Not available to non-superusers. System executables, boot files, low-level system repair/maintenance, etc). User: '/usr' Options: '/opt' Temp: '/tmp' Temporary files. Boot: '/boot' Contains boot loader and the kernel (loader). Binaries: '/bin' Executables and most Unix commands. Many others: Too many to list, and varies by Linux distribution.

Everything in Linux is seen and accessible as a file. This allows comon tools to manage files, data, processes, etc.

USR:

DEV:

Booting

  1. BIOS Loads and runs boot loader: '/boot'
  2. Boot Loader finds Kernel Image and loads it into main memory and starts it: '/boot'
  3. Kernal inits devices and drivers.
  4. Kernel mounts root filesystem: '/'
  5. Kernel starts 'init' program with process ID of 1 and the User Space starts: '/etc'

Editing how kernel boots will require low-level development skills.

User Space Startup

  1. Init kicked-off by Kernel boot.
  2. Essential low-level services (udevd and syslogd among others) are started.
  3. Network configuration is initialized.
  4. Mid- and high-level services (cron, printing, etc) are started.
  5. Login prompts, GUIs, and other high-level applications are loaded into memory.

Summary

Three primary components of Linux: User Processes <-> Kernel <-> Hardware

Check out notes from Microsoft Reactor session linux on azure.

References

A great deal of the basics were gleened from Ryan's Tutorials

Specific grep, filter, and some other commands were copied from Ryan's Tutorials' Linux Cheatsheet and Grep Cheatsheet and whenever possible, variables were changed to protect the innocent.

Apt and DPKG details were gleened from this AskUbuntu.com article

TechRepublic discusses adding an openGPG key, now that apt-key is deprecated.

ChrisJean.com for tips on 'which' and 'find' tools.

Introduction to Linux on Azure

Return to Root readme