Get Power From Command Line

The Command Line Skills You Missed

Mosky
Financial Freedom
Power Freedom
Mosky
Architect, Pinkoi

• Data Science With Python

• Statistical Regression With

Python

• Hypothesis Testing With Python

• Practicing Python 3

• The mediums.

• The PyCons, etc.

Outline

1. Introduction to command line

2. Text processing: e.g., CSV, JSON, parallelizing.

3. System management: e.g., les, processes, monitoring.

4. Work on a remote machine: e.g., SSH, SSH tunneling.

5. Developing on CLI
fi
What commands?

• POSIX
• Ubuntu 22.04 LTS Server

• Portable Operating System Interface

• curl, gawk, git, htop, less,
lsof, man db, rsync, strace,
• IEEE Std 1003.1-2008

tmux, vim, ssh, etc.

• List of Unix commands – Wikipedia

• The de facto standard

• tree, ripgrep, jq, etc.

• The modern alternatives

• bat (cat), fd f nd ( nd), etc.

-
-
i
fi
How to use this deck?

• Practice makes perfect:

• Try every single command in this deck, and 


downloading the deck as a PDF may make it easier.

• Be careful of the spaces and newlines after paste.

• Visit the links, but don't read; scan them to index in your brain.
Introduction to command line
“Command Notes”

• Memorize all the commands?

No.

• You may try to note them, and

make them searchable.

• Inputs & Outputs.

• Tags.

• Like →
Get your terminal emulator

• iTerm for macOS

• See also: Mosky’s Preferences

in macOS § iTerm and search
“Bash 5” in the doc!

• Windows Terminal
• GNOME Terminal / Konsole
Connect to ...

• macOS with GNU/Linux commands

• Install the GNU/Linux commands on macOS

• WSL

• Windows Subsystem for Linux

• Install Linux on Windows with WSL

• Ubuntu

• Or,
 The container

• Get Docker

• $ docker pull ubuntu:22.04

$ docker network create ubuntu n
$ docker run dit p 2222 22 p
8000 8000 network ubuntu n
name ubuntu c ubuntu:22.04
$ docker exec it ubuntu c bash

• ubuntu c# yes | unminimize &&

apt install curl gawk git htop
less lsof man db rsync strace
tmux vim ssh y
:
-
-
-
-
-
-
-
-
-
-
:


-


-
-
-
-



 The container

• Get Docker

$ hints as a normal • $ docker pull ubuntu:22.04

user. $ docker network create ubuntu n
$ docker They
runaredit
defaultpin 2222 22 p
the Ubuntu Server,
8000 8000 network ubuntu n
but not in the
name ubuntu c ubuntu:22.04
container image.
$ docker exec it ubuntu c bash
ubuntu c# hints
as a superuser and • ubuntu c# yes | unminimize &&
in the Ubuntu apt install curl gawk git htop
container.
less lsof man db rsync strace
tmux vim ssh y
:
-
-
-
-
-
-
-
-
-
-
-
:


-


-
-
-
-



Checkpoint: Log into the container

• You shall see →

• Hints:

• $ docker stop ubuntu c;

docker rm ubuntu c # to
start over.

• $ docker start ubuntu c

$ docker exec it ubuntu c
bash # after reboot if any.

-
-
-
-


-

Read the manual

• $ man man • $ type man

# Try `\ k` <n> <u>
# and <h> <q> `\ k` • $ type ls

• $ man k ls • $ type for

• $ ls help | less # <q> to exit. • Google: “man yes”

• $ ls help | grep ' l' • explainshell.com

• $ help for • Bash Reference Manual

-
-
-
-
-

-


-

Read the manual

• $ man man • $ type man

# Try `\ k` <n> <u>
# and <h> <q> `\ k` • $ type ls

• $ man k ls • $ type for

• $ ls help | less # <q> to exit. • Google: “man yes”

• $ ls help | grep ' l' • explainshell.com

• $ help for • Bash Reference Manual

Two spaces.
-
-
-
-
-

-


-

Edit commands like a ninja

• Bash Keystrokes – Mosky’s Notes

• See also:

• $ ec<Tab>Hello, World! • “CTRL-p CTRL-q” to detach –

Docker Docs

• $ <Up><C-w>Ctrl-W!
• macOS Shortcuts – Mosky’s
• $ <Up><M-C-h>H! Notes

• $ <Up><C-h>? • Windows Shortcuts – Mosky’s

Notes
• Also works in Python shell and
many other places.

Edit commands like a ninja

• Bash Keystrokes
Or <C-p>, if not
from the docker.
– Mosky’s Notes
• See also:

• $ ec<Tab>Hello, World! • “CTRL-p CTRL-q” to detach –

Docker Docs

• $ <Up><C-w>Ctrl-W! “Space” vs.

Or <M-Rubout>.
The <Rubout> is
• macOS
“Alphanumeric” Shortcuts – Mosky’s
• $ <Up><M-C-h>H! Notes

<Delete> or
• $ <Up><C-h>? <Backspace>.
• Windows Shortcuts – Mosky’s
The <M> is Notes
• Also works in
<Opt> or <Alt>.
Python shell and
many other places.

Checkpoint: “Hey, Ctrl-K!”

• $ echo Hello, World!

• $ <Up><your keystrokes>
Hey, Ctrl-K!

• Hints:

• <C-a>, <M-f>, <C-k>



Quoting

• $ echo Hello, World! # 2 arguments.

• $ echo 'Hello, World!' # 1 argument.

• $ echo Hello, $HOSTNAME!

• $ echo "Hello, $HOSTNAME!"

• $ echo "Hello, \$HOSTNAME!"

• $ echo 'Hello, $'HOSTNAME!

• $ echo $'Hello,\nWorld' # ANSI-C Quoting

Pipe, AND, OR

• $ echo 'Hello, World!' | grep Hello

• $ echo 'Hello, World!' | grep Hello && echo Found!

• $ echo 'Hello, World!' | grep Hey && echo Found!

• $ echo 'Hello, World!' | grep Hey || echo Not found.

• $ echo 'Hello, World!' | grep Hello && echo Found! || echo Not found.

• $ echo 'Hello, World!' | grep Hey && echo Found! || echo Not found.

Checkpoint: Explain the echo by yourself

• explainshell.com

• $ echo 'Hello, World!' | grep Hello && echo Found! || echo Not found.
Loops

• $ cd ~ # cd: change directory

• ~$ touch a.txt b.txt c.txt

• $ for name in *.txt; do echo mv $name ${name/.txt/.csv}; done

# It's a dry run.

• $ for name in *.txt; do echo mv $name ${name/.txt/.csv}; done

• $ while true; do date; docker container top ubuntu c; echo; sleep 3; done

• $ until docker start ubuntu c; do sleep 3; done

 Redirections
le descriptor:name = 0:stdin, 1:stdout, 2:stderr

• $ cd ~

• ~$ echo Hello, File! >hello.txt

• $ cat hello.txt

• $ cat hello.txt | grep Hello

• $ grep <hello.txt Hello

• $ grep <hello.txt Hello >out.txt

• $ grep <hello.txt Hello >>out.txt # appends to out.txt.

fi

• $ grep <hello.txt bad Hello >out.txt # still prints errors.

• $ grep <hello.txt bad Hello >out.txt 2>err.txt

• $ grep <hello.txt bad Hello >out.txt 2>&1

• $ grep <hello.txt bad Hello &>both.txt

• $ grep <hello.txt bad Hello 2>/dev/null

-
-
-
-
-
-
-
-
-
-

• $ cat <<HEREDOC | grep paste

> Hello, Here Doc!
> It's useful to paste something.
> HEREDOC

• $ cat <<<'Hello, Here String!'

• $ python3 <<<'print(1234 1234)'

# C's unsigned long is 0 to 18446744073709551615, btw.
# Useful!



*
*









Expansions & Substitutions

• $ echo x.{txt,csv} • $ echo a.*

• $ echo ~ • $ echo x.* # x.*

• $ echo ~root # or any username. • $ echo ?.*

• $ cd ~ • $ echo [abcx].txt

• ~$ touch a.{txt,csv} • $ echo [abcx].csv

-
>

• $ echo $name • See also:

• $ echo ${name} • Bash scripting cheatsheet §

Parameter expansions

• $ echo ${name/.txt/.csv}
• Shell Style Guide § Variable
• $ echo `echo 'Hello, Echo!'` expansion
• $ echo $(echo 'Hello, Echo!')

• $ cat <(echo 'Hello, Cat!')

 Environments & Parameters

• Bash scans its own 
 • $ env | grep A= # <empty>

environment variables and creates
shell parameters.
• $ export A

• $ echo $PATH • $ # export A=20

• $ A=1 env | grep A= # A=1 • $ env | grep A= # A=2

• $ echo $A # A is empty. • $ A=3 echo $A # 2

• $ A=2 •$ # $ A=3 echo 2

• $ echo $A # 2 • $ export n A && env | grep A=

=

=
=
-

-
>

-
>
-
-
>
>

-
>

Job Control

• $ vi a.txt # and <C-z>. • $ sleep 3 && echo Hi! & #

• $ vi b.txt # and <C-z>. • $ bash c 'sleep 3 && echo Hi!'

# and <C-z>.
• $ jobs
• $ bg
• $ fg # <CTRL-G> shows f lename.

• $ fg - # like Cmd-Tab or Alt-Tab.

• $ fg 1 # `:q` to exit vim.

• $ kill %2

i
=

=
=

 Aliases

• $ alias ll • $ alias 'cd '

• $ type ll • $ cd /usr/bin

• $ ls help | grep ' -[alF]' • usr: User System Resources

• $ alias hi=history •$

• $ hi 10
.
.
-
-

.
.

History

• <Up>, <Down>; <C-p>, <C-n>

• $ history 10

• $ history a
• <C-r>, <C-s>: search history.

• $ history n
• $ <a long long command> #TAG
• $ history help | grep ' -[an]'
• $ <C-r>#TAG

• $ stty ixon # <C-s> needs it.

• <C-c>: discard the editing.

-
-
-
-

Checkpoint: Hack the which

• $ which echo
/usr/bin/echo

• $ ??? which echo

bash: which: No such f le or directory

• Hints:

• which searches the PATH for the executable le.






i

fi
Text processing
 TSV
Create TSV le using cat

• $ cd

• ~$ cat <<EOS >cmds.tsv

> <Open https: en.wikipedia.org/wiki/List_of_Unix_commands,
copy the table, and paste here.>
> EOS

fi
/
/






 TSV
Create TSV le using vim

• $ cd

• ~$ vi cmds.tsv

• (vim) i # goes the insert mode.

• (vim) <Open https: en.wikipedia.org/wiki/List_of_Unix_commands,

copy the table, and paste here.>

• (vim) <Esc> # backs to the normal mode.

• (vim) :x<CR> # goes the command line mode, write and quit.

fi

/
/



• If any garbled text:

• ubuntu c$ LANG=C.UTF-8 bash

• or,

• (host)$ docker exec it ubuntu c env LANG=C.UTF-8 bash

-
-

-
 • $ wc l cmds.tsv # wc: word count • $ grep <cmds.tsv cat
# grep cat cmds.tsv
• $ cat cmds.tsv
• grep: g/re/p in the ed
• $ head cmds.tsv command, globally search for a
• $ tail cmds.tsv regular expression and print
matching lines.

• $ grep <cmds.tsv '^.cat'

• $ grep <cmds.tsv '^.cat' o

=
-
=
=

TSV is literally everywhere.

Checkpoint: Copy the head and paste to a sheet

• A sheet in Google Sheets, Excel,

LibreO ce Calc, like →

• Hints:

• Just like how did you paste the

table from the browser to the
terminal.

• Copy the output on the

terminal and paste it to the
sheet.
ffi
TSV (cont.)

• $ seq 1 3

• $ seq 1 3 | column

• $ seq 1 3 | column | hexdump c

• $ seq 1 3 | column t

-
-

 • $ head <cmds.tsv | column s '<C-v><TAB>' t

• $ column help | grep ' -[st]'

• $ head <cmds.tsv -1
# $ head n 1 cmds.tsv

• $ head <cmds.tsv -1 | sed 's/(Option code)//'

• sed: stream editor

• $ sed <cmds.tsv n e 1p e /^.cat/p

• $ sed help | grep ' -[ne]'

=
=
=
-
-
-
-
-
-
-


-
-

• $ cut <cmds.tsv f 2 | head

• $ cut <cmds.tsv f 2 | tail +2 | head

• $ cut <cmds.tsv f 2 | tail +2 | sort

• $ cut <cmds.tsv f 2 | tail +2 | sort | uniq

• $ cut <cmds.tsv f 2 | tail +2 | sort | uniq c

• $ cut <cmds.tsv f 2 | tail +2 | sort | uniq c | sort n

-
-
-
-
-
-

-
-

-
• $ awk <cmds.tsv -F $'\t' '{ print $2 }' | head

• $ awk <cmds.tsv -F $'\t' '$5 ~ /BSD/' | column s $'\t' t

• $ awk <cmds.tsv -F $'\t' '$5 ~ /BSD/ { counts[$2]++ } END { for (cat in

counts) print counts[cat] "\t" cat }'

• $ awk <cmds.tsv -F $'\t' '$5 ~ /BSD/ { counts[$2]++ } END { for (cat in

counts) print counts[cat] "\t" cat }' | sort n

-
-

• See also:

• The GNU Awk User’s Guide

• 6.3.2.2 Comparison Operators

• 6.1.1.3 Regular Expression Constants

• 8.1.5 Scanning All Elements of an Array

 Checkpoint:
How many mandatory text processing does POSIX have?

• $ ??? <cmds.tsv ??? | wc l

???

• Hints:

• Either grep or awk is a good choice.

• Try and if you choose awk.

• “Text processing”

• “Mandatory”

=
=
&
&
-


TSV (cont.)

• $ cut <cmds.tsv f 2 | head

• $ awk <cmds.tsv '{ print $2 }' | head

• $ vimdiff <(cut <cmds.tsv f 2 | head) <(awk <cmds.tsv '{ print $2 }' |

head)
# `:qa` to exit.

• $ awk <cmds.tsv -F $'\t' '{ print $2 }' | head



-

• $ cut <cmds.tsv f 2 | head >a

• $ awk <cmds.tsv '{ print $2 }' | head >b

• $ vimdiff a b

• $ diff a b

• $ diff a b y # using the side by side format.

• $ diff a b u # using the unif ed format.

• $ diff help | grep ' -[yu]'

-
-

-
-

• $ sudo apt install bat y # or,

• # apt install bat y

• $ alias bat=batcat

• $ diff a b | bat # <q> to exit.

-

Interact with the system clipboards

• (macOS)$ pbpaste | tail -1 | pbcopy

• The way I make slides with highlighted codes:

• $ pbpaste | highlight u utf-8 -S python s solarized light k Monaco

-K 28 -O rtf | pbcopy # and then paste into a slide.

• (X11)$ xclip o | tail -1 | xclip

• (WSL)$ powershell.exe Get-Clipboard | tail -1 | clip.exe

• See also: vor0nwe/clip – Gist for a neat clip script in WSL.

-
-

-
-

-
cURL

• $ curl https://google.com

• $ curl v https://google.com

• $ curl -L https://google.com
-

 JSON with jq

• $ sudo apt install jq y # or,

• # apt install jq y

• jq is like sed for JSON.

• $ cd

• ~$ curl https://raw.githubusercontent.com/5etools mirror-1/5etools mirror-1.github.io/

master/data/bestiary/bestiary mm.json >monsters.json # or,

• ~$ curl https://raw.githubusercontent.com/hazmole/TheGiddyLimit.github.io/master/
data/bestiary/bestiary mm.json >monsters_zh_tw.json

• $ wc l monsters.json # 70811

-
-

-
-
-
>
-

-
• $ jq <monsters.json

• $ jq <monsters.json | less

• $ jq <monsters.json -C | less -R

• $ grep <monsters.json dragon color=always | less -R

• $ jq <monsters.json | bat l json

• $ bat monsters.json

• $ vi monsters.json # `:q` to exit.

-
-
-

• $ jq <monsters.json '.monster[] | .name' | head

• $ jq <monsters.json '.monster[] | select(.type == "dragon") | .name' | head

• $ jq <monsters.json '.monster[] | select(.type == "dragon") |

[.hp.average, .name, .type, .page]' | head

 • $ jq <monsters.json '.monster[] | select(.type == "dragon") |

[.hp.average, .name, .type, .page] | @tsv' | head

• $ jq <monsters.json '.monster[] | select(.type == "dragon") |

[.hp.average, .name, .type, .page] | @tsv' r | head

• $ jq <monsters.json '["HP", "Name", "Type", "Page"], (.monster[] |

select(.type == "dragon") | [.hp.average, .name, .type, .page]) | @tsv' r |
head

• $ jq <monsters.json '["HP", "Name", "Type", "Page"], (.monster[] |

select(.type == "dragon") | [.hp.average, .name, .type, .page]) | @tsv' r
>dragons.tsv

-
-
• $ sort <dragons.tsv # Sort by HP.

• $ sort <dragons.tsv k 2 debug | head # Try to sort by Name.

• $ sort <dragons.tsv k 2,2 debug | head

• $ sort <dragons.tsv t $'\t' k 2,2 debug | head

-
-
-
-
-
-
-
-

-
-

 Checkpoint:
What is the first undead monster in the book?
• $ jq ??? | tail -1
???

• Hints:

• “undead”

• “page”



JSON with gron

• $ sudo apt install gron y # or,

• # apt install gron y

• $ jq <monsters.json '.monster[] | select(.type == "dragon") | .name' | head

• $ jq <monsters.json '[ .monster[] | select(.type == "dragon") ]' >dragons.json

• $ bat dragons.json

• $ gron dragons.json

• $ gron dragons.json | grep -E 'json\[[0-9]+\]\.name ='

• $ gron dragons.json | grep -E 'json\[[0-9]+\]\.name =' | gron u

 Fixed string, BRE, ERE, vs. glob

. * [a z] ^$ ? + |

Fixed string . * [a z] ^$ ? + | $ grep -F

$ grep
BRE . * [a z] ^$ ? + |
$ grep -G

ERE . * [a z] ^$ ? + | $ grep -E

Glob . * [a z] ^$ ? + |
-
-
-
-
-


 Test the regex types

• $ echo 'i++' | grep -F '.+' o # f xed string (not regex)

<empty>

• $ echo 'i++' | grep -G '.+' o # basic regular expression (BRE)

i

• $ echo 'i++' | grep -E '.+' o # extended regular expression (ERE)

i
+
+
+

-
-
-
i






• See also:

• Regex cheatsheet for BRE, ERE, Vim, Python's re, PCRE (Perl).

• regex101: a regex debugger for Python, JavaScript, Java, Golang, PHP.

• jq Manual

• tomnomnom/gron – GitHub

• yq for YAML.
Checkpoint: The regex type of less may be?

• BRE-like ... ERE-like ... or not regex?

• Hints:

• How to put i into less?

• How to search something in less?

+
+
Bulk editing

• $ cp dragons.tsv cats.tsv

• $ cp dragons.tsv cats_2.tsv

• $ head cats .tsv

• $ sed s/Dragon/Cat/ cats .tsv | less

• $ sed e s/Dragon/Cat/ e s/dragon/cat/ cats .tsv | less

• $ sed e s/Dragon/Cat/g e s/dragon/cat/g cats .tsv | less

• $ sed e s/Dragon/Cat/g e s/dragon/cat/g cats .tsv i

-
-
-
*

-
*
-
-

*
*
*
-

CSV with CSVKit

• $ sudo apt install csvkit y # or,

• # apt install csvkit y

• $ jq <monsters.json '["HP", "Name", "Type", "Page"], (.monster[] |

select(.type == "dragon") | [.hp.average, .name, .type, .page]) | @tsv' r
>dragons.tsv

• $ jq <monsters.json '["HP", "Name", "Type", "Page"], (.monster[] |

select(.type == "dragon") | [.hp.average, .name, .type, .page]) | @csv' r
>dragons.csv

• $ bat dragons.csv

-
-
• $ csvcut <dragons.csv c Name,HP

• $ csvsort <dragons.csv c HP,Page

• $ csvstat <dragons.csv c HP,Name,Type

• $ csvlook <dragons.csv # a Markdown table

• $ csvgrep <dragons.csv c Name m 'Adult Black Dragon'

• $ csvsql <dragons.csv query "select avg(HP) from stdin where Name like
'Adult%'"
-
-
-
-
-
-
-
>
-

• See also:

• Tutorial – csvkit

• 1.4. in2csv: the Excel killer

• 3.3. csvsql and sql2csv: ultimate power

Parallelizing

• $ time sleep 1

• $ seq 3 | xargs i echo sleep {} # xargs: extended arguments

• $ seq 3 | xargs i echo sleep 1

• $ time seq 3 | xargs i sleep 1 # takes 3 seconds.

• $ time seq 3 | xargs i -P 3 sleep 1 # takes 1 second.

• $ man xargs | less +'/ -[iI]'

-
-
-
-

System management
 File management

• $ cd • $ ls

• ~$ mkdir mydir && cd mydir • $ ll

• ~/mydir$ touch a b .h • $ type ll

• $ mkdir p x/y/z • $ ll t
# List and sort by time.
• $ cp b c
• $ ll -S h
• $ rm b # List and sort by size.






• $ sudo apt install tree y # or, • $ tree

• # apt install tree y • $ tree a # List all f les.

• $ cd ~/mydir • $ tree -L 1 # List one level deep.

• ~/mydir$ ls • $ tree f # Print full paths.

-
-

-
i

 • $ cd

• ~$ ln s mydir mydir_link # Make a symbolic link.

• $ ls mydir_link

• $ ll mydir_link

• $ cp r mydir mydir_backup # Copy recursively.

• $ ll mydir_backup

-
-

 • $ tar czf mydir.tar.gz mydir • $ du h d 1 # du: disk usage

• $ rmdir mydir • $ df h # df: disk free

# failed not empty.

• $ rmdir mydir/x/y/z
# OK!

• $ rm r mydir/

• $ tar xzf mydir.tar.gz

• $ tree
-
>
-
-
-

-
.



.
.



Checkpoint:
Describe one difference between cp r and cp a

• Read and search the manual!

• Hints:

• $ man cp # `/` to search; `n` to next; `h` to get help.

• $ cp help | grep
-
-
.
.
.
-

-
Terminology

EN program process thread concurrency parallelism

TW 程式 程序 執⾏緒 並⾏性 平⾏性

CN 程序 进程 线程 并发性 并⾏性
Process management

• $ ps # ps: process status

• $ ps a # a: includes other users.

• $ ps ax # x: includes other ttys or no tty's.

# tty: teletypewriter

• $ ps axu # u: user oriented format

• $ ps f # f: "forest"

• $ man ps | grep -E ' (VSZ|RSS|TTY|STAT|START|TIME) '

-



• $ vi test.txt # and <C-z>.

• $ ps u | grep vi

• $ ps u | grep [v]i

• $ ps u | sed n e 1p e /[v]i/p

• $ ps axu | grep [v]i |

awk '{ rss += $6 } END { printf "%.4g MB\n", rss/1024 }'

• (macOS)$ ps axu | grep [C]hrome |

awk '{ rss += $6 } END { printf "%.4g MB\n", rss/1024 }'
-

-



 • $ vi test.txt # and <C-z>. • $ # kill <pid>

• $ pgrep v # <pid> • $ pkill f test # -SIGTERM

• $ pgrep vi # <pid> • $ fg

• $ pgrep test # <empty> • $ vi test.txt # and <C-z>.

• $ pgrep f test # <pid> • $ pkill f test -SIGKILL

• $ pgrep f test l • See also:

• $ pgrep help | grep ' -[fl]' • Signal (IPC) – Wikipedia

-
-
-
-
-
-
-
>
-

>
-
-
>

-
>

 Monitoring

• $ sudo apt install iotop iftop y # or, # if: interface

• # apt install iotop iftop y

• $ free h

• $ man free | grep -E ' (total|used|free|shared|buff/cache|available)' -A 1

• $ vi test.txt # and <C-z>.

• $ top

• $ htop

• Select a process, and <F9> (kill)

 • $ cd • Open another terminal:

• ~$ python3 m http.server &> my.log • # iftop # <q> to exit.

# <C-c> to interrupt.
• $ tail f ~/my.log
• Open http://localhost:8000/.

• $ less +F ~/my.log
# Try <C-c> ` i` `/http` <F>.

• $ watch tail -10 ~/my.log

-
-
-



• The commands which are useful but not relevant to containers:

• $ sudo iotop

• Important system logs:

• $ sudo vi /var/log/syslog

• $ sudo vi /var/log/auth.log

Checkpoint: Monitor your HTTP server by less

• Try to get three new logs.

• Hints:

• Click on your browser!

Monitoring (cont.)

• $ python3 c 'print(1234)' • $ lsof p `pgrep bash t pts/1`

# lsof: list open f les
• $ strace python3 c 'print(1234)'
• $ lsof /usr/bin/bash
• $ sudo strace p <pid>
• $ man lsof | less i +/^examples
• $ ps x
• $ htop
• $ pgrep bash
• Select a process, and:

• $ tty

• $ pgrep bash t pts/1

• <s> (strace)

# pts: pseudo terminal slave

• <l> (lsof)

-
-

-
-
-
-

-





 • $ sudo apt install sysstat y # or, • $ man sar | grep 'Report ' -B 1

• # apt install sysstat y • $ sar f /var/log/sysstat/sa10

s 12 00 e 15 00 r human
• $ sar 1 # Report memory stats.
# Report CPU stats per second.
• See also:

• $ sar
# Report today's CPU stats. • sysstat/sysstat – GitHub

• $ sar f /var/log/sysstat/sa10 • Amazon CloudWatch

• $ sar f /var/log/sysstat/sa10 • Grafana

s 12 00 e 15 00
-
-
-
-
-


:
:


-
-
:
:

-
-
-

Container management

• OCI: Open Container Initiative

• Docker le: part of OCI spec.

• image: container image

• Why?
• Once you pack your app as a
container image, it can be run
anywhere seamlessly.
fi
 • (host)$ docker container commit ubuntu c ubuntu cli power
# $ docker container commit

• $ docker image ls
# $ docker images
=
=
=
=
=
=


-
.
.
.

-
-


 • $ docker image save ubuntu cli power >ubuntu cli power.tar

• $ docker image rm ubuntu cli power

# $ docker rmi ubuntu cli power

• $ docker image ls

• $ docker image load <ubuntu cli power.tar

• $ # docker image (save|load) docker image (save|load)

• $ docker image ls
=
=
=

-
-
-
-
-
=
-
=
-
-
=



-
-

• $ docker container create name ls expert ubuntu cli power man ls

• $ docker container ls a # $ docker ps a

• $ docker container start ls expert

• $ docker container ls a # Exited (0) 4 seconds ago

• $ docker container start a ls expert # Start and attach.

• $ docker container rm ls expert

• $ # docker container (create|start|stop|rm)

docker container (create|start|stop|rm)
-
-
-
-
-
-
-
=
-
=
>
=
-

-
-
=
=

=


-
-

 • $ docker container cp ubuntu c:/root/cmds.tsv ~/cmds.tsv

# $ docker cp

• $ docker container cp ~/cmds.tsv ubuntu c:/root/cmds_2.tsv

• $ docker container run dit v ~/container:/root name volume c ubuntu cli

power
# $ docker run
=
=
=
=
=
=


.
.
.
.
.

-
.
-
-
-
-
-



-
-
-
Work on a remote machine
Run programs after disconnect

• $ tmux # tmux: terminal multiplexer

• (tmux)$ while true; do date; sleep 1; done # and then <C-b><d>.

• $ tmux ls # ls: the alias of list sessions.

• $ tmux a # a: attach

• (tmux) <C-b><?> # List key bindings; <q> to exit.

• (tmux) <C-c><C-d>

• $ tmux ls

• The way I start the server of JupyterLab:

• $ tmux new s jupyter lab d 'cd && jupyter lab no browser'

-
-
-
-
-
-
The local key pair

• (local)$ ll ~/.ssh/id_rsa

• If nothing there:

• (local)$ ssh keygen

• (local)$ cat ~/.ssh/id_rsa.pub # pub: public key

• id_rsa is the private key which proves who you are.

• Do not share id_rsa – the private key – to anyone.

• Simply generate a new pair when you have a little doubt about the old pair.
-

Add a user in the remote host

• (remote)# new_user=<new username>

• (remote)# useradd m s /bin/bash $new_user

• (remote)# su $new_user

• <new user>$ mkdir ~/.ssh

<new user>$ cat <<EOS >~/.ssh/authorized_keys
> <paste the content of id_rsa.pub>
> EOS
<new user>$ exit

• (remote)# service ssh start




-
-



Connect to a remote host

• (local)$ ssh ssh://<user>@localhost:2222

# $ ssh <user>@localhost p 2222
# $ ssh <host> # if port is 22 and usernames are the same.

• If there is a warning and you're sure you changed something in the remote:

• $ sed <line no>d ~/.ssh/known_hosts i

=
=
=
-
-





ssh -A: Allow the remote to use the local key pair

• (local) $ ssh ssh://<user>@localhost:2222

• (remote)$ ssh ubuntu c

# password:

• (local) $ ssh ssh://<user>@localhost:2222 -A

• (remote)$ ssh ubuntu c # OK

• The ubuntu-c works since the docker run --network enables the DNS
resolution between containers.

• See also: Consul, for the similar functionality without Docker.

.
.
.
-
-



ssh -J: Jump to a host in a private network

• (local)$ ssh ubuntu c

# Could not resolve hostname ubuntu c

• (local)$ ssh -J ssh://<user>@localhost:2222 ubuntu c

• -J is securer than -A; use -J when applicable.

-
>
-


-

Transfer files between local and remote

• (remote)$ cp cmds.tsv ~<user>

• (local) $ rsync e 'ssh ssh://<user>@localhost:2222' :~/cmds.tsv ~

# $ rsync <user>@<host ~/cmds.tsv ~ # if port is 22.

• (local) $ rsync e 'ssh ssh://<user>@localhost:2222' ~/cmds.tsv :~/cmds_2.tsv

• Deprecating scp – LWN.net, but scp is still a common way nowadays:

• (local)$ scp scp://<user>@localhost:2222/~/cmds.tsv ~

# $ scp <user>@<host ~/cmds.tsv ~ # if port is 22.

• (local)$ scp ~/cmds.tsv scp://<user>@localhost:2222/~/cmds_2.tsv

-
-

>
:
>
:



Mount a remote directory

• (local)$ sudo apt install sshfs # or macFUSE for macOS.

• (local)$ mkdir ~/sshfs_container

• (local)$ sshfs <user>@localhost:/home/<user> p 2222 ~/sshfs_container

• (local)$ ls ~/sshfs_container

• (local)$ umount ~/sshfs_container

ssh -L: Forward a local port to a remote port

• As known as “SSH tunneling”.

• (remote)$ python3 m http.server 1234 # <C-c> to interrupt.

• (local)$ ssh ssh://<user>@localhost:2222 -L 4321:ubuntu c:1234 -Nf

• Open http://localhost:4321/.

• localhost:4321 

→ localhost:2222 → ubuntu-c:22 

→ ubuntu-c:1234 → http.server 1234

• (local)$ pgrep f 4321

• (local)$ # pkill f 4321

-
-
-

Checkpoint:
Access the HTTP server by http://localhost:8765/
• Open http://localhost:8765/ and there should be a page.

• Hints:

• Use ssh -L.

 ssh -D: Start a proxy server

• (local)$ ssh ssh://<user>@localhost:2222 -D 1080 -Nf

• (remote)# iftop

• Con gure the system proxy or Proxy SwitchyOmega for Chrome.

• (remote)# python3 m http.server 80

• Open http://ubuntu-c/.

• For accessing the services in a private network by private DNS and standard ports!

• If the server has another public IP, try Google “my ip”.

• $ man ssh | grep ' -[LDNf]' -A 1

fi

System proxy for macOS

Proxy SwitchyOmega for Chrome
• See also:

• A simple ~/.ssh/con g for macOS

• A simple Bash con g for both macOS and Linux

fi
fi
Developing on CLI
Edit a config

• $ cp ~/.bashrc ~/backup_bashrc

• $ echo 'alias bat=batcat' >>~/.bashrc

• Re-enter the shell to take e ect.

• $ grep batcat ~/.bashrc n

• $ sed <line no>d ~/.bashrc i

• $ grep batcat ~/.bashrc n

-
-
ff
-

• $ vi ~/.bashrc • $ vi ~/.bashrc

• (vim) <C-End> # <Fn-Ctrl-Right> • (vim) /cat<CR>

• (vim) i<Right><CR> # or `a<CR>` • (vim) n

• (vim) alias bat=batcat # literally • (vim) dd

• (vim) <Esc> # backs to Normal • (vim) :x<CR>

• (vim) :x<CR> • $ grep batcat ~/.bashrc n

• $ grep batcat ~/.bashrc n

.

.
.

-
-

 The mini-cheatsheet of Vim

1. / n

• Search; Next.

2. i <Esc> :x

• Go Insert; Back to Normal; Write and quit.

3. dd

• Delete a line.

• It's all you need to know now.


Of course, the more you know, the more e ciency you'll get. 💪

ffi
Spell checking

• $ echo 'Oops. Does it spll like this?'<C-x><C-e>

• (vim) :set spell<CR>

• (vim) /sp<CR>

• (vim) z= # List spelling suggestions.

• (vim) 1<CR>

• (vim) :x<CR>

• See also:

• $ vimtutor # Learn Vim in 30 mins!

• (vim) :help dd, for example.

• Vim Keystrokes – Mosky’s Notes

• Vim’s Search Tips – Mosky’s Notes

• Vim’s Replace Tips – Mosky’s Notes

• Mosky's Vim Con gs

fi

Checkpoint: Add your bat

• (host)$ docker exec it ubuntu c bash

• ubuntu c# bat ~/.bashrc # It should work immediately.

• Hints:

• You may use echo to add.

• Use grep to con rm.

• Re-enter the shell to take e ect.

-
fi
-
ff
-

Developing
Slak: Collect data from Slack like a pro. ⚡

1. System Tools: the common tools which are 2. Project Tools: the tools of this project, good to
supposed to be installed in the system.
be managed within the project by, for example,
pipenv.

• git: the version control system.

• ipdb: a simple yet beautiful debugger.

• pip: the Python package manager.

• pytest: the testing framework.

• pipenv: one of Python package environment

managers.
3. Personal Tools: the tools of a developer, may be
installed in one's system.

• The Docker (containers) provides OS-level

isolation; Pipenv provides Python • flake8: the static analyzer (linter).

interpreter and package–level isolation.

• mypy: the static type checker.

• black: one of the formatters.

 • $ sudo apt install pip y # or, # System Tool

• # apt install pip y

• $ pip install pipenv # System Tool

• $ cd

• ~$ git clone depth 1 https://github.com/moskytw/slak.git

-
-
-

• ~$ cd slak

• ~/slak$ bat Pipf le

• $ pipenv sync dev • $ pipenv shell

• $ slak • (slak)$ slak

• $ python slak.py • $ python slak.py

• $ for cmd in python python3 slak; • $ for cmd in python python3 slak;
do type $cmd; done do type $cmd; done

-
i

• (slak)$ python m ipdb slak.py # Project Tool, `q` to exit.

• $ pytest # Project Tool

• $ pytest cov

• $ pytest cov cov report html

• $ cd htmlcov

• ~/slak/htmlcov$ python m http.server # <C-c> to interrupt.

• Open http://localhost:8000/.

• $ exit # or <C-d> to exit the shell of pipenv.

-
-
-
-

-
-
-
-
-

 • (my mac)$ for cmd in flake8 mypy black xxx; do type $cmd; done
flake8 is hashed (/usr/local/bin/flake8)
mypy is aliased to `\mypy ignore missing imports'
black is aliased to `black line length 79 skip string normalization'
bash: type: xxx: not found

• $ for cmd in flake8 mypy black xxx; do type $cmd >/dev/null; done
bash: type: xxx: not found
-
-
-
-
-

-
-
-


-
-
-
-


-






• $ cd ~/slak

• ~/slak$ flake8 slak.py

• $ mypy slak.py

• $ black slak.py

• The di erent languages have di erent toolsets; however, the concepts and
the categories are similar, e.g.:

• $ vi /etc/nginx/sites enabled/your site.conf

• $ nginx tc /etc/nginx/sites enabled/your site.conf

# Test the conf g f le.
ff
-
i
-
i
-
ff
-
-



• See also:

• git - the simple guide: “no deep shit ;)”

• Git Documentation

• Cheat Sheets

• Pro Git, the book

• Reference Manual

• Mosky’s Pipenv Notes

 In a large project

• It's all about locate the path and line number:

• f nd/fd: locate by path keywords.

• grep/rg: locate by content keywords.

• ctags: locate by pre-built “tags”.

• Language Server: a server keeps analyzing your code in background.

i
• $ cd /usr/lib/python3.10

• /usr/lib/python3.10$ f nd name ' server '

• $ sudo apt install fd f nd y # or,

• # apt install fd f nd y

• $ alias fd=fdf nd

• $ fd server # fd: f nd
i
-

i
i
-
i
-
i

-
-
*

• $ cd /usr/lib/python3.10

• /usr/lib/python3.10$ grep r 'Serving HTTP' n

• $ sudo apt install ripgrep y # or,

• # apt install ripgrep y

• $ rg 'Serving HTTP' # rg: ripgrep

-

-
-

 • $ f nd name 'server.py' exec grep 'Serving HTTP' n {} +

• $ fd '^server.py$' -X rg 'Serving HTTP' {}

• $ man f nd | less +'/ exec '

• $ man fd | less +'/ -[Xx]'

i
-
i
-
-

• Ctags & Language Server:

• $ sudo apt install universal ctags

$ ctags -R
$ vim +'help tags | only'

• A 10 MB–level solution.

• neoclide/coc.nvim – GitHub for the Language Server solution.

• A GB-level solution.



-


The end

• But hope it's the start of your

journey of getting the CLI power.

• Having fun is still the key, 


as always.

• May the man guide you!

Image Credits

• “The Power”
 linux-explained-for-new-users-including-cat-

https://www.facebook.com/nixcraft/photos/ command-explained-httpswwwcybercitib/
meme-based-upon-iamnotanartist_-comic- 3034767979869893/

for-linuxunix-users/3443131449033542/

• “The Dragon”

• “The Terminal”
 https://unsplash.com/photos/F6920BvzrZE

https://en.wikipedia.org/wiki/
Computer_terminal#/media/ • “The Miniatures”

File:DEC_VT100_terminal_transparent.png
https://unsplash.com/photos/Q4Honp3Pyqs

• “What Is a Container?”
 • “The Banshee Appears”


https://www.docker.com/resources/what- https://en.wikipedia.org/wiki/Banshee#/
container/
media/File:The_Banshee_Appears_(1862).jpg

• “The Cat”
 • “The Earth”


https://www.facebook.com/nixcraft/posts/ https://unsplash.com/photos/Q1p7bh3SHj8