Sigolyori’s Blog - Missing Semester: Shell Tools and Scripting

아래 노트북은 MIT에서 제공하는 Missing semester의 연습문제를 풀이한 것입니다. 이번 내용은 셸 툴 및 스크립팅(Shell Tools and Scripting) 에 대한 것입니다.

Read man ls and write an ls command that lists files in the following manner

Includes all files, including hidden files
Sizes are listed in human readable format (e.g. 454M instead of 454279954)
Files are ordered by recency
Output is colorized A sample output would look like this

-rw-r–r– 1 user group 1.1M Jan 14 09:53 baz
drwxr-xr-x 5 user group 160 Jan 14 09:53 .
-rw-r–r– 1 user group 514 Jan 14 06:42 bar
-rw-r–r– 1 user group 106M Jan 13 12:12 foo
drwx——+ 47 user group 1.5K Jan 12 18:08 ..

!man ls

옵션의 기능은 아래와 같습니다. - -a: 디렉토리 및 첫 시작이 .(dot)인 파일도 포함합니다. 따라서 숨긴 파일도 확인할 수 있습니다. - -l: long format으로 출력합니다. - -h: 사람이 읽을 수 있는 사이즈 형식으로 변경합니다. - -t: 수정 시간에 따른 내림차순으로 정렬됩니다. - -G: 색상화되어 출력합니다.

!ls -alt -h -G

total 2560
-rwxrwxrwx  1 hyk  staff   1.9K Dec 12 14:50 gradio.ipynb
-rwxrwxrwx  1 hyk  staff    15K Dec 12 11:51 fastai-chap1.ipynb
-rwxrwxrwx@ 1 hyk  staff    43K Dec 11 19:59 missing-ch01.ipynb
-rwxrwxrwx  1 hyk  staff    33K Dec 11 19:51 missing-ch02.ipynb
-rwxrwxrwx  1 hyk  staff   4.0K Dec  8 17:43 ._missing-ch01.ipynb
drwxrwxrwx  1 hyk  staff   128K Dec  2 12:28 .
drwxrwxrwx  1 hyk  staff   128K Dec  2 12:28 ..
drwxrwxrwx  1 hyk  staff   128K Dec  2 12:28 image
-rwxrwxrwx  1 hyk  staff    15K Dec  2 12:28 sparkml.ipynb
-rwxrwxrwx  1 hyk  staff   6.0K Dec  2 12:28 .DS_Store

Write bash functions marco and polo that do the following. Whenever you execute marco the current working directory should be saved in some manner, then when you execute polo, no matter what directory you are in, polo should cd you back to the directory where you executed marco. For ease of debugging you can write the code in a file marco.sh and (re)load the definitions to your shell by executing source marco.sh.

missing-semester 폴더의 workbook으로 이동하여 2번을 수행하겠습니다.

%cd ../../../missing-semester/workbook

/Volumes/Samsung_T5 1/git/missing-semester/workbook

marco.sh는 아래와 같이 작성합니다. marco를 사용하면 pwd(현재 경로)가 dir이란 변수에 담기게 됩니다.

#!/bin/bash
marco () {
    dir=$(pwd)
}

polo.sh는 다음과 같습니다. 위에서 저장한 dir 변수로 이동하는 함수를 작성합니다.

#!/bin/bash
polo () {
cd "$dir"
}

Say you have a command that fails rarely. In order to debug it you need to capture its output but it can be time consuming to get a failure run. Write a bash script that runs the following script until it fails and captures its standard output and error streams to files and prints everything at the end. Bonus points if you can also report how many runs it took for the script to fail.


 #!/usr/bin/env bash

 n=$(( RANDOM % 100 ))

 if [[ n -eq 42 ]]; then
    echo "Something went wrong"
    >&2 echo "The error was using magic numbers"
    exit 1
 fi

 echo "Everything went according to plan"

As we covered in the lecture find’s -exec can be very powerful for performing operations over the files we are searching for. However, what if we want to do something with all the files, like creating a zip file? As you have seen so far commands will take input from both arguments and STDIN. When piping commands, we are connecting STDOUT to STDIN, but some commands like tar take inputs from arguments. To bridge this disconnect there’s the xargs command which will execute a command using STDIN as arguments. For example ls | xargs rm will delete the files in the current directory.

Your task is to write a command that recursively finds all HTML files in the folder and makes a zip with them. Note that your command should work even if the files have spaces (hint: check -d flag for xargs).

If you’re on macOS, note that the default BSD find is different from the one included in GNU coreutils. You can use -print0 on find and the -0 flag on xargs. As a macOS user, you should be aware that command-line utilities shipped with macOS may differ from the GNU counterparts; you can install the GNU versions if you like by using brew.

missing-semeter 폴더에 존재하는 html 파일 리스트를 받아, 파일을 하나의 zip 파일로 압축하겠습니다.

%cd ..

-print0는 R의 paste0와 비슷합니다. null 문자로 find에 의해 찾아진 객체를 나누고, xargs의 -0 옵션은 null 문자를 delimeter로 사용하게 됩니다. 따라서 찾아진 파일들을 구분하고, 하나의 zip 파일로 만들 수 있습니다.

!find . -name "*.html" -print0 | xargs -0 tar -cf html.zip

(Advanced) Write a command or script to recursively find the most recently modified file in a directory. More generally, can you list all files by recency?

!find . -maxdepth 1 -type d -exec sh -c "ls {} -lt | head -2 | tail -1" \;

위 코드로 Current Directory 내 존재하는 디렉토리(폴더)에 한해 가장 최근 수정된 파일을 조회할 수 있습니다.

-exec 옵션 내에는 sh -c로 쉘 스크립트를 실행시켰습니다.

따라서 find 명령에 따라 조회된 각각의 디렉토리에 대해 ls 명령이 실행되는 구조입니다.