• Authoring Books with R Markdown
  • 1 Welcome to Shell-Genomics!
  • 2 Introducing the Shell
    • 2.1 What is a shell and why should I care?
    • 2.2 How to access the shell
    • 2.3 Navigating your file system
    • 2.4 Tip
    • 2.5 Challenge
    • 2.6 Solution
      • 2.6.1 Shortcut: Tab Completion
  • 3 Navigating Files and Directories
    • 3.1 Moving around the file system
    • 3.2 Finding hidden directories
    • 3.3 Exercise
    • 3.4 Solution
    • 3.5 Examining the contents of other directories
    • 3.6 Navigating practice
    • 3.7 Exercise
    • 3.8 Solution
    • 3.9 Full vs. Relative Paths
    • 3.10 Relative path resolution
    • 3.11 Exercise
    • 3.12 Solution
      • 3.12.1 Navigational Shortcuts
  • 4 Working with Files and Directories
    • 4.1 Working with Files
      • 4.1.1 Our data set: FASTQ files
      • 4.1.2 Wildcards
    • 4.2 Exercise
    • 4.3 Solution
    • 4.4 Exercise
    • 4.5 Solution
    • 4.6 Command History
    • 4.7 Exercise
    • 4.8 Solution
    • 4.9 Exercise
    • 4.10 Solution
    • 4.11 Details on the FASTQ format
    • 4.12 Exercise
    • 4.13 Solution
    • 4.14 Creating, moving, copying, and removing
      • 4.14.1 Copying Files
      • 4.14.2 Creating Directories
      • 4.14.3 Moving / Renaming
      • 4.14.4 File Permissions
      • 4.14.5 Removing
    • 4.15 Exercise
    • 4.16 Solution
  • 5 Redirection
    • 5.1 Searching files
    • 5.2 Nucleotide abbreviations
    • 5.3 Determining quality
    • 5.4 Exercise
    • 5.5 Solution
    • 5.6 Redirecting output
    • 5.7 File extensions
    • 5.8 Exercise
    • 5.9 Solution
    • 5.10 Exercise
    • 5.11 Solution
    • 5.12 File extensions - part 2
    • 5.13 Custom grep control
    • 5.14 File manipulation and more practices with pipes
    • 5.15 Writing for loops
    • 5.16 Using Basename in for loops
    • 5.17 Exercise
    • 5.18 Solution
    • 5.19 Exercise
    • 5.20 Solution
  • 6 Writing Scripts and Working with Data
    • 6.1 Writing files
    • 6.2 Which Editor?
    • 6.3 Control, Ctrl, or ^ Key
    • 6.4 Exercise
    • 6.5 Solution
    • 6.6 Writing scripts
    • 6.7 Custom grep control
    • 6.8 Exercise
    • 6.9 Solution
    • 6.10 Making the script into a program
    • 6.11 Moving and Downloading Data
      • 6.11.1 Getting data from the cloud
  • 7 Project Organization
    • 7.1 Tip
    • 7.2 Exercise
    • 7.3 Solution
    • 7.4 Organizing your files
    • 7.5 Documenting your activity on the project
    • 7.6 Exercise
    • 7.7 Solution
    • 7.8 Navigating in Nano
    • 7.9 References
  • Published with bookdown

Shell-genomics

3 Navigating Files and Directories

3.1 Moving around the file system

We’ve learned how to use pwd to find our current location within our file system. We’ve also learned how to use cd to change locations and ls to list the contents of a directory. Now we’re going to learn some additional commands for moving around within our file system.

Use the commands we’ve learned so far to navigate to the shell_data/untrimmed_fastq directory, if you’re not already there.


cd
cd ~/shell_data
cd untrimmed_fastq

What if we want to move back up and out of this directory and to our top level directory? Can we type cd shell_data? Try it and see what happens.


cd
cd ~/shell_data
cd untrimmed_fastq
cd shell_data

Your computer looked for a directory or file called shell_data within the directory you were already in. It didn’t know you wanted to look at a directory level above the one you were located in.

We have a special command to tell the computer to move us back or up one directory level.


cd ~/shell_data
cd untrimmed_fastq
cd ..

Now we can use pwd to make sure that we are in the directory we intended to navigate to, and ls to check that the contents of the directory are correct.


cd
cd ~/shell_data
cd untrimmed_fastq
pwd
 
## /Users/ggiaever/shell_data/untrimmed_fastq

cd
cd ~/shell_data
cd untrimmed_fastq
ls
## SRR097977.fastq
## SRR098026.fastq
## bad-reads-script.sh
## bad_reads_2022_2022.txt
## bad_reads_2022_2022_2022_2022.txt
## bad_reads_2022_2022_2022_2022_2022_2022.txt
## bad_reads_2022_2022_2022_2022_2022_2022_2022_2022.txt
## scripted_bad_reads.txt
## scripted_bad_reads_2022.txt
## scripted_bad_reads_2022_2022_2022.txt
## scripted_bad_reads_2022_2022_2022_2022_2022.txt
## scripted_bad_reads_2022_2022_2022_2022_2022_2022_2022_2022.txt
## seq_info_2022_2022.txt
## seq_info_2022_2022_2022_2022.txt
## seq_info_2022_2022_2022_2022_2022_2022.txt
## seq_info_2022_2022_2022_2022_2022_2022_2022_2022.txt
## species_EnsemblBacteria.txt
## species_EnsemblBacteria_2022.txt
## species_EnsemblBacteria_2022_2022_2022.txt
## species_EnsemblBacteria_2022_2022_2022_2022_2022.txt
## species_EnsemblBacteria_2022_2022_2022_2022_2022_2022_2022_2022.txt

From this output, we can see that .. did indeed take us back one level in our file system.

You can chain these together like so:


cd
cd ~/shell_data
cd untrimmed_fastq
ls ../../
## @sockeye.arc.ubc.ca
## Applications
## Brewfile
## Desktop
## Documents
## Downloads
## Google Drive
## Library
## Makevars
## Mambaforge-MacOSX-x86_64.sh
## Miniconda3-latest-MacOSX-arm64.sh
## Miniconda3-latest-MacOSX-x86_64.sh
## Movies
## Music
## Parallels
## Pictures
## Public
## STAR-2.7.10a
## Your appointment at the U.S. Consulate for passport services - ggiaever@gmail.com - Gmail.pdf
## Zotero
## backup-rstudio-prefs
## backuup-config-rstudio
## bashrc64
## build.sh
## conda.sh
## condash
## cpc2_result.txt
## dc_workshop
## dc_workshop_log_2022_11_27.sh
## dc_workshopfiles_for_igv
## echo
## err_md5sum.txt
## example1.cpp
## fasterq.tmp.Guris-MacBook-Pro-2.local.67466
## find.txt
## gsea_home
## hello.cpp
## helloworld
## homebrew
## igv
## index.bam
## iupred2a_result.txt
## metakeypair
## metakeypair-nopass.pem
## metakeypair.pem
## metakeypair.pub
## miniconda3
## miniconda3-intel
## my_key
## my_key.pub
## node_modules
## omp-test.c
## opt
## package-lock.json
## package.json
## path
## pfam_result.txt
## quality
## quant
## rstudio-diagnostics
## shell_data
## signalP_result.txt
## snakemake-tutorial
## species_EnsemblBacteria.txt
## sratoolkit.3.0.0-mac64
## sub.tar.gz
## test-openblas.c
## tnp
## zprofileworks.txt
## zshrc_backup.txt

prints the contents of /Users.

3.2 Finding hidden directories

3.3 Exercise

First navigate to the shell_data directory. There is a hidden directory within this directory. Explore the options for ls to find out how to see hidden directories. List the contents of the directory and identify the name of the text file in that directory.

Hint: hidden files and folders in Unix start with ., for example .my_hidden_directory

3.4 Solution


cd
cd ~/shell_data
ls -a
## .
## ..
## .DS_Store
## .hidden
## sra_metadata
## untrimmed_fastq

In most commands the flags can be combined together in no particular order to obtain the desired results/output.


cd
cd ~/shell_data
cd untrimmed_fastq
ls -Fa
ls -laF
## ./
## ../
## .DS_Store
## SRR097977.fastq
## SRR098026.fastq
## bad-reads-script.sh*
## bad_reads_2022_2022.txt
## bad_reads_2022_2022_2022_2022.txt
## bad_reads_2022_2022_2022_2022_2022_2022.txt
## bad_reads_2022_2022_2022_2022_2022_2022_2022_2022.txt
## scripted_bad_reads.txt
## scripted_bad_reads_2022.txt
## scripted_bad_reads_2022_2022_2022.txt
## scripted_bad_reads_2022_2022_2022_2022_2022.txt
## scripted_bad_reads_2022_2022_2022_2022_2022_2022_2022_2022.txt
## seq_info_2022_2022.txt
## seq_info_2022_2022_2022_2022.txt
## seq_info_2022_2022_2022_2022_2022_2022.txt
## seq_info_2022_2022_2022_2022_2022_2022_2022_2022.txt
## species_EnsemblBacteria.txt
## species_EnsemblBacteria_2022.txt
## species_EnsemblBacteria_2022_2022_2022.txt
## species_EnsemblBacteria_2022_2022_2022_2022_2022.txt
## species_EnsemblBacteria_2022_2022_2022_2022_2022_2022_2022_2022.txt
## total 82272
## drwxr-xr-x@ 24 ggiaever  staff      768 Nov 26 18:03 ./
## drwxrwxr-x@  6 ggiaever  staff      192 Oct 20 19:06 ../
## -rw-r--r--@  1 ggiaever  staff     6148 Nov 26 15:55 .DS_Store
## -rw-r--r--@  1 ggiaever  staff    47552 Nov 15  2017 SRR097977.fastq
## -rw-r--r--@  1 ggiaever  staff    43332 Nov 15  2017 SRR098026.fastq
## -rwxr-xr-x@  1 ggiaever  staff       76 Oct 23 22:50 bad-reads-script.sh*
## -rw-r--r--   1 ggiaever  staff    23214 Nov 26 18:03 bad_reads_2022_2022.txt
## -rw-r--r--   1 ggiaever  staff    23214 Nov 26 17:12 bad_reads_2022_2022_2022_2022.txt
## -rw-r--r--   1 ggiaever  staff    23214 Nov 25 19:02 bad_reads_2022_2022_2022_2022_2022_2022.txt
## -rw-r--r--   1 ggiaever  staff    23214 Nov 24 23:24 bad_reads_2022_2022_2022_2022_2022_2022_2022_2022.txt
## -rw-r--r--   1 ggiaever  staff    23214 Nov 26 18:03 scripted_bad_reads.txt
## -rw-r--r--   1 ggiaever  staff    23214 Nov 26 17:16 scripted_bad_reads_2022.txt
## -rw-r--r--   1 ggiaever  staff    23214 Nov 26 17:12 scripted_bad_reads_2022_2022_2022.txt
## -rw-r--r--   1 ggiaever  staff    23214 Nov 25 19:02 scripted_bad_reads_2022_2022_2022_2022_2022.txt
## -rw-r--r--   1 ggiaever  staff    23214 Oct 23 22:51 scripted_bad_reads_2022_2022_2022_2022_2022_2022_2022_2022.txt
## -rw-r--r--   1 ggiaever  staff      179 Nov 26 18:03 seq_info_2022_2022.txt
## -rw-r--r--   1 ggiaever  staff      179 Nov 26 17:12 seq_info_2022_2022_2022_2022.txt
## -rw-r--r--   1 ggiaever  staff      179 Nov 25 19:02 seq_info_2022_2022_2022_2022_2022_2022.txt
## -rw-r--r--   1 ggiaever  staff      179 Nov 24 23:24 seq_info_2022_2022_2022_2022_2022_2022_2022_2022.txt
## -rw-r--r--   1 ggiaever  staff  8352381 Nov 26 18:03 species_EnsemblBacteria.txt
## -rw-r--r--   1 ggiaever  staff  8352381 Nov 26 17:16 species_EnsemblBacteria_2022.txt
## -rw-r--r--   1 ggiaever  staff  8352381 Nov 26 17:12 species_EnsemblBacteria_2022_2022_2022.txt
## -rw-r--r--   1 ggiaever  staff  8352381 Nov 25 19:02 species_EnsemblBacteria_2022_2022_2022_2022_2022.txt
## -rw-r--r--   1 ggiaever  staff  8352381 Oct 23 23:01 species_EnsemblBacteria_2022_2022_2022_2022_2022_2022_2022_2022.txt

3.5 Examining the contents of other directories

By default, the ls commands lists the contents of the working directory (i.e. the directory you are in). You can always find the directory you are in using the pwd command. However, you can also give ls the names of other directories to view. Navigate to your home directory if you are not already there.


cd

Then enter the command:


cd
ls 
## @sockeye.arc.ubc.ca
## Applications
## Brewfile
## Desktop
## Documents
## Downloads
## Google Drive
## Library
## Makevars
## Mambaforge-MacOSX-x86_64.sh
## Miniconda3-latest-MacOSX-arm64.sh
## Miniconda3-latest-MacOSX-x86_64.sh
## Movies
## Music
## Parallels
## Pictures
## Public
## STAR-2.7.10a
## Your appointment at the U.S. Consulate for passport services - ggiaever@gmail.com - Gmail.pdf
## Zotero
## backup-rstudio-prefs
## backuup-config-rstudio
## bashrc64
## build.sh
## conda.sh
## condash
## cpc2_result.txt
## dc_workshop
## dc_workshop_log_2022_11_27.sh
## dc_workshopfiles_for_igv
## echo
## err_md5sum.txt
## example1.cpp
## fasterq.tmp.Guris-MacBook-Pro-2.local.67466
## find.txt
## gsea_home
## hello.cpp
## helloworld
## homebrew
## igv
## index.bam
## iupred2a_result.txt
## metakeypair
## metakeypair-nopass.pem
## metakeypair.pem
## metakeypair.pub
## miniconda3
## miniconda3-intel
## my_key
## my_key.pub
## node_modules
## omp-test.c
## opt
## package-lock.json
## package.json
## path
## pfam_result.txt
## quality
## quant
## rstudio-diagnostics
## shell_data
## signalP_result.txt
## snakemake-tutorial
## species_EnsemblBacteria.txt
## sratoolkit.3.0.0-mac64
## sub.tar.gz
## test-openblas.c
## tnp
## zprofileworks.txt
## zshrc_backup.txt

The cd command works in a similar way.

Try entering:


cd ~/shell_data
cd untrimmed_fastq

This will take you to the untrimmed_fastq directory

3.6 Navigating practice

3.7 Exercise

Navigate to your home directory. From there, list the contents of the untrimmed_fastq directory.

3.8 Solution

3.9 Full vs. Relative Paths

The cd command takes an argument which is a directory name. Directories can be specified using either a relative path or a full absolute path. The directories on the computer are arranged into a hierarchy. The full path tells you where a directory is in that hierarchy. Navigate to the home directory, then enter the pwd command.


cd  
pwd  
## /Users/ggiaever

You will see:

/Users/your-username

This is the full name of your home directory. This tells you that you are in a directory called your-username, which sits inside a directory called home which sits inside the very top directory in the hierarchy. The very top of the hierarchy is a directory called / which is usually referred to as the root directory. So, to summarize: your-username is a directory in home which is a directory in /. More on root and home in the next section.

Now enter the following command:


cd /Users/ggiaever/shell_data/.hidden

This jumps forward multiple levels to the .hidden directory. Now go back to the home directory.


cd ~

You can also navigate to the .hidden directory using:


cd ~/shell_data/.hidden

These two commands have the same effect, they both take us to the .hidden directory. The first uses the absolute path, giving the full address from the home directory. The second uses a relative path, giving only the address from the working directory. A full path always starts with a /. A relative path does not.

A relative path is like getting directions from someone on the street. They tell you to “go right at the stop sign, and then turn left on Main Street”. That works great if you’re standing there together, but not so well if you’re trying to tell someone how to get there from another country. A full path is like GPS coordinates. It tells you exactly where something is no matter where you are right now.

You can usually use either a full path or a relative path depending on what is most convenient. If we are in the home directory, it is more convenient to enter the full path. If we are in the working directory, it is more convenient to enter the relative path since it involves less typing.

Over time, it will become easier for you to keep a mental note of the structure of the directories that you are using and how to quickly navigate amongst them.

3.10 Relative path resolution

3.11 Exercise

Using the filesystem diagram below, if pwd displays /Users/thing, what will ls ../backup display?

  1. ../backup: No such file or directory
  2. 2012-12-01 2013-01-08 2013-01-27
  3. 2012-12-01/ 2013-01-08/ 2013-01-27/
  4. original pnas_final pnas_sub

File System for Challenge Questions

3.12 Solution

3.12.1 Navigational Shortcuts

The root directory is the highest level directory in your file system and contains files that are important for your computer to perform its daily work. While you will be using the root (/) at the beginning of your absolute paths, it is important that you avoid working with data in these higher-level directories, as your commands can permanently alter files that the operating system needs to function. In many cases, trying to run commands in root directories will require special permissions which are not discussed here, so it’s best to avoid them and work within your home directory. Dealing with the home directory is very common. The tilde character, ~, is a shortcut for your home directory. In our case, the root directory is two levels above our home directory, so cd or cd ~ will take you to /Users/your-username and cd / will take you to /. Navigate to the home directory:


cd ~

Then enter the command:


cd ~
ls 
## @sockeye.arc.ubc.ca
## Applications
## Brewfile
## Desktop
## Documents
## Downloads
## Google Drive
## Library
## Makevars
## Mambaforge-MacOSX-x86_64.sh
## Miniconda3-latest-MacOSX-arm64.sh
## Miniconda3-latest-MacOSX-x86_64.sh
## Movies
## Music
## Parallels
## Pictures
## Public
## STAR-2.7.10a
## Your appointment at the U.S. Consulate for passport services - ggiaever@gmail.com - Gmail.pdf
## Zotero
## backup-rstudio-prefs
## backuup-config-rstudio
## bashrc64
## build.sh
## conda.sh
## condash
## cpc2_result.txt
## dc_workshop
## dc_workshop_log_2022_11_27.sh
## dc_workshopfiles_for_igv
## echo
## err_md5sum.txt
## example1.cpp
## fasterq.tmp.Guris-MacBook-Pro-2.local.67466
## find.txt
## gsea_home
## hello.cpp
## helloworld
## homebrew
## igv
## index.bam
## iupred2a_result.txt
## metakeypair
## metakeypair-nopass.pem
## metakeypair.pem
## metakeypair.pub
## miniconda3
## miniconda3-intel
## my_key
## my_key.pub
## node_modules
## omp-test.c
## opt
## package-lock.json
## package.json
## path
## pfam_result.txt
## quality
## quant
## rstudio-diagnostics
## shell_data
## signalP_result.txt
## snakemake-tutorial
## species_EnsemblBacteria.txt
## sratoolkit.3.0.0-mac64
## sub.tar.gz
## test-openblas.c
## tnp
## zprofileworks.txt
## zshrc_backup.txt

This prints the contents of your home directory, without you needing to type the full path.

The commands cd, and cd ~ are very useful for quickly navigating back to your home directory. We will be using the ~ character in later lessons to specify our home directory.