Linux: File Systems and devices
- Details
- Last Updated: Thursday, 28 September 2023 23:36
- Published: Tuesday, 19 November 2019 04:02
- Hits: 4369
Filesystems, Partitions and devices:
When you get any storage medium, whether it's usb flash drive, or hard disk or cdrom, they all store bits as 1 or 0 in every location of the device. We need to have a way so that we can identify which group of bits are together to form a picture or a song or a text file, etc. This is where the File system (FS) comes into picture. It establishes a particular way of storing data to retrieve it later on.
Very good intro here: https://www.tldp.org/LDP/sag/html/disk-usage.html
HARD DISK:
Each device is rep by separate device file. Hard disk interface are either IDE hard disk or SCSI hard disk. SCSI was the earliest hard disk, followed by IDE. IDE were parallel ATA or PATA i/f (parallel AT attachment). These have largely been replaced by SATA (or serial ATA) i/f since it's intro in 2000, due to lower pin count, reduced complexity and higher speeds. SATA rev 1 had max transfer speed of 1.5Gbits/sec, while rev2 had 3.0Gbits/sec. Latest rev3 has transfer speed of 6.0Gbits/sec. SATA pin i/f consists of 7 data pins and 15 power pins in 2 set of pins. Out of 7 data pins, 3 are gnd pins, and remaining 4 are differential tx/rx pins (A+/A- for transmit, B+/B- for receive). differential signaling reduces noise at high speed transmissions. 15 power pins have 3.3V/5.0V/12.0V power supply pins along with ground pin. Smaller SATA devices combine data pins and power pins in the same set, with only 5V supply lines provided to reduce pin count.
Now we have SSD Hard disk, which have very different i/f, and much faster speeds.
Disk partition:
Each HD can be divided into several partitions, which allows different partitions of the same disk to behave as separate hard disks. For all practical purposes, partitions can be thought s separate hard disk. So, we'll talk at partition level.
First sector of each HD (NOT partition) contains the partitioning info. This 1st sector is called master boot sector.
Very good intro here: https://www.minitool.com/lib/boot-sector.html
some more info here: https://www.bydavy.com/2012/01/lets-decrypt-a-master-boot-record/
Boot sector (or Master Boot Sector or Master Boot record (MBR) is more common term) usually refers to the first sector of hard disk. It's used for loading and transferring processor control right to operating system. After power up, boot process starts from ROM on motherboard, and then control is transferred to hard disk. This boot sector is the very first location read from hard disk.
MBR contains a small program that reads the partition table, checks which partition is active (that is, marked bootable), and reads the first sector of that partition, the partition's boot sector (the MBR is also a boot sector, but it has a special status and therefore a special name). This boot sector contains another small program that reads the first part of the operating system stored on that partition (assuming it is bootable), and then starts it. Control is transferred to OS at this point.
MBR is all contained in 1st sector of hard disk (cyl 0, head 0 sector 1). All sectors are 512 bytes.
The original partitioning scheme for PC hard disks allowed only four partitions. This quickly turned out to be too little in real life. to overcome this, primary partition was allowed to be subdivided further into more partitions, which were called logical partitions. Such a primary partition was called extended partition. First sector of each primary partition as well that of the disk is called boot sector, and contains boot pgm.
Each partition and extended partition has its own device file. The naming convention for these files is that a partition's number is appended after the name of the whole disk, with the convention that 1-4 are primary partitions (regardless of how many primary partitions there are) and number greater than 5 are logical partitions (regardless of within which primary partition they reside). Linux FS uses block, instead of sectors, where each block size=1024 bytes=2 sectors.
NOTE: size conversion are 1024 bytes = 1KB, 1024 KB=1MB, 1024MB=1GB. However, almost all linux cmds approximate 1024 with 1000 for easy calculation, so sizes may differ b/w diff cmds. So, 1GB is treated as 1000MB, and 1MB is treated as 1000KB, and 1KB is treated as 1000 bytes, even though 1 block = 2 sectors = 1024 bytes. So, this results in an error of about 10% compared to exact calculation.
Naming convention:
In Windows, each partition is assigned a "drive letter name" as C:/ (called as C drive), etc. In Linux, everything is treated as a file. So, all devices and partitions are essentially files. All devices are given file names in the form /dev/xxyN, where xx=hd (for IDE disks) and sd (for SCSI disks), y=a,b,c etc indicating 1st disk, 2nd disk, etc, while N=partition number on that disk. So, /dev/sda => refers to 1st disk, while /dev/sda2 refers to 2nd partition on 1st disk. We use "sd" instead of "hd" even though hard disks are now all SATA IDE (probably due to legacy)
fdisk: cmd "fdisk" provides all info about devices. It's used for disk partition mgmt/info too.
prompt> sudo fdisk -l
Disk /dev/sda: 128.0 GB, 128035676160 bytes, 250069680 sectors => Here hard disk is 128GB in size. It's dual boot laptop, which has Windows and Linux installed. If we treat each sector as 1024 bytes, and be exact in our calculations, then 128035676160 bytes = 250069680 sectors = 28035676160/1024KB = 125034840KB = 125034840/1024MB = 122104.3 MB = 122104.3/1024 GB = 119.24GB. However, hard disk is reported as 128GB, which uses multiplication of 1000 instead of 1024, so 128035676160 bytes = 128.03 GB.
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x948dbc73
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 1026047 512000 7 HPFS/NTFS/exFAT => primary partition is from sda1 to sda4. sda1 size=512K blocks*1024/1000KB=524.3MB (assuming factors of 1000 instead of 1024). This is Windows system boot partition. It has boot, EFI dir in it.
/dev/sda2 1026048 166098943 82536448 7 HPFS/NTFS/exFAT => size=82536K blocks* 1024/1000=84516MB=84.5GB. This is Windows system partition, where all user pgm, etc are kept. This is the partition we are in when we log into our windows m/c.
/dev/sda3 248018944 250066943 1024000 27 Hidden NTFS WinRE => sda3 size=1024K blocks*1024/1000=1048.6MB=1.05GB. This is Windows Recovery partition.
/dev/sda4 166098944 248018943 40960000 5 Extended => 4th primary partition is an extended partition and contains 2 logical partitions, sda5 and sda6. size=40960K blocks *1024/1000 = 41.94GB. This extended partition is the one where we have Linux installed. It contains Linux OS (OS+user area) and memory swap space
/dev/sda5 166100992 168198143 1048576 83 Linux => This is first logical partition of extended partition. size=1048K blocks*1024/1000= 1073MB=1.07GB
/dev/sda6 168200192 248018943 39909376 8e Linux LVM => 2nd logical parition. size=39909K blocks*1024/1000 = 40866MB=40.86GB. This contains centos-root and centos-swap spaces as indicated below. Total blocks=35807,232+4096,000=39903232 blocks, pretty close to those indicated above
Partition table entries are not in disk order => NOTE: sda1, sda2, sda4 and sda3 occupy continuous sectors w/o any missing sectors in b/w them.
Disk /dev/mapper/centos-root: 36.7 GB, 36666605568 bytes, 71614464 sectors => blocks=35807232. so, size=36.7GB (due to rounding of 1000)
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos-swap: 4194 MB, 4194304000 bytes, 8192000 sectors => blocks=4096000. so, size=4194MB (due to rounding of 1000). Note: I chose 4GB swap space during installation of CentOS, which equated to 4*1024*1024KB=4194304KB=4194304000 bytes (where 1GB=1024MB, 1MB=1024KB, 1KB=1000 bytes. So, installation process calculates GB differently than the one calculated by Linux OS)
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
parted: parted is another cmd for mgmt/info of disk partitions. It is most commonly used for partitioning of hard disks.
prompt> sudo parted -l => this also uses rounding of 1000, same as that of fdisk
Model: ATA SAMSUNG MZ7TD128 (scsi)
Disk /dev/sda: 128GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 1049kB 525MB 524MB primary ntfs boot
2 525MB 85.0GB 84.5GB primary ntfs
4 85.0GB 127GB 41.9GB extended
5 85.0GB 86.1GB 1074MB logical xfs => This shows logical partition 5 has 1 GB size, same as what fdisk showed.
6 86.1GB 127GB 40.9GB logical lvm => Here, logical partition 6 shows 40.9GB size, same as what fdisk showed
3 127GB 128GB 1049MB primary ntfs diag
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/centos-swap: 4194MB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 4194MB 4194MB linux-swap(v1)
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/centos-root: 36.7GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 36.7GB 36.7GB xfs
------
dd: To copy contents of any device, we can use "dd" cmd. dd copies raw bytes from specified location, and has nothing to do with FS.
ex: sudo dd if=/home/file1.img of=/dev/sdb status=progress oflag=sync bs=4M => this copies file1.img from i/p file to of file /dev/sdb. Here device to be written to is the device name and not the prtition name, as we want to write the file to the device, which will create multiple partitions as needed. If we arite the file to a particular partition then booting from this new device may not work, as the boot may require particular info in main partition to boot up. option status shows status progress (otherwise no status is shown, so the device will seem stuck as it takes couple of minutes for large files to be copied), block size (bs) to transfer is set to 4M (default is 512 Bytes which will take a long time to transfer multi GB file). sudo is used since usually this cmd requires root privilege
To view contents of MBR, type below 2 cmds in a terminal:
prompt> dd if=/dev/sda of=mbr.bin bs=512 count=1 => dd cmd copies contents of any device byte by byte. if=input file, of=output file, bs=size in bytes to rd/wrt at a time, count=num of input blocks to copy. So, this cmd copies 512 bytes from 1st sector of /dev/sda device (usually hard disk) to file mbr.bin.
prompt> od -xa mbr.bin" => od dumps contents of file in octal or other formats. -x => imples hex format, -a implies print ascii char for each byte. On my hard disk, it lists below content (each line has 16 bytes):
000000 63eb d090 00bc 8e7c 8ec0 bed8 7c00 00bf => The following 446 bytes taken up by master boot program of hard disk is from 0x0000 hex to 0x01BD hex. 2 bytes are written as pair where MSB byte is written before LSB byte, so within a pair, order is reversed, i.e 63eb has 1st byte as eb, and 2nd byte as 63, d090 has 1st byte as 90, and 2nd byte as d0, and so on ... This code contains the bootstrap loader code, which is either GRUB or ...? FIXME?
k c dle P < nul | so @ so X > nul | ? nul
000010 b906 0200 f3fc 50a4 1c68 cb06 b9fb 0004
ack 9 nul stx | s $ P h fs ack K { 9 eot nul
000020 bebd 8007 007e 7c00 0f0b 0e85 8301 10c5
= > bel nul ~ nul nul | vt si enq so soh etx E dle
000030 f1e2 18cd 5688 5500 46c6 0511 46c6 0010
b q M can bs V nul U F F dc1 enq F F dle nul
000040 41b4 aabb cd55 5d13 0f72 fb81 aa55 0975
4 A ; * U M dc3 ] r si soh { U * u ht
000050 c1f7 0001 0374 46fe 6610 8000 0001 0000
w A soh nul t etx ~ F dle f nul nul soh nul nul nul
000060 0000 0000 faff 9090 c2f6 7480 f605 70c2
nul nul nul nul del z dle dle v B nul t enq v B p
000070 0274 80b2 79ea 007c 3100 8ec0 8ed8 bcd0
t stx 2 nul j y | nul nul 1 @ so X so P <
000080 2000 a0fb 7c64 ff3c 0274 c288 be52 7c05
nul sp { sp d | < del t stx bs B R > enq |
000090 41b4 aabb cd55 5a13 7252 813d 55fb 75aa
4 A ; * U M dc3 Z R r = soh { U * u
0000a0 8337 01e1 3274 c031 4489 4004 4488 89ff
7 etx a soh t 2 1 @ ht D eot @ bs D del ht
0000b0 0244 04c7 0010 8b66 5c1e 667c 5c89 6608
D stx G eot dle nul f vt rs \ | f ht \ bs f
0000c0 1e8b 7c60 8966 0c5c 44c7 0006 b470 cd42
vt rs ` | f ht \ ff G D ack nul p 4 B M
0000d0 7213 bb05 7000 76eb 08b4 13cd 0d73 845a
dc3 r enq ; nul p k v 4 bs M dc3 s cr Z eot
0000e0 0fd2 de83 be00 7d85 82e9 6600 b60f 88c6
R si etx ^ nul > enq } i stx nul f si 6 F bs
0000f0 ff64 6640 4489 0f04 d1b6 e2c1 8802 88e8
d del @ f ht D eot si 6 Q A b stx bs h bs
000100 40f4 4489 0f08 c2b6 e8c0 6602 0489 a166
t @ ht D bs si 6 B @ h stx f ht eot f !
000110 7c60 0966 75c0 664e 5ca1 667c d231 f766
` | f ht @ u N f ! \ | f 1 R f w
000120 8834 31d1 66d2 74f7 3b04 0844 377d c1fe
4 bs Q 1 R f w t eot ; D bs } 7 ~ A
000130 c588 c030 e8c1 0802 88c1 5ad0 c688 00bb
bs E 0 @ A h stx bs A bs P Z bs F ; nul
000140 8e70 31c3 b8db 0201 13cd 1e72 c38c 1e60
p so C 1 [ 8 soh stx M dc3 r rs ff C ` rs
000150 00b9 8e01 31db bff6 8000 c68e f3fc 1fa5
9 nul soh so [ 1 v ? nul nul so F | s % us
000160 ff61 5a26 be7c 7d80 03eb 8fbe e87d 0034
a del & Z | > nul } k etx > si } h 4 nul
000170 94be e87d 002e 18cd feeb 5247 4255 0020
> dc4 } h . nul M can k ~ G R U B sp nul => NOTE: we see text "GRUB" here meaning it has GRUB BootLoader
000180 6547 6d6f 4800 7261 2064 6944 6b73 5200
G e o m nul H a r d sp D i s k nul R => NOTE: we see text "geom Hard Disk Read Error". This is for printing Error by the bootloader.
000190 6165 0064 4520 7272 726f 0a0d bb00 0001
e a d nul sp E r r o r cr nl nul ; soh nul
0001a0 0eb4 10cd 3cac 7500 c3f4 0000 0000 0000 =>
4 so M dle , < nul u t C nul nul nul nul nul nul
0001b0 0000 0000 0000 0000 bc73 948d 0000 => Master boot pgm code ends here at 0x01bd (total 446 bytes), last 6 bytes are supposed to be 4 bytes of Disk Id/signature (optional) followed by null "0000" (order is incorrectly mentioned in the link above). Here disk signature is "948d bc73" from MSB to LSB, which is same as what was reported by fdisk cmd above = 0x948dbc73
0001be 2080 => These 16 bytes are partition 1. 1st byte is status. It's 80 for bootable partition and 00 for non bootable one. Here, this partition is bootable as status=80, other 3 are non bootable. It's bootable as this is the boot/EFI partition from windows.
0001c0 0021 dd07 3f1e 0800 0000 a000 000f => 5th byte is partition type/Id which indicates the type of parition and hence the FS. It is for use by OS. It's not standardized, and Linux OS just ignores it. Here it's 0x07, which is Microsoft/IBM FS (NTFS, exFAT, etc). size=524MB, FS=NTFS.
0001ce dd00 => These 16 bytes are partition 2.
0001d0 3f1f fe07 ffff a800 000f d000 09d6 => Partition type = 0x07. FS=NTFS
0001de fe00 => These 16 bytes are partition 3.
0001e0 ffff fe27 ffff 7800 0ec8 4000 001f => Partition type = 0x27, which is Winodows Recovery partition. size=1GB, FS=NTFS,
0001ee fe00 => These 16 bytes are partition 4.
0001f0 ffff fe05 ffff 7800 09e6 0000 04e2 => Partition type = 0x05, which is extended partition type. This has 2 logical partitons, as explained in fdisk cmd above
0001fe aa55 => these last 2 bytes of 512 bytes "55AA" marks end of MBR. It is used as a signature for MBR. 0x55 is 511th byte(addr 0x01fe), while 0xAA is 512th byte (addr 0x01ff)
After the master boot sector, the next sectors contain the actual partitions. Parttion Typpe/Id is specified for each partition, in the MBR. Value in this specifies the FS, and originated from IBM/Microsoft in early days of computer. However, it's ignored by Linux.
file: file cmd figures out the type of a file, since it's hard for a user in linux to know file type (since extensions have no meaning in linux)
file mbr.bin => displays contents of file that was dumped from MBR above
mbr.bin: x86 boot sector; => This says that this file is a boot sector file. It then knows how to read partition info from remaining bytes
partition 1: ID=0x7, active, starthead 32, startsector 2048, 1024000 sectors; => 1000K*0.5KB/sector=500MB
partition 2: ID=0x7, starthead 221, startsector 1026048, 165072896 sectors; => 78.7GB
partition 3: ID=0x27, starthead 254, startsector 248018944, 2048000 sectors; => 1GB. This is windows recovery partition.
partition 4: ID=0x5, starthead 254, startsector 166098944, 81920000 sectors,=> 80000K*0.5KB=40000MB=40GB
code offset 0x63, OEM-ID " м", Bytes/sector 190, sectors/cluster 124, reserved sectors 191, FATs 6, root entries 185, sectors 64514 (volumes <=32 MB) , Media descriptor 0xf3, sectors/FAT 20644, heads 6, hidden sectors 309755, sectors 2147991229 (volumes > 32 MB) , physical drive 0x7e, dos < 4.0 BootSector (0x0)
Now, we can repeat same exercise for external ssd hard disk connected via usb.
prompt> dd if=/dev/sdb of=ssd.bin bs=512 count=1=> assuming external hard disk is on /dev/sdb
prompt> hexdump ssd.bin => This is hexdump of 1st sector of this external ssd hard disk connected via usb. It shows 1st sector as boot sector, and shows the only partition it has (partition 1)
0000000 0000 0000 0000 0000 0000 0000 0000 0000
*
00001b0 0000 0000 0000 0000 9c01 078d 0000=> since there is no bootstrap code here, most entries are 0. Here disk signature is "078d 9c01" from MSB to LSB
0001be 2000 => these 16 bytes are partition 1. 1st byte=00 => it's non-bootable partition
00001c0 0021 fe07 ffff 0800 0000 9da7 773b
0001ce 0000 => partition 2 to partition 4 are all 0 => no more partitions
00001d0 0000 0000 0000 0000 0000 0000 0000 0000
*
00001f0 0000 0000 0000 0000 0000 0000 0000 aa55 => aa55 indicates end of MBR
file ssd.bin => shows file type of MBR file from external hard disk
ssd.bin: x86 boot sector; partition 1: ID=0x7, starthead 32, startsector 2048, 2000395687 sectors, extended partition table (last)\011, code offset 0x0 => size=2000395687*0.5KB/sector=1,000,197,843.5KB = 1TB
Filesystems (FS): It's a very important concept especially for linux, as everything in Linux is treated as file. A FS is the methods and data structures that an OS uses to keep track of files on a disk or partition; that is, the way the files are organized on the disk. FS are stored on partition of disk, and only 1 kind of FS is allowed on a particular partition. Thus, for ex, ext3 FS might be stored on partition 1 of HDD.
Unix filesystems are arranged as a big tree, with tree hier rooted at "/" or root.These files can be spread out over several devices and they can be remote or local file system. Linux supports numerous file system types.
These are the most common disk file systems (file systems suited for use on disk storage media).
1. Ext (extended FS): File systems started with Minix file systems, then was enhanced to Ext in 1992 (solved max parttition size problem). Newer versions came later as Ext2, Ext3, Ext4.
2. Ext2: enhanced version of Ext which supported inode modification, timestamps, etc.
3. Ext3: most popular enhanced version of ext2 supporting journalism. Std linux FS on most dist . Can support 2TB max file size, and 16TB max Partition size.
4. Ext4: Can support 16TB max file size, and 1EB max Partition size. Not supported on windows.
5. FAT32 (FAT=File Allocation System, proprietry windows FS): FAT was originally designed in 1970 for floppy drives. limits file size to 4.2GB, and parttition size to 2TB. Windows limits parttition size to 32GB. It's the only one that was supported on Windows systems staring from Windows95 in 1995. FAT32 is very popular on USB flash drives, where they come formatted with FAT32 FS, FAT32 FS is supported across all OS and all devices. However due to file size limitation, new devices come commonly formatted with NTFS.
6. NTFS (Windows New Tech (NT) FS, improved FAT32 proprietry windows FS): File size limit is 2TB, while partition size limit is 256TB. Most widely used as an alternative to FAT32, since NTFS-3G driver is provided with most linux dist for rd/wrt. Default for Ubuntu, Windows NT(1993), Windows 2000(XP onwards). partition tables on MBR (Master Boot Record) disks only support max partition size of 2TB, GPT volumes used to get to larger partition size.
7. exFAT: known as extended FAT, it was introduced in 2006 for USB flash drives and memory SD cards. It's used in places where NTFS is not feasible due to data structure overhead, but file sizes > 4GB are needed, so FAT32 is not possible. exFAT is now standard FS for SDXC cards > 32GB size. Although drivers for this are included in Linix Kernel 5.4, most linux distro don't have drivers for exFAT.
Misc:
7. ISO9660, Universal disc format(UDF) => optical disc file system for use in CD/DVD)
8. IBM DB2: database based file system. Instead of hier structured mgmt, files here are identified by rich metadata as type, topic, etc.
9. NFS :network file system in which one machine (client) requires access to data stored on another machine (NFS server). It's an open std, and was developed by SUN in 1984. server runs nfsd daemon on some port, client connects to that port and requests for data, which is passed to client by nfsd.
Making FS: Before a partition or disk can be used as a filesystem, it needs to be initialized, and the bookkeeping data structures need to be written to the disk. This process is called making a filesystem. Various data structures such as superblock, inode , data block, directory block , and indirection block are kept in various sectors of partition to allow Linux OS to retrieve files.
mkfs: This cmd used as a front end i/f for making various FS. Most linux distro use mkfs wrapper which calls mke2fs cmd, which does the real FS creation.
If we type mkfs and hit tab, we see all supported FS. i.e mkfs.ext2 creates ext2 FS, similarly for others. Their option differ slightly, so read doc for each of them when using.
ex: mkfs.ext2 -c /dev/fd0H1440 => -c searches for bad blocks, and initializes the bad block list.
ext2 FS has following structure: https://www.nongnu.org/ext2-doc/ext2.html
The best way to learn making a FS, and see what contents get created in it in various sectors, is to follow this link: https://www.howtogeek.com/443342/how-to-use-the-mkfs-command-on-linux/
ex: First create a image file with all 0 in it.Then create a FS by using mkfs.ext2 cmd. Then we can mount this image file just like any other FS.
> dd if=/dev/zero of=~/fs.img bs=1M count=50 => This creates a 50MB image file (with 1MB block size) called fs.img with all 0s in it.
> ls -al ~/fs.img => To show size of file
-rw-rw-r--. 1 kailash kailash 52428800 Dec 10 03:56 /home/kailash/fs.img => 50MB (=50*1024*1024Bytes=52428800 bytes)
> file ~/fs.img
/home/rakesh/fs.img: Linux rev 1.0 ext2 filesystem data (mounted or unclean), UUID=1ca78acd-c8c4-4e02-b4db-10d76395f252 => assigns a random 16 byte UUID
> mkfs.ext2 ~/fs.img => asks for confirmation, and then creates ext2 FS
> sudo mkdir /mnt/tmp_dev
> sudo mount ~/fs.img /mnt/tmp_dev => mounts the FS at /mnt/tmp_dev
> sudo cp ~/tmp.txt /mnt/tmp_dev/. => this copies tmp.txt file to newly mounted FS. This FS now behaves like any other dir on our hard disk
> hexdump -n2048 ~/fs.img => this dumps first 2048 bytes of FS. As can be seen, first 1024 bytes (block 0) are all 0, implying no boot record is present. then next 1024 bytes (block 1) has superblock, and then other data in rest of the blocks. The superblock is always located at byte offset 1024 from the beginning of the file, block device or partition formatted with Ext2 and later variants (Ext3, Ext4).
NOTE: Not all disks or partitions are used as filesystems. A swap partition, for example, will not have a filesystem on it. Linux boot floppies don't contain a filesystem, only the raw kernel. Not having a FS saves space. Raw disks are used to make image copies of them (using dd cmd).
Linux cmds:
1. lsblk = list block: lists information about all available block devices, however, it does not list information about RAM disks. Examples of block devices are a hard disk, flash drives, CD-ROM etc. So, this is a very useful cmd to see all available storgae devices.
$ lsblk => Shows all devices with thier major:minor number, RM=1 implies removable device, RO=1 implies readonly, TYPE implies whether it's disk, partition within a disk (part), rom memory, etc. We'll talk about mount point later. option -p prints full device path, as /dev/sda etc, else it only shows sda, sdb, etc.
prompt> lsblk ( on my dual boot linux/windows laptop)
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 119.2G 0 disk => sda usually is the main hard disk from which the OS boots. At Linux isntallation time, we usually divide this hard drive into separate partitions (over here in 6 partitions, some of which are Windows partitions, 4 are primary partitions, 2 are logical partition)
├─sda1 8:1 0 500M 0 part /run/media/rajesh/System => on Linux Mint, it's mounted on /media/rajesh/System. It's partition1, the Windows bootable partition in MBR table.
├─sda2 8:2 0 78.7G 0 part /run/media/rajesh/Windows => It's partiton2, where windows OS is insatlled
├─sda3 8:3 0 1000M 0 part => It's partition3, windows recovery partition
├─sda4 8:4 0 1K 0 part
├─sda5 8:5 0 1G 0 part /boot
└─sda6 8:6 0 38.1G 0 part
├─centos-root 253:0 0 34.2G 0 lvm /
└─centos-swap 253:1 0 3.9G 0 lvm [SWAP]
sr0 11:0 1 1024M 0 rom
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 931.5G 0 disk
├─sda1 8:1 0 260M 0 part /boot/efi => sda1, sda2 etc refer to partition within the disk sda
├─sda2 8:2 0 16M 0 part
├─sda3 8:3 0 246.7G 0 part /media/rajesh/Windows => This is windows partition, since we store Windows OS in a separate partiton than Linux OS, although both are residing on same physical disk.
├─sda4 8:4 0 980M 0 part
├─sda5 8:5 0 7.5G 0 part [SWAP] => here 7.5G of sda was allocated for SWAP and separate partition called "sda5" was created for it.
└─sda6 8:6 0 676.1G 0 part / => Here almost all of disk, 676G, was allocated to root file system "/". This is where everything in linux is stored, starting from root dir.
sdb 8:16 0 953.9G 0 disk => This is external ssd drive connected via usb
└─sdb1 8:17 0 953.9G 0 part /run/media/rajesh/samsung-ssd => It has 1 partition mounted on path shown
sr0 11:0 1 1024M 0 rom => This is another device "rom" (usually cd drive on computer) named as sr0. On almost all computers, we'll see atleast these 2 devices, sda and sr0.
lsblk -f => This lists the FS for each partition as well as the UUID of each device. UUID is 128 bit universal unique id, assigned
Mounting under linux:
In linux, you see devies under /dev/* (i.e lsblk devices listed as sda etc are in /dev/sda ....). However, if we try to access /dev/sda contents directly, we see that it's just a link, and we get a message "not a regular file". All these refer to a device file. For ex /dev/cdrom refers to CD ROM device. This is not the contents of whatever disc you might wish to insert into your optical drive, but rather it is a reference to the bit of hardware (and probably software drivers) that you might call on to show that to you. We can use "dd" cmd to read the contents of all of this device as a big string of bytes (known as disk image), however it's not very meaningful info. FS type tells us how to interpret that string of bytes. In linux, we use cmd "mount" to ask OS to figure out FS of this device and interpret contents based on that. mount attches FS found on some device to the big file tree, so that we can access it as if it's one tree at /somedir/. When you mount /dev/cdrom to some path in your tree you attach its contents to your file system.
ex: mount-t iso9660 /dev/cdrom /media/cdrom => mount cmd tells the system: "take this very long string of bytes that you have in /dev/cdrom, interpret it as a directory tree in the iso9660 format (boot sector of device/partition has necessary info to help interpret the FS on the device), and allow me to access it under the location /media/cdrom. We create this dir "/media/cdrom". It can be any dir anywhere on the system. By convention, FS are usually mounted in /media dir. -t is optional, as mount can itself figure out the FS by reading first few bytes of that device/partition,
mount cmd uses a lot of heuristics to figure out FS of device, which may not always work. That's why it's sometimes necessary to provide -t option to specify the FS explicitly.
ex: sudo umount /media/cdrom => This unmounts whatever was mounted at mount point "/media/cdrom, so that the FS is no more acccessible
ex: mount => shows all FS mounted on this device (FS starting at /)
/dev/hda1 on / type ext3 (rw,acl,user_xattr) => shows that / FS is mounted on hda1 physicaly attached to m/c (hard drive 1), i.e all of partition /dev/sda1 is a FS accessible with "/"
/dev/cciss/c0d0 on /tmp type ext3 (rw,acl,user_xattr) => /tmp is mounted on another drive c0d0 physicaly attached to m/c.
duke4.abc.com:/vol/fvol528/kagr on /home/rakesh type nfs (rw, ..) => /home/rakesh is mounted on diff m/c duke4.abc.com. This is nfs FS (as it's on diff m/c)
ex: mount => on all laptop shows a lot of FS mounted. imp ones are:
/dev/sda1 on /run/media/rakesh/System type fuseblk (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,blksize=4096)
/dev/sda2 on /run/media/rakesh/Windows type fuseblk (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,blksize=4096)
/dev/sda5 on /boot type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/centos-root on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota) => /centos-root is mounted on Root dir "/".
/dev/mmcblk0p1 on /media/sd_card type fuseblk (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,blksize=4096) => This is sd card that I mounted by using "mount" cmd
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) => shows proc is mounted on /proc (even though /roc is not a FS)
Partitioning HD:
Usually we want to partition our HD into sevral partitions to keep them safe from each other, i.e crash in one partition won't damage the other partition. General practice is to have separate partitions for /, /boot, and swap. You cannot create separate partitions for the following directories: /bin, /etc, /dev, /initrd, /lib, and /sbin. The contents of these directories are required at bootup and must always be part of the / partition.
It is also recommended that you create separate partitions for /var and /tmp. This is because both directories typically have data that is constantly changing. Not creating separate partitions for these filesystems puts you at risk of having log file fill up our / partition.
-------
disk usage cmds
1. df => shows free disk space (df=disk free)
df -kT =>shows amount of diskspace available on file system (for all of the filesystem). -T option shows file system type too. -k shows size in 1K blocks (default in RHEL)
Ex: /home/rajesh> df -T | grep kagrawal => df shows all filesystems, to see diskspace for kagrawal only, we do a grep
near13.srv.abc.com:/vol/fvol156/kagrawal => implies kagrawal has home dir on this server.
nfs 1024000 512416 511584 51% /home/kagrawal
#nfs shows that file system type is nfs => shown only with -T option
#the first column is space allocated yo your unix home dir the second column is space you have used
#the third column is space which is available to you and the percentage you have used
Ex: /home/rajesh> df . => shows usage for current dir. To see usage for some other dir, type: df dir1/dir2
Ex: /proj/dsp/Testbenches $ df .
Filesystem 1K-blocks Used Available Use% Mounted on
coupe.abc.com:/vol/fvol32/proj1 => shows the root dir for the current dir and where it's mounted
5242880 5242848 32 100% /proj/name2 => size is 5.2GB
ex: /home/rajesh> df => on a laptop, this is what it looks like
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/centos-root 35789748 18501240 17288508 52% / => centOS-root FS is mounted on /
/dev/sda5 1038336 174224 864112 17% /boot
tmpfs 770336 68 770268 1% /run/user/1000
/dev/sda1 511996 54848 457148 11% /run/media/kailash/System
/dev/sda2 82536444 44197628 38338816 54% /run/media/kailash/Windows
/dev/mmcblk0p1 62504960 1622400 60882560 3% /media/sd_card
2. du => disk usage
du -skh dir1/* => Summarize disk usage of each FILE, recursively for directories. Reports for current dir and below.
-k says report in block_size=1K, while -h says to report it in human readable format (as 234M, etc)
-s summarizes directories (reports only 1 line with total size for each matching dir)
du -sk * .??* | sort -n => to sort all files/dir (including dot files) in your current dir
du -ch | grep total => lists 1 line with total size of dir
Installing File system drivers on Linux: Each of these devices or partitons within a device can have it's own separate FS. However, the OS has to know how to read/wrt these FS, else it may not work on partitons with different FS, than what it knows. Usually Linux OS know ext and FAT/NTFS, and work pretty well with them. Since most devices already come formatted with NTFS, Linux has no problems readingwriting to these. However, we may need to install drivers for certain FS if they are not installed by default.
installing NTFS drivers:
sudo yum install epel-release => Nux repository depends on the EPEL software repository. If the EPEL repository is not enabled, enable it using this cmd
sudo yum install ntfs-3g -y; => available in yum repo
Once installed, linux OS should automatically be able to mount the NTFS drive. If not, we can always manually mount by using mount cmd
installing exFAT drivers:
Most linux distro don't provide support for exFAT FS. So, we can get an error "cannot mount exFAT" when we insert a device which has exFAT FS. To be able to mount exFAT filesystem on CentOS you’ll need to install the free FUSE exFAT module and tools which provide a full-featured exFAT file system implementation for Unix-like systems. However, yum repository for CentOS don't haave these exFAT pacakages available. However, these pkg are available from Nux Dextop (3rd party) repository, and installed using yum with steps as below:
sudo yum install epel-release => Nux repository depends on the EPEL software repository. If the EPEL repository is not enabled, enable it using this cmd
sudo rpm -v --import http://li.nux.ro/download/nux/RPM-GPG-KEY-nux.ro => import the repository GPG key
sudo rpm -Uvh http://li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-5.el7.nux.noarch.rpm => enable the Nux repository by installing rpm pkg
sudo yum install exfat-utils fuse-exfat => now install the exfat-fuse and exfat-utils packages from nux repo
Once installed, linux OS should automatically be able to mount the exFAT drive. If not (i.e we don't see any response on inserting SD card), we can always manually mount by using mount cmd:
> lsblk => This shows that SD acrd is being recognized by system, but it's not able to read it's contents (as we don't see it under devices in gui)
mmcblk0 179:0 0 59.6G 0 disk
└─mmcblk0p1 179:1 0 59.6G 0 part /media/sd_card
> sudo mount -t exfat /dev/mmcblk0p1 /media/sd_card => As soon as we run this cmd, we see SD card shows up under devices in gui (sometimes it's necessary to specify FS using -t, as OS may still have trouble identifying the FS of the device, even though the FS drivers might be installed, this is due to various heuristics being used to figure out the file system)
FUSE exfat 1.3.0
> sudo umount /media/sd_card => once done, we should unmount the sd card, before ejecting it out.
---------------------------