Differences

This shows you the differences between two versions of the page.

--- drives [2015/07/14 11:32]
slackermedia
+++ drives [2021/06/03 19:48] (current)
@@ Line 1: / Line 1: @@
-[[{arrowp.png|border:0;background:none;width:0;display:inline-block;position:absolute;top:0;left:0;}dotdesktop|]]
+======Hard Drives and Linux======
-[[{arrown.png|border:0;background:none;width:0;display:inline-block;position:absolute;top:0;margin-left:2.5em;}mimetypes|]]
+Linux does hard drives astonishingly well. So well, in fact, that most hard drive rescue services default to Linux because its toolset is so rich.
-======Hard Drives and Linux======
+<WRAP info>
+**TL;DR** \\
+If you do not want to go into hard drive forensics for a living, you may be most interested in:
+\\
+  * [[#kvpm|kvpm]], a graphical hard drive manager
+  * [[#udf|udf]], a truly universal format that you should use for external drives under 2TB in size
+</WRAP>
 Hard drives can broadly be classified as either internal or external. Internal drives tend to be easier to manage, because they are not portable; they are integrated in some way with your operating system, even if it is just extra storage space that you use on an as-needed basis.
-External hard drives are more complex, but only because they tend to move around. You may not move it often, or you might literally have it on your keychain. The bottom line is that external hard drives tend to visit not just one computer, and not even just one operating system, and as a multimedia artist, you are likely to have to deal with that frequently.
+Here is a profile of your internal drive:
-=====Incoming Drives=====
+=====Internal Drives=====
-The good news is that drives coming to your studio can, more often than not, be read by Slackware. There are only a few caveats:
+The drive inside your Linux computer is either an SSD or a traditional spinning disk drive, or you possibly are using both. Your drive(s) can be seen as one pool of available disk space by Linux, or conversely one drive can be seen as several different, artificially-separated disks even though it's actually one physical drives. Computers are able to perform this perceptual magick through "partitions": imaginary boundaries that an OS agrees to respect as an (air-quotes) Different Disk than the rest of the drive (or, when disks actually //are// physically separate, the computer can see all of the disks as [air-quotes] One Big Drive).
-Macs mostly use HFS+ formatted drives. HFS+ is a relatively volatile file system; it is rather prone to failure and corruption, but luckily tools to repair them tend to be pretty good.
+There is nothing special about hard drives in terms of storage except that they are relatively fast and efficient in managing a lot of information. Abstractly, though, they are no different that, for instance, a tape archive (or a quarter-inch tape, or similar media). In fact, hard drives can be used directly as, basically, a tape drive.
+<WRAP alert>
+Do not try this at home, you could possibly erase important data.
+</WRAP>
-=====Mac Compatible=====
+Assume we have attached a drive and it appears as ''/dev/sdx'' on our computer. Raw data can be written to the drive without any formatting, and independent of any file system:
-If you have a drive that claims to be "Mac compatible" and you want to use it on both Linux and Mac OS, then it is probably formatted as **HFS+**. HFS+, in addition to being one of the least stable file systems on the market, is crafted intentionally to be incompatible with other systems; in order to write //to// HFS+ from Linux, you must disable journaling on the drive.
+<code>
+# echo 'Do not try this at home unless you really know what you are doing.'
+# echo "hello" | dd of=/dev/sdx
+# head -c{1..5} /dev/sdx
+hello#
+</code>
-You can, however, read //from// it without doing anything to the drive.
+The data "hello" was written as raw bytes on the drive. It was not written as a file, so if you plug the drive into another computer, it will not look like there is anything on the drive (if the computer even understands how to mount the drive), but the string "hello" is still on the drive. It's just written as raw data.
-Disabling the journal on an HFS+ drive must be done from within Mac OS. If you do not have access to Mac OS, then you cannot write to the HFS+ drive, **do not** attempt to write to the drive; you could do damage to the files.
+This is actually how data was stored for a very long time,but eventually the disadvantages became too much to bear and someone invented a "file system" which does exactly what its name suggests: it creates a //system// for managing //files//. With a file system, you don't need to know the exact byte count to retrieve your data off of a hard drive; the drive itself remembers that for you. Rather than reading raw data, bytes 1 to 5, for the string "hello", we could instead just make a request to our drive for our "hello" file, and no matter where that file was or how many times we have revised and added to it, the computer can quickly and easily find it on the drive and show it to us.
-If you are stuck with a "Mac compatible" drive and want to use it as an active "normal" drive in your studio, with full read and write capabilities, then your best bet is to copy all of the data off of the drive, [[#format|re-format it]], and then copy the data back onto it.
+Your Linux drive is running an open source file system, probably either ''ext4'' or ''jfs''. These file systems, although open source and free for anyone (person or corporation) to use, the major closed source operating systems decline to include support for them. This means that the drives are recognised as being blank or simple "un-readable" by another OS. It's advantageous to use the file systems nevertheless, because aside from being some of the most stable on the market, they have features that help Linux run efficiently.
-If you require the drive to remain compatible with a Mac as well as Linux (and, as a side benefit, Windows), then use the [[#udf|UDF]] format.
+Once a drive has a file system, it can be mounted and used by your operating system. On Linux, when you attach a drive to your computer (internally or externally), the drive (upon detection) is assigned a node in the ''/dev'' directory. Drive nodes are dynamically created on an as-needed basis. Internal drives are assigned nodes as they are detected by the system, which is //usually// predictable after you notice the pattern but really it depends on the motherboard and which slots your drives are plugged into.
-If these are not valid options for you, then use Mac OS to disable the journal on the drive. Mac OS may later re-activate the journal without notice, so you may have to do this often.
+The first detected drive is assigned ''/dev/sda''. The "sd" prefix denotes the type of drive it is (actually it's historically inaccurate, but as a revisionist you can think of "sd" meaning "Sata Disk"), and the "a" is the first letter of many alphabets. That node represents the physical drive itself, which is different that what's //on// the drive. If a partition is found on that drive (most drives have at least one partition), then it gets a node ''/dev/sda1''. If there is yet another partition, then it would be assigned ''/dev/sda2''.
-=====Windows Compatible=====
+The second drive found gets ''/dev/sdb'' and its partition ''/dev/sdb1'', and so on.
-If you have a drive that claims to be "Windows compatible" and you want to use it on both Linux and Windows, then it is probably formatted as **NTFS** or **ExFAT**, with a small chance of it being **FAT32**.
+These nodes only represent that drives are attached. They are not directories that you can open and view data in, they are metadata about your system.
-None of these filesystems are particularly //good// but they are all well-supported by Linux (not, however, by Mac OS).
+The data on the drives is used by your Linux system, and internal drives are usually automatically mounted by Linux because those drives appear in the file ''/etc/fstab'' as drives that are to be mounted upon boot.
-If you are stuck with a "Windows compatible" drive and want to use it as an active "normal" drive in your studio, you probably can, as is. There are several inconveniences that you may notice (file size limitations, permission issues), but as external drives go, everything should basically work as expected.
-A better option is to copy all of the data off the drive, re-format it as a [[#udf|UDF]] file system, and put all of the data back on. UDF brings along with it several benefits, including Window and Mac compatibility, and elimination of the quirks of Window hard drive formats.
+The take away points from this overview of your internal Linux drives are:
-=====Linux Compatible=====
+  * Use ''ext4'' or ''jfs'' or any Linux-native format for drives that are directly a part of your operating system to avoid unexpected results in how you OS works.
+  * Drives get nodes in ''/dev'' based on when they were detected, and how many partitions exist on them.
+  * Internal drives are automatically mounted by the system only if they are listed in ''/etc/fstab''.
-If you are using a drive just on Linux systems (recommended, but not always possible), then you can keep your drives in a native Linux format. The only practical advantage of using a native Linux format on an external drive is that they happen to be very robust file systems. They are comparatively difficult to corrupt or break, they are fast, well-designed and maintained, case-sensitive, and have very few limitations in a pragmatic sense.
+There is no quiz on this, but now you know.
-The immediate //disadvantage// of using native formats for external drives on Linux is that none of them were really designed for external use. That is, most of them assume that the drive is **inside** the computer. This tends to not matter much until you start swapping drives with other Linux users or computers, at which point file permissions can become a problem.
-Permissions can be managed for external drives, but you have to be intentional and mindful of it.
+====Formatting Internal Drives====
-A better option is to copy all of the data off the drive, re-format it as a [[#udf|UDF]] file system, and put all of the data back on. UDF brings along with it several benefits, including Window and Mac compatibility, and elimination of the quirks of Window hard drive formats.
+If you purchase an additional internal drive for your Linux computer, you probably should be using a Linux-native file system. There is probably no advantage in using a non-native file system, because it's an internal drive; other computers are not plugging into in asking to use data off of it (and if they are, they are doing so over your network, and TCP/IP makes basically everything universal).
+(The possible exception here is that you are getting a large drive that you want to use as shared storage space between a drive running Linux and a drive running some other OS. This is not recommended, but if you do this, then treat the drive as an [[#udf|external drive]]. Slackermedia does not support this, because the other OS adds a significant variable to how your data is being managed, so you are, respectfully, on your own!)
+{{anchor:intformat}}
+**To format an internal drive for use with Linux:**
+<WRAP indent>
+Determine the device node of the drive you are going to format by first seeing what drives are already part of your system.
+Use the ''lsblk'' command to view all block devices (hard drives) attached to your computer. If the ''lsblk'' command is not clear to you, or you want to double-check what it tells you, you can investigate further:
-Partly as an answer to the filesystem problem, a few Standards groups came up with UDF, the Universal Disk Format. It was mostly intended as the replacement for ISO-9660, and did become the official filesystem for CD-RW, DVD-RW, and Blu-Ray.
+<code>
+$ mount | egrep '.*sd.*'
+/dev/sda2 on / type jfs (rw)
+/dev/sda1 on /boot type vfat (rw,fmask=177,dmask=077)
+/dev/sdb1 on /home type jfs (rw)
+</code>
-The down side is that it does not use journaling, making data recovery after a crash or accidental unplugging a little riskier. It does not use partitions, but that is not usually an issue for an external drive.
+In this example, there are two drives already in use by the system: one being used as the boot and system drive (''sda''), and another (''sdb'') used exclusively for the home directory.
-The good news is that it is open source, can use UTF-8 filenames that are as long as 255 bytes, file sizes and filesystem sizes of 2TB. Like FAT, UDF does not bother with permissions, making it ideal for external drives on Unix. At the very worst, even considering some of the features left out of UDF, it is a better and more flexible option than FAT, and has no patent issues to contend with (Micrososft sometimes sues companies for using random features in FAT).
+Compare that list to what the computer actually has attached:
+<code>
+$ ls -1 /dev/sd*
+sda
+sda1
+sda2
+sdb
+sdb1
+sdc
+sdc1
+</code>
+In this example, there is a third drive not in use by the system, labelled ''sdc''. This is the new drive that needs formatting. Notice that it does have a partition on it already, but that's only because most all drives purchased from a modern computer store are pre-formatted, presumably so that users do not have to learn about formatting themselves.
+To prevent copy-paste errors, this article uses the ''/dev/sdX'' designation.
+Keep in mind that your drive in real life could be anything from ''sdb'' to ''sdz'', depending on how many actual drives you have plugged in. Usually, the first drive you plug in is going to come up as ''sdb'' because ''sda'' is the drive running your computer, but be aware of your actual setup and use your head. //You do not want to format the wrong drive.//
+If you are unsure that you are targeting at the correct drive, mount it and have a look at what's on it:
+<code>
+$ su -c 'mount /dev/sdX1 /mnt/hd'
+$ cd /mnt/hd
+$ ls
+.
+..
+Acme Drivers
+Acme Backup Pro Plus
+$ df -h /mnt/hd | awk '{print $2}'
+Size
+.8T
+</code>
+In this example, the drive is mounted at ''/mnt/hd'' (a pre-existing directory for quickly mounting drives on Slackware) and is shown to contain basically nothing, if we ignore the obligatory drivers and bloatware bundled by the vendor on the drive.
+Confirming the size of the drive provides further reinforcement: yes, this really is the 3TB drive you have purchased.
+With that settled, unmount (with the ''umount'' [sic] command) the drive so that you can perform surgery on it:
+<code>
+$ cd ~
+$ su -c 'umount /dev/sdX*'
+</code>
+Create a fresh partition table on the device. A partition table just tells a computer what kind of partition to look for when reading the drive. Operations like re-formatting entire drives justifiably require root permissions:
+<code>
+$ su
+# parted /dev/sdX mklabel gpt
+</code>
+Historically, the de facto partition label was ''msdos'' because that was (and still is) the most ubiquitous; msdos-style partitioning is universally recognised. For drives larger than 2TB, a ''gpt'' partition label must be used, because ''msdos'' partition labels cannot scale to 2TB.
+Very little actually rides on this, it's just a matter of whose identifier you want to use. It has nothing to do with how your data is secured or kept, it's just an identifier so that the computer knows what to look for when it mounts a drive.
+Next, find out how big your disk is:
+<code>
+# parted /dev/sdX print | grep Disk
+</code>
+For the sake of this example, assume the drive is 2834020 MB (2.8TB) in size.
+Create a partition that spans the whole drive:
+<code>
+# parted /dev/sdX mkpart primary 1 2834020
+</code>
+This creates a partition that starts at the first megabyte (1) and spans all the way until the 2,834,020th megabyte.
+<WRAP important>
+Do not start your partition at the 0th megabyte or you will get the error ''Warning: The resulting partition is not properly aligned for best performance''. Start your partition at 1. You are sacrificing 1024 bytes, but it's worth it.
+</WRAP>
+Now the drive has a partition; all it needs now is a file system. Remember, a partition is indicated by a number trailing the device node. In this example, the location of your new partition is ''/dev/sdX1''.
+For a Linux native drive, use ''ext4'':
+<code>
+# mkfs.ext4 -L penguindrive /dev/sdX1
+</code>
+Or ''jfs'':
+<code>
+# mkfs.jfs -L penguindrive /dev/sdX1
+</code>
+The drive is now formatted. It's best to create a permanent, standard place for it on your system. Assuming that it is going to be used as extra storage space:
+<code>
+# mkdir /storage
+</code>
+To make the drive automatically mount, add it to ''/etc/fstab''. For example, to have it mount as extra storage at boot time, add a line like this:
+<code>
+LABEL=penguindrive   /storage  jfs  rw  1 1
+</code>
+If you do not know the label of your drive, use ''lsblk -f''. If your drive has no label, then use the PARTUUID (use UUID if you partition is ''msdos'') instead:
+<code>
+LABEL=penguindrive   /storage  jfs  rw  1 1
+PARTUUID=7280201c-fc5d-40f2-a9b2-466611d3d49e /storage  jfs  rw  0  2
+</code>
+And finally, mount the drive by mounting all drives listed in ''/etc/fstab'':
+<code>
+mount -a
+</code>
+</WRAP>
+=====External Drives=====
+External hard drives are more complex, but only because they tend to move around. You may not move it often, or you might literally have it on your keychain and take it everwhere with you. The bottom line is that external hard drives tend to visit not just one computer, indeed not even just one operating system, and as a multimedia artist, you are likely to have to deal with that frequently.
+====Incoming Drives====
+In this context, an "incoming drives" is a drive you do not own but need to work with.
+Good news: drives coming to you from clients or collaborators, more often than not, are plug-and-play **for reading** on Linux.
+There are only a few caveats:
+  * Macs mostly use HFS+ formatted drives. HFS+ is a relatively volatile file system; it is rather prone to failure and corruption, but luckily tools to repair them tend to be pretty good. Even so, it is safer to minimise contact with HFS+ drives; get the data you need from the drive, and disconnect it.
+  * Windows formats generally pose no issue; it is mostly all reverse-engineered by now, and so it's mostly a matter of attaching the drive and reading data from it as you would any other drive.
+**Writing** to a foreign drive is less simple.
+**Mac Drives**
+<WRAP indent>
+HFS+ drives are crafted to be intentionally incompatible with other systems (the journal is not open source or even open spec); in order to write //to// HFS+ from Linux, you **must** disable journaling on the drive first.
+Disabling the journal on an HFS+ drive must be done from within Mac OS. If you do not have access to Mac OS, then you cannot write to the HFS+ drive, **do not** attempt to write to the drive; you could do serious damage to the drive's file system.
+If you are stuck with a "Mac compatible" drive and want to use it as an active "normal" drive in your studio, with full read and write capabilities, then your best bet is to copy all of the data off of the drive, [[#extformat|re-format it]], and then copy the data back onto it.
+If you require the drive to remain compatible with a Mac as well as Linux (and, as a side benefit, Windows), then use the [[#udf|UDF]] format.
+If these are not valid options for you, then use Mac OS to disable the journal on the drive. Mac OS may later re-activate the journal without notice, so you may have to do this often.
+</WRAP>
+**Windows Drives**
+<WRAP indent>
+Most Windows drive formats are well reverse-engineered at this point and can be both read from and written to without any special action taken on your part.
+</WRAP>
+{{anchor:udf}}
+=====UDF: the Universal Universal Disk Format=====
+Partly as an answer to the problem of having no universally-accepted file system and partly out of the need for a good file system, a few Standards groups came up with **UDF**, the Universal Disk Format. It was mostly intended as the replacement for ISO-9660, and did become the official file system for CD-RW, DVD-RW, and Blu-Ray.
+There are two significant disadvantages of UDF:
+  * Maximum volume size is 2TB (on hard drives).
+  * It does not use journaling, making data recovery after a crash or accidental unplugging a little riskier.
+It is open source, it can use UTF-8 filenames that are as long as 255 bytes, file sizes and file system sizes of 2TB. UDF does not bother with permissions, making it ideal for external drives.
 Since it was primarily intended for optical media, creating a UDF volume is different from formatting a drive for any other filesystem.
-    The drive being formatted must have no partitions on it. This is entirely unlike any other filesystem, but it is necessary for some operating systems to accurately detect the UDF filesystem.
+Slackermedia recommends the UDF format for any external drive that you intend to use with more than just your own computer. It avoids both file permission frustration and file size limitations, but maintains all the other UNIX features that one would expect from a drive. It works well on thumbdrives as well as tradition drives. By being a universal format, it ensures that the data that matters to you the most is always available to you, regardless of what OS you happen to have on hand.
-    To get rid of the existing partition on a drive, zero out the first 4096 bytes of the drive.
-    dd if=/dev/zero of=/dev/sdx bs=512 count=4096
+====Formatting a Drive for UDF====
-    Note that the bytesize (bs) is not flexible. It must be 512.
+The drive being formatted must have no partitions on it. This is entirely unlike any other filesystem, but it is necessary for some operating systems to accurately detect the UDF filesystem.
-    Next, find out the block count for your drives:
+To get rid of the existing partition on a drive, zero out the first 4096 bytes of the drive. This effectivel **erases all of the data on the drive** so you should be doing this only on an empty drive or a drive with data on it that you do not wish to use.
-    $ df -i /dev/sdx
+<code>
-    Inodes   IUsed    IFree
+# dd if=/dev/zero of=/dev/sdx bs=512 count=4096
-    2040230   619    2039611
+</code>
-    Finally, create the filesystem so that it spans the entire drive.
+Note that the bytesize (bs) is not flexible. It must be 512.
-    mkudffs --blocksize=512 \
+Create the filesystem so that it spans the entire drive:
-    --udfrev=0x0201 \
-    --lvid="myUdfDrive" \
+<code>
-    --vid="myUdfDrive" \
+mkudffs --utf8 \
-    --media-type=hd --utf8 \
+--blocksize=512 \
-    /dev/sdx || echo "fail"
+--udfrev=0x0201 \
+--lvid="penguindf" \
+--vid="penguindf" \
+--media-type=hd \
+/dev/sdx || echo "failed"
+</code>
 Now you can mount and use the drive on any platform.
-[EOF]
+{{anchor:extformat}}
+====Formatting Drives on Linux====
+This section reiterates the usual steps in formatting a drive, with an emphasis on external drives (read the section on [[#intformat|formatting internal drives]] if you are adding a drive to you computer internally). If you are formatting to UDF (you probably should be), read the section on [[#udf|UDF]].
+<WRAP indent>
+**1.** Identify the drive's device node.
+With the drive //unplugged// from the computer:
+<code>
+# ls /dev/sd*
+sda  sda1  sdb  sdb1
+</code>
+Then plug the drive in and do the same command again:
+<code>
+# ls /dev/sd*
+sda  sda1  sdb  sdb1  sdc  sdc1
+</code>
+The new node is the drive you have just attached.
+**2.** Unmount the drive.
+It probably isn't mounted, but just in case...
+<code>
+# umount /dev/sdX*
+</code>
+**3.** Create a fresh partition table:
+<WRAP important>
+From this point on, the device ''sdx'' (a rare drive node, since it would require you to have 23 drives attached to your computer) is used to protect people from copying and pasting commands and unintentionally erasing a real drive. Replace ''sdx'' with the actual label of your target drive.
+</WRAP>
+<code>
+# parted /dev/sdx mklabel msdos
+</code>
+Or, for drives lager than 2TB:
+<code>
+# parted /dev/sdx mklabel gpt
+</code>
+**4.** Get the drive size:
+<code>
+# parted /dev/sdx unit MB print | grep Disk
+Disk /dev/sdx: 63417MB
+Disk Flags:
+</code>
+**5.** Create a partition spanning from 1 MB to the last available MB (which you will have gotten from the ''parted print'' command):
+<code>
+# parted /dev/sdx mkpart primary 1 63417
+</code>
+**6.** Put a file system on the partition you just created.
+<code>
+# mkfs.jfs -L rockhopper /dev/sdx
+</code>
+Or for an ''ext4'' file system (more common across Linuxes)
+<code>
+# mkfs.ext4 -L rockhopper /dev/sdx
+</code>
+Also consider using [[#udf|UDF]] for drives under 2TB.
+**7.** Unplug the drive and plug it back in.
+</WRAP>
+{{anchor:kvpm}}
+====KVPM Volume and Partition Manager====
+[[https://sourceforge.net/projects/kvpm/|KVPM]] is a graphical interface to the ''parted'' command and some related hard drive tools.
+Install KVPM from [[http://slackbuilds.org]]
+Launch **kvpm** from the **K Menu**.
+Once kvpm has launched, look at the list of drives for one that matches your drive in size and vendor. For safety purposes, you should **not** have any other external drives plugged into your computer at this point, so only the internal hard drive(s) and you target drive should be visible.
+Right-click on your target drive and choose **Add disk partition**.
+Accept the default options so that you are using all available space on the drive.
+[{{ kvpm_part.png | Create a partition on a drive.}}]
+One the new partition appears, right-click on the partition inside the drive and choose **Filesystem operations** → **Make filesystem**.
+[{{ kvpm_mkfs.png | Create a filesystem in the partition on your drive. }}]
+Use the filesystem type of your choice, give the drive a name (or "label"), and click **OK**.
+[{{ kvpm_fs.png | Name your drive and set it loose. }}]
+<WRAP centeralign>
+<wrap fa>[[dotdesktop|R]]</wrap> <wrap fa>[[start|S]]</wrap> <wrap fa>[[mimetypes|Q]]</wrap>
+</WRAP>
-[[{arrown.png|border:0;background:none;width:0;display:inline-block;float:right;}mimetypes|]][[{arrowp.png|border:0;background:none;width:0;float:right;}dotdesktop|]]