Monday, July 11, 2005

Testing Tape Drives in Linux

This section will cover the following topics:
1. The different ways in which Linux treats tape devices,
2. The different backup utilities inbuilt / easily available,
3. The different testing routines available, and
4. How to write your own tape testing software.

1. The different ways Linux treats tape drives:

One needs to know that there is a basic difference between the way tape drives are treated in Windows and Linux. Windows allows access to tape drives through direct interrupts. Whereas Linux makes it a little easier by treating them as files on which normal file operations can be written to generate easy testing routines. Normally, a tape drive is detected as "/dev/st0" if it is connected to a single standalone systems. It is treated as a character device, meaning that it cannot be mounted. Remember, only drives in which the sectors can be randomly accessed, can be mounted. That is because, a file system can be built on them and data stored randomly by using the File Allocation Table. In character / streaming devices, file system cannot be written. Neither is it possible to access them randomly. Recently some softwares such as DATMAN are available, which make it possible to write and read randomly on the tape devices. Also other file systems such as "RATFS"( or Random Access Tape File System ), are under design.

Having known the basics of how a Linux system treats a tape device, it is often necessary to know that a lot of drivers have been written to use these devices. Some of them are a. st and b. sg. The first one detects the tape device as a character device which can be manipulated using different backup softwares such as "tar", "cpio" or "dump". Utilities such as mt have been written upon those drivers. The other one, ie., "sg" ( standing for SCSI Generic ), is a little complex one, in the sense that it makes it easier for a user to generate Windows assembly like calls for diagnostic operations, inquiry operations, reading and editing of log pages etc. This is more important where it is necessary to deal with the tape device on a more basic level than when using backup or utility softwares such as mt.

2. The different backup utilities inbuilt / easily available:

Interms of the backup utilities that are available for Linux, the abundance is clearly very well known, but the only characteristics that almost all of them lack, is the speed of data transfer. Almost all utilities are slow in data transfer. Some modifications were made in earlier days , like development of softwares such as "star" ( which made up for "tar"'s slow backup rate ), and dump. The following softwares can be summed up for usage in Linux, along with their features:

1. tar -- The most well known, easily useable, and universal program
2. star -- Modification of tar for higher data transfer rate
3. cpio -- Stands for "Copy Input to Output". Helpful while copying data from one tape to another.
4. pax -- Quite feasable backup software
5. dump -- System backup provided in many linux distributions. Provides proper log file
giving all the necessary related details, can be scheduled easily, proper media
maintainance / administration operations can be performed, and network backups can be
6. dd -- Not basically a backup software, it is used to just "directly dump" any selected
input file to the tape drive with any format, for later recovery in the same format.
Block sizes can be varied during data transfer, tapes can be blanked properly, and
custom data can be written using other options such as count, bs etc.
7. gtar -- Gnome Tar. Provides, verification of data, log files etc.
8. ctar -- Another variant. I have personally not checked, but something must be new
9. amanda backup -- A very versatile, advanced network backup software.
10. KDat -- Good graphical front end.

Third party utilities:

1. Arc Serve for Linux
2. NovaBackup for Linux
3. Microlite Backup Edge ( provides One Button Disaster Recovery )
4. Lone-tar
5. BRU ( Backup / Restore Utility )
6. Burt's backup Utility
7. Uni Backup
8. Archeia - A very good backup softwares

3. The different testing routines available:

In Linux, almost all the ways of testing can be implemented. The different utilities for testing tape devices are --

1. SCU - SCSI Command Utility
2. dt - Developed along with SCU
3. dd
4. mt

SCU is a very very complex utility that can perform many different tape operations such as

a. Reading, and editing Log pages and Mode pages of tape drives
b. Block, pattern and loop manipulated read and write operations
c. Sending inquiry commands
d. Very precise estimation and maintenance of speed, time and data.
e. tape positioning based testing

It can be obtained online for download.

dt is another software which is as versatile as SCU, and can also be obtained online along with SCU.

dd is a good data transfer utility which helps transfer data with different block sizes, formats, and different capacities ( which can be precisely specified ).

mt is a very good tape drive testing utility which can be used to - format, position, erase and long test-runs on tape drives. It provides many ways of handling driver specific operations.

4. Your own Tape testing routines:

I have shown an example of each utility that i have mentioned in the previous section, though there are hundreds of other variants which can be written as shell scripts etc.

SCU allows batch testing using a configuration file. This can be invoked using
"-s" arguement. The following commands written in SCU Configuration file perform the activities mentioned beside them.

Example test.scu:

scan edt /* Scan for devices */
show edt /* Show the devices detected ( whether they are detected by OS or not */
set log log_file_name /* set the log page for storing operation log */
set statistics on /* Make it verbose */
show inquiry pages /* Show inquiry pages displaying vendor specific pages, serial number, etc */
write media bs 60k limit 10000m /* write 10 GB of data to tape with block size of
60 KB and with default pattern "c39" */
exit /* exit the utility */

Run the above batch file with command : scu -s test.scu

To perform the same operation as above, the following command works well.

dt of=/dev/st0 bs=60k limit=10000m log=log_file_name

In the examples basic functions such as retentioning, quick and long erasing, and positioning operations are specified. "-f" arguement is used to specify the file name ( here /dev/st0)

Quick Erase: mt -f /dev/st0 weof 2

weof - write end of file
2 - no. of end_of_files to be written

Long Erase: mt -f /dev/st0 erase

Make partition: mt -f /dev/st0 stoptions can-partitions
indicates that tape drive will be used for making partitions.

mt -f /dev/st0 mkpartition <>
for making partition.

Spacing: mt -f /dev/st0 fsf <> ( forward spacing by count files )
mt -f /dev/st0 bsf <> ( reverse spacing )
mt -f /dev/st0 fsr <>
mt -f /dev/st0 bsr <> ( same as above, but records here )
mt -f /dev/st0 eom ( space to end of recorded media )
mt -f /dev/st0 rewind ( space to beginning )

Seeking: mt -f /dev/st0 seek <> ( seek to given posiiton

Can be used to write data of desired pattern, at desired block size and the specified no. of times as mentioned below:

dd if=/dev/zero of=/dev/st0 bs=10k count=100

bs -- block size
count -- No. of loops

These are the basic operations, but they can be modified for more complex and beautiful routines that can be run for hour together to test comprehensively.