TAR(5) FreeBSD File Formats Manual TAR(5) NAME tar -- format of tape archive files DESCRIPTION The tar archive format collects any number of files, directories, and other filesystem objects (symbolic links, device nodes, etc.) into a single stream of bytes. The format was originally designed to be used with tape drives that operate with fixed-size blocks, but is widely used as a general packaging mechanism. General Format A tar archive consists of a series of 512-byte records. Each filesystem object requires a header record which stores basic metadata (pathname, owner, permissions, etc.) and zero or more records containing any file data. The end of the archive is indicated by two records consisting entirely of zero bytes. For compatibility with tape drives that use fixed block sizes, programs that read or write tar files always read or write a fixed number of records with each I/O operation. These ``blocks'' are always a multiple of the record size. The most common block size---and the maximum sup- ported by historical implementations---is 10240 bytes or 20 records. (Note: the terms ``block'' and ``record'' here are not entirely standard; this document follows the convention established by John Gilmore in docu- menting pdtar.) Old-Style Archive Format The original tar archive format has been extended many times to include additional information that various implementors found necessary. This section describes a variant that is compatible with most historic tar implementations. The header record for an old-style tar archive consists of the following: struct tarfile_header_old { char name[100]; char mode[8]; char uid[8]; char gid[8]; char size[12]; char mtime[12]; char checksum[8]; }; The remaining bytes in the header record are filled with nulls. name Pathname, stored as a null-terminated string. Some very early implementations only supported regular files. However, a common early convention added a trailing "/" character to indicate a directory name, allowing directory permissions and owner informa- tion to be archived and restored. mode File mode, stored as an octal number in ASCII. uid, gid User id and group id of owner, as octal number in ASCII. size Size of file, as octal number in ASCII. mtime Modification time of file, as an octal number in ASCII. This indicates the number of seconds since the start of the epoch, 00:00:00 UTC January 1, 1970. Note that negative values should be avoided here, as they are handled inconsistently. checksum Header checksum, stored as an octal number in ASCII. To compute the checksum, set the checksum field to all spaces, then sum all bytes in the header using unsigned arithmetic. This field should be stored as six octal digits followed by a null and a space character. Note that for many years, Sun tar used signed arith- metic for the checksum field, which can cause interoperability problems when transferring archives between systems. This error was propagated to other implementations that used Sun tar as a reference. Modern robust readers compute the checksum both ways and accept the header if either computation matches. Early implementations of tar varied in how they terminated these fields. Early BSD documentation specified the following: the pathname must be null-terminated; the mode, uid, and gid fields must end in a space and a null byte; the size and mtime fields must end in a space; the checksum is terminated by a null and a space. For best portability, writers of tar archives should fill the numeric fields with leading zeros. Early Extensions Very early tar implementations only supported regular files. Two early extensions added support for directories, hard links, and symbolic links. Early tar archives indicated directories by adding a trailing / to the name. The size field was often used to indicate the total size of all files in the directory. This was intended to facilitate extraction on systems that pre-allocated directory storage; most modern readers should simply ignore the size field for directories. To support hard links and symbolic links, linktype and linkname fields were added: struct tarfile_entry_common { char name[100]; char mode[8]; char uid[8]; char gid[8]; char size[12]; char mtime[12]; char checksum[8]; char linktype[1]; char linkname[100]; }; The linktype field indicates the type of entry. For backwards compati- bility, a NULL character here indicates a regular file or directory. An ASCII "1" here indicates a hard link entry, ASCII "2" indicates a sym- bolic link. The linkname field holds the name of the file linked to. POSIX Standard Archives POSIX 1003.1 defines a standard tar file format that is read and written by POSIX-compliant implementations of pax(1). This format is often called the ``ustar'' format, after the magic value used in the header. (The name is an acronym for ``Unix Standard TAR''.) It extends the for- mat above with new fields: struct tarfile_entry_posix { char name[100]; char mode[8]; char uid[8]; char gid[8]; char size[12]; char mtime[12]; char checksum[8]; char typeflag[1]; char linkname[100]; char magic[6]; char version[2]; char uname[32]; char gname[32]; char devmajor[8]; char devminor[8]; char prefix[155]; }; typeflag Type of entry. POSIX adopted the BSD linktype field and extended it with several new type values: ``0'' Regular file. NULL should be treated as a synonym, for compatibility purposes. ``1'' Hard link. ``2'' Symbolic link. ``3'' Character device node. ``4'' Block device node. ``5'' Directory. ``6'' FIFO node. ``7'' Reserved. Other A POSIX-compliant implementation must treat any unrecog- nized typeflag value as a regular file. In particular, writers should ensure that all entries have a valid file- name so that they can be restored by readers that do not support the corresponding extension. Uppercase letters "A" through "Z" are reserved for custom extensions. Note that sockets are not archivable. magic Contains the magic value ``ustar'' followed by a NULL byte to indicate that this is a POSIX standard archive. Full compliance requires the uname and gname fields be properly set. (Note that GNU tar archives uses a trailing space rather than a trailing NULL here and are therefore not POSIX standard archives.) version Version. This should be ``00'' (two copies of the ASCII digit zero) for POSIX standard archives. (Note that GNU tar archives fill this with a space and a null.) uname, gname User and group names, as null-terminated ASCII strings. These should be used in preference to the uid/gid values when they are set and the corresponding names exist on the system. devmajor, devminor Major and minor numbers for character device or block device entry. prefix First part of pathname. If the pathname is too long to fit in the 100 bytes provided by the standard format, it can be split at any / character with the first portion going here. If the prefix field is not empty, the reader will prepend the prefix value and a / character to the regular name field to obtain the full path- name. Note that all unused bytes must be set to NULL. Field termination is specified slightly differently by POSIX than by pre- vious implementations. The magic, uname, and gname fields must have a trailing NULL. The pathname, linkname, and prefix fields must have a trailing NULL unless they fill the entire field. (In particular, it is possible to store a 256-character pathname if it happens to have a / as the 156th character.) POSIX requires numeric fields to be zero-padded in the front, and allows them to be terminated with either space or NULL characters. Pax Interchange Format There are many attributes that cannot be portably stored in a POSIX ustar archive. POSIX defined a ``pax interchange format'' that uses two new types of entries to hold text-formatted metadata that applies to follow- ing entries. Note that a pax interchange format archive is a ustar ar- chive in every respect. The new data is stored in ustar-compatible ar- chive entries that use the ``x'' or ``g'' typeflag. In particular, older implementations that do not fully support these extensions will extract the metadata into regular files, where the metadata can be examined as necessary. An entry in a pax interchange format archive consists of one or two stan- dard entries, each with its own header and data. The first optional entry stores the extended attributes for the second entry. The first entry has an "x" typeflag and a size field that indicates the total size of the extended attributes. The extended attributes themselves are stored as a series of text-format lines encoded in the portable UTF-8 encoding. Each line consists of a decimal number, a space, a key string, an equals sign, a value string, and a new line. The decimal number indi- cates the length of the entire line, including the initial length field and the trailing newline. Keys are always encoded in portable 7-bit ASCII. Keys in all lowercase are reserved for future standardization. Vendors can add their own keys by prefixing them with an all uppercase vendor name and a period. Note that, unlike the historic header, numeric values are stored using decimal, not octal. atime, ctime, mtime File access, inode change, and modification times. These fields can be negative or include a decimal point and a fractional value. uname, uid, gname, gid User name, group name, and numeric UID and GID values. The user name and group name stored here are encoded in UTF8 and can thus include non-ASCII characters. The UID and GID fields can be of arbitrary length. linkpath The full path of the linked-to file. Note that this is encoded in UTF8 and can thus include non-ASCII characters. path The full pathname of the entry. Note that this is encoded in UTF8 and can thus include non-ASCII characters. realtime.*, security.* These keys are reserved by SUSv3 and may be used for future stan- dardization. size The size of the file. Note that there is no length limit on this field, allowing tar archives to store files much larger than the historic 8GB limit. SCHILY.* Vendor-specific attributes used by Joerg Schilling's star imple- mentation. Includes SCHILY.devminor, SCHILY.devmajor, and SCHILY.acl. VENDOR.* XXX document other vendor-specific extensions XXX Any values stored in an extended attribute override the corresponding values in the regular tar header. Note that compliant readers should ignore the regular fields when they are overridden. This is important, as existing archivers are known to store non-compliant values in the standard header fields in this situation. There are no limits on length for any of these fields. In particular, numeric fields can be arbitrar- ily large. All text fields are encoded in UTF8. Compliant writers should store only portable 7-bit ASCII characters in the standard ustar header and use extended attributes whenever a text value contains non- ASCII characters. In addition to the x entry described above, the pax interchange format also supports a g entry. The g entry is identical in format, but speci- fies attributes that serve as defaults for all subsequent archive entries. The g entry is not widely used. GNU Tar Archives The GNU tar program added new features by starting with an early draft of POSIX and using three different extension mechanisms: They added new fields to the empty space in the header (some of which was later used by POSIX for conflicting purposes); they allowed the header to be continued over multiple records; and they defined new entries that modify following entries (similar in principle to the x entry described above, but each GNU special entry is single-purpose, unlike the general-purpose x entry). As a result, GNU tar archives are not POSIX compatible, although more lenient POSIX-compliant readers can successfully extract most GNU tar ar- chives. struct tarfile_entry_gnu { char name[100]; char mode[8]; char uid[8]; char gid[8]; char size[12]; char mtime[12]; char checksum[8]; char typeflag[1]; char linkname[100]; char magic[6]; char version[2]; char uname[32]; char gname[32]; char devmajor[8]; char devminor[8]; char atime[12]; char ctime[12]; char offset[12]; char longnames[4]; char unused[1]; struct { char offset[12]; char numbytes[12]; } sparse[4]; char isextended[1]; char realsize[12]; }; typeflag GNU tar uses the following special entry types. 7 GNU tar treats type "7" records identically to type "0" records, except on one obscure RTOS where they are used to indicate the pre-allocation of a contiguous file on disk. D This indicates a directory entry. Unlike the POSIX-stan- dard "5" typeflag, the header is followed by data records listing the names of files in this directory. Each name is preceded by an ASCII "Y" if the file is stored in this archive or "N" if the file is not stored in this archive. Each name is terminated with a null, and an extra null marks the end of the name list. The purpose of this entry is to support incremental backups; a program restoring from such an archive may wish to delete files on disk that did not exist in the directory when the ar- chive was made. Note that the "D" typeflag specifically violates POSIX, which requires that unrecognized typeflags be restored as normal files. In this case, restoring the "D" entry as a file could interfere with subsequent creation of the like-named directory. K The data for this entry is a long linkname for the fol- lowing regular entry. L The data for this entry is a long pathname for the fol- lowing regular entry. M This is a continuation of the last file on the previous volume. GNU multi-volume archives gaurantee that each volume begins with a valid entry header. To ensure this, a file may be split, with part stored at the end of one volume, and part stored at the beginning of the next vol- ume. The "M" typeflag indicates that this entry contin- ues an existing file. Such entries can only occur as the first or second entry in an archive (the latter only if the first entry is a volume label). The size field spec- ifies the size of this entry. The offset field at bytes 369-380 specifies the offset where this file fragment begins. The realsize field specifies the total size of the file (which must equal size plus offset). When extracting, GNU tar checks that the header file name is the one it is expecting, that the header offset is in the correct sequence, and that the sum of offset and size is equal to realsize. FreeBSD's version of GNU tar does not handle the corner case of an archive being continued in the middle of a long name or other extension header. N Type "N" records are no longer generated by GNU tar. They contained a list of files to be renamed or symlinked after extraction; this was originally used to support long names. The contents of this record are a text description of the operations to be done, in the form ``Rename %s to %s\n'' or ``Symlink %s to %s\n''; in either case, both filenames are escaped using K&R C syn- tax. S This is a ``sparse'' regular file. Sparse files are stored as a series of fragments. The header contains a list of fragment offset/length pairs. If more than four such entries are required, the header is extended as nec- essary with ``extra'' header extensions (an older format that's no longer used), or ``sparse'' extensions. V The name field should be interpreted as a tape/volume header name. This entry should generally be ignored on extraction. magic The magic field holds the five characters ``ustar'' followed by a space. Note that POSIX ustar archives have a trailing null. version The version field holds a space character followed by a null. Note that POSIX ustar archive use two copies of the ASCII digit ``0''. atime, ctime The time the file was last accessed and the time of last change of file information, stored in octal as with mtime. longnames This field is apparently no longer used. Sparse offset / numbytes Each such structure specifies a single fragment of a sparse file. The two fields store values as octal numbers. The fragments are each padded to a multiple of 512 bytes in the archive. On extraction, the list of fragments is collected from the header (including any extension headers), and the data is then read and written to the file at appropriate offsets. isextended If this is set to non-zero, the header will be followed by addi- tional ``sparse header'' records. Each such record contains XXX more details needed XXX realsize A binary representation of the size, with a much larger range than the POSIX file size. Other Extensions One common extension, utilized by GNU tar, star, and other newer tar implementations, permits binary numbers in the standard numeric fields. This is flagged by setting the high bit of the first character. This permits 95-bit values for the length and time fields and 63-bit values for the uid, gid, and device numbers. GNU tar supports this extension for the length, mtime, ctime, and atime fields. Joerg Schilling's star program supports this extension for all numeric fields. Note that this extension is largely obsoleted by the extended attribute record provided by the pax interchange format. Another early GNU extension allowed base-64 values rather than octal. This extension was short-lived and such archives are almost never seen. However, there is still code in GNU tar to support them; this code is responsible for a very cryptic warning message that is sometimes seen when GNU tar encounters a damaged archive. SEE ALSO ar(1), pax(1), tar(1), STANDARDS The tar utility is no longer a part of any official standard. It last appeared in SUSv2. It has been supplanted in subsequent standards by pax(1). The ustar format is defined in IEEE Std 1003.1 (``POSIX.1'') as part of the specification for the pax(1) utility. The pax interchange file format is new with IEEE Std 1003.1-2001 (``POSIX.1''). HISTORY A tar command appeared in Sixth Edition Unix. John Gilmore's pdtar pub- lic-domain implementation (circa 1987) was highly influential and formed the basis of GNU tar. Joerg Shilling's star archiver is another open- source (GPL) archiver (originally developed circa 1985) which fea- tures complete support for pax interchange format. FreeBSD 5.1 October 1, 2003 FreeBSD 5.1