Name

cookfs - vfs::cookfs

Table Of Contents

• Table Of Contents
• Synopsis
• Description
• COMMANDS
• MOUNT OPTIONS
• COOKFS STORAGE
• COMPRESSSION
• COOKFS METADATA
• ASIDE AND WRITE TO MEMORY
• See Also
• Keywords
• Copyright

Synopsis

Description

Package vfs::cookfs is a Tcl virtual filesystem (VFS) that allows storing one or more files in a single file. This is similar to mk4vfs, zipvfs and many other archive formats.

The main difference is that cookfs was designed for Tcl-related use and is optimized for both shipping Tcl scripts as well as delivering large payloads in an efficient way.

COMMANDS

::cookfs::Mount ?options? archive local ?options?

::vfs::cookfs::Mount ?options? archive local ?options?

Mounts cookfs archive. Archive is read from archive and can later be accessed from Tcl using local as mount point. Options can preceed or succeed archive and local arguments.

The command returns a cookfs handle which is also a command that can be used to perform certain actions on cookfs archive.

For definition of available options, see MOUNT OPTIONS.

::cookfs::Unmount fsid

Unmounts a cookfs archive. This is called internally by tclvfs. Command vfs::unmount should be used to unmount Tcl archives, not cookfs::Unmount directly.

For cookfs archives it is very important to properly unmount them after performing all operations as many changes are only stored on unmount operation.

cookfsHandle aside filename

Uses a separate file for storing changes to an archive. This can be used to keep changes for a read-only archive in a separate file.

See ASIDE AND WRITE TO MEMORY for more details on aside files and how they can be used.

cookfsHandle writetomemory

Stores all further changes to this archive only in memory. This can be used to store temporary data or applying changes that do not persist across cookfs filesystem remounts and/or application restarts.

See ASIDE AND WRITE TO MEMORY for more details on write to memory feature.

cookfsHandle optimizelist base filelist

Takes a list of files and optimizes them to reduce number of pages read by cookfs. This is mainly useful when unpacking very large number of files.

Parameter base specifies path to be prepended when getting file information. When list of files is relative to archive root, base should be empty. When filelist contains paths relative to subdirectory of archive root, base should specify path to this subdirectory.

For example if archive contains files contents/dir1/file1 and contents/dir1/subdir/file2, the following code can be used: Specifying filelist relative to a directory and proper base:

% puts [$fsid optimizelist contents/dir1 {file1 subdir/file2}]
subdir/file2
file1

Specifying filelist relative to root directory:

% puts [$fsid optimizelist "" {contents/dir1/file1 contents/dir1/subdir/file2}]
contents/dir1/subdir/file2
contents/dir1/file1

cookfsHandle getmetadata parameterName ?defaultValue?

Gets a parameter from cookfs metadata. If parameterName is currently set in metadata, value for it is returned.

If parameterName is not currently set, defaultValue is returned if it was specified. If defaultValue was not specified and parameter is not set, an error is thrown.

See COOKFS METADATA for more details on metadata storage in cookfs archives.

cookfsHandle setmetadata parameterName value

Set parameterName to specified value. If parameter currently exists, it is overwritten.

See COOKFS METADATA for more details on metadata storage in cookfs archives.

cookfsHandle writeFiles ?filename1 type1 data1 size1 ?filename2 type2 data2 size2 ?..???

Write one or more files to cookfs archive. Specified as list of one or more 4-element entries describing files. Each filename specifies name of the file, relative to archive root. Elements type and data specify source for adding a file as well as actual data. Size specifies size of the file. If specified as empty string, it is calculated based on type.

The following values for type are accepted:

• file - data specifies path to file that should be put in the cookfs archive; path is relative to current working directory, same as for command such as file copy; file is copied directly and does not require staging file in memory, unlike writing a file in cookfs archive using open

• data - data is data to be stored in file as binary

• channel - data is a valid Tcl channel that should be read by cookfs; channel is read from current location until end or until size bytes have been read

cookfsHandle filesize

Returns size of file up to last stored page. The size only includes page sizes and does not include overhead for index and additional information used by cookfs.

cookfsHandle smallfilebuffersize

Returns size of all files that are queued up to be written.

MOUNT OPTIONS

The following options can be specified when mounting a cookfs archive:

-readonly

Mount archive as read-only. Archive must exist and be a valid cookfs archive, otherwise mount will fail.

-writetomemory

Enable write to memory feature for this mount.

See ASIDE AND WRITE TO MEMORY for more details on write to memory feature.

-pagesize bytes

Specifies maximum size of a page. This specifies how much bytes a chunk (storing entire file, part of a file or multiple files) can be. Setting larger values increases memory usage, but may increase compression ratio, especially for compressors such as bzip2 or LZMA, which compress larger chunks of information more efficiently.

See COOKFS STORAGE for more details on how cookfs stores files, index and metadata.

-pagecachesize numPages

Maximum number of pages to be stored in memory when reading data from cookfs archive. If 0, no cache is used. Otherwise, up to numPages are kept in memory for efficient access. Increasing the value directly affects speed of reading operations, but also increases memory.

Maximum number of memory used for cache is number of pages to cache multiplied by maximum size of a page.

See COOKFS STORAGE for more details on how cookfs stores files, index and metadata.

-smallfilesize bytes

Specifies threshold for small files. All files smaller than this value are treated as small files and are stored and compressed as multiple files for efficiency.

See COOKFS STORAGE for more details on how cookfs stores files, index and metadata.

-smallfilebuffer bytes

Specifies maximum buffer for small files. Specifying larger value allows storing more files in memory before saving them on disk. This can produce better compression ratio.

See COOKFS STORAGE for more details on how cookfs stores files, index and metadata.

-volume

Register mount point as Tcl volume - useful for creating mount points in locations that do not exist - such as archive://.

-compression none|zlib|bzip2|custom

Compression to use for storing new files.

See COMPRESSSION for more details on compression in cookfs.

-compresscommand tcl command

For custom compression, specifies command to use for compressing pages.

See COMPRESSSION for more details on compression in cookfs.

-decompresscommand tcl command

For custom compression, specifies command to use for decompressing pages.

See COMPRESSSION for more details on compression in cookfs.

-endoffset offset

Read archive starting at specified offset, instead of reading from end of file.

This feature is used when cookfs archive is not stored at end of file - for example if it is used in tclkit and followed by mk4vfs archive.

-setmetadata list

Set specified metadata after mounting. Convenience for setting metadata in mount command. List should be a name-value pairs list indicating parameter names and values to set.

-noregister

Do not register this mount point with tclvfs. Mainly for internal use.

-nocommand

Do not register Tcl command for this cookfs handle. Mainly for internal use.

-bootstrap tcl code

Bootstrap code for cookit binaries. Mainly for internal use.

-pagehash hash

Hash function to use for comparing if pages are equal. This is mainly used as pre-check and entire page is still checked for. Defaults to md5, can also be crc32. Mainly for internal/testing at this moment. Do not use.

-fsindexobject fsiagesObject

Do not create cookfs::fsindex object, use specified fsindex object. Mainly for internal use.

-pagesobject pagesObject

Do not create cookfs::pages object, use specified pages object. Mainly for internal use.

COOKFS STORAGE

Cookfs uses cookfs::pages for storing all files and directories in an archive. Information on which pages store what files is stored in cookfs::fsindex object, that is stored as part of archive.

Cookfs stores files in one of the following ways:

1. Entire file is stored in a single cookfs page. This happens if file's size is larger than -smallfilesize, but smaller than -pagesize.

2. Entire file is stored in multiple cookfs pages. This happens if file's size is larger than -smallfilesize, and larger than -pagesize.

3. Multiple files are stored in a single cookfs page. This happens if file's size is smaller than -smallfilesize.

First and second case are similar and a limit on page size is mainly used to allow faster seek operations and increate I/O operation speed.

When compressing large files as a single stream (such as in zipvfs and mk4vfs), seeking operation requires re-reading and ignoring all data up to new location. With this approach, seek operations are instant since only new pages are read and data prior to new seek location is ignored.

Storing multiple files in a single page is used to increase compression ratio for multiple small files - such as Tcl scripts. All small files (whose size does not exceed -smallfilesize) are grouped by their extension, followed by their file name. This allows files such as pkgIndex.tcl to be compressed in same page, which is much more efficient than compressing each of them independantly.

COMPRESSSION

Cookfs uses compression to store pages and filesystem index more efficiently. Pages are compressed as a whole and independant of files. Filesystem index is also compressed to reduce file size.

Compression is specified as -compression option when mounting an archive. This option applies to newly stored pages and whenever a file index should be saved. Existing pages are not re-compressed on compression change.

Cookfs provides the following compressions:

• none - no compression is used at all; mainly used for testing purposes or when content should not be packed at all

• zlib - use zlib compression to compress - same as with mk4vfs and zipvfs

• bz2 - use bzip2 for compression; requires specifying --enable-bz2 when building cookfs

• custom - use Tcl commands for compressing and decompressing pages

For custom compression, -compresscommand and -decompresscommand have to be specified. These commands will be used to compress data and will be called with data to be compressed or decompressed as additional argument.

For example the following are sample compress and decompress commands:

# use vfs::zip to (de)compress data and encode it in HEX
proc testc {d} {
    binary scan [vfs::zip -mode compress $d] H* rc
    return "HEXTEST-$rc"
}
proc testd {d} {
    set rc [vfs::zip -mode decompress [binary format H* [string range $d 8 end]]]
    return $rc
}
vfs::cookfs::Mount archive archive -compression custom -compresscommand testc -decompresscommand testd

See COOKFS STORAGE for more details on how files are stored in cookfs.

Default values for -pagesize, -smallfilesize and -smallfilebuffer parameters provided by cookfs have been chosen carefully. These are outcome of performing multiple tests. It is not recommended to change it for zlib compression.

COOKFS METADATA

Cookfs supports storing metadata in an archive. Metadata are any number of properties stored in a cookfs archive.

Cookfs does not impose any limits or rules on naming of properties as well as possible values, but it is recommended that cookfs internal properties are stored using cookfs.* prefix and cookit related properties use cookit.* format.

User properties do not need to use any prefix or suffix, but it is recommended to use form of appname.*.

ASIDE AND WRITE TO MEMORY

Cookfs supports a feature called aside changes. It stores all changes in a separate file. It can be used for updating files in read-only archives such as standalone binaries or data on read-only media.

Aside functionality can be enabled for read-only and read-write archives. All types of changes can be performed and they will be stored in the new file. Operations such as deletions from a file are also possible.

Aside changes are persistent. It is possible to mount an archive, setup an aside file and perform changes and unmount it. When remounting the archive, changes will not be visible. However, after enabling aside and pointing to same file as before, changes will instantly be visible.

Aside changes do not cause increased use of memory - changes are written to disk in same fashion, using new file instead of the original one.

Write to memory feature is slightly different. It allows read-only and read-write archives to store all changes in memory and those will not be saved back on disk. This feature is mainly intended for handling temporary files or running legacy code that requires write access to files inside a read-only archive.

See Also

cookfs_fsindex, cookfs_pages

Keywords

cookfs, vfs

Copyright

Copyright © 2010-2011 Wojciech Kocjan