--- /dev/null
+/* ========================================================================
+ * Copyright 1988-2006 University of Washington
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *
+ * ========================================================================
+ */
+
+ Mailbox Format Characteristics
+ Mark Crispin
+ 11 December 2006
+
+
+ When a mailbox storage technology uses local files and
+directories directly, the file(s) and directories are layed out in a
+mailbox format.
+
+I. Flat-File Formats
+
+ In these formats, a mailbox and all the messages inside are a
+single file on the filesystem. The mailbox name is the name of the
+file in the filesystem, relative to the user's "mail home directory."
+
+ A flat-file format mailbox is always a file, never a directory.
+This means that it is impossible to have a flat-file format mailbox
+that has inferior mailbox names under it (so-called "dual-usage"
+mailboxes). For some inexplicable reason, some people want this.
+
+ The mail home directory is usually the same as the user login
+home directory if that concept is meaningful; otherwise, it is some
+other default directory (e.g. "C:\My Documents" on Windows 98). This
+can be redefined by modifying the c-client source code or in an
+application via the SET_HOMEDIR mail_parameters() call.
+
+ For example, a mailbox named "project" is likely to be found in
+the file "project" in the user's home directory. Similarly, a mailbox
+named "test/trial1" (assuming a UNIX system) is likely to be found in
+the file "trial1" in the subdirectory "test" in the user's home
+directory.
+
+ Note that the name "INBOX" has special semantics and rules, as
+described in the file naming.txt.
+
+ The following flat-file formats are supported by c-client as of
+the time of this writing:
+
+. unix This is the traditional UNIX mailbox format, in use for nearly
+ 30 years. It uses a line starting with "From " to indicate
+ start of message, and stores the message status inside the
+ RFC822 message header.
+
+ unix is not particularly efficient; the entire mailbox file
+ must be read when the mailbox is open, and when reading message
+ texts it is necessary to convert the newline convention to
+ Internet standard CR LF form. unix preserves UIDs, and allows
+ the creation of keywords.
+
+ Only one process may have a unix-format mailbox open
+ read/write at a time.
+
+. mmdf This is the format used by the MMDF mailer. It uses a line
+ consisting of 4 <CTRL/A> (0x01) characters to indicate start
+ and end of message. Optionally, there may also be a unix
+ format "From " line. It otherwise has the same
+ characteristics as unix format.
+
+. mbx This is the current preferred mailbox format. It can be
+ handled quite efficiently by c-client, without the problems
+ that exist with unix and mmdf formats. Messages are stored
+ in Internet standard CR LF format.
+
+ mbx permits shared access, including shared expunge. It
+ preserves UIDs, and allows the creation of keywords.
+
+. mtx This is supported for compatibility with the past. This is
+ the old Tenex/TOPS-20 mail.txt format. It can be handled
+ quite efficiently by c-client, and has most of the
+ characteristics of mbx format.
+
+ mtx is deficient in that it does not support shared expunge;
+ it has no means to store UIDs; and it has no way to define
+ keywords except through an external configuration file.
+
+. tenex This is supported for compatibility with the past. This is
+ the old Columbia MM format. This is similar to mtx format,
+ only it uses UNIX-style bare-LF newlines instead of CR LF
+ newlines, thus incurring a performance penalty for newline
+ conversion.
+
+. phile This is not strictly a format. Any file which is not in a
+ recognized format is in phile format, which treats the entire
+ contents of the file as a single message.
+
+
+II. File/Message Formats
+
+ In these formats, a mailbox is a directory, and each the messages
+inside are separate files inside the directory. The file names of
+these files are generally the text form of a number, which also
+matches the UID of the message.
+
+ In the case of mx, the mailbox name is the name of the directory
+in the filesystem, relative to the user's "mail home directory." In
+the case of news and mh, the mailbox name is in a separate namespace
+as described in the file naming.txt.
+
+ A file/message format mailbox is always a directory. This means
+that it is possible to have a file/message format mailbox that has
+inferior mailbox names under it (so-called "dual-usage" mailboxes).
+For some inexplicable reason, some people want this.
+
+ Note that the name "INBOX" has special semantics and rules, as
+described in the file naming.txt.
+
+ The following file/message formats are supported by c-client as of
+the time of this writing:
+
+. mx This is an experimental format, and may be removed in a future
+ release. An mx format mailbox has a .mxindex file which holds
+ the message status and unique identifiers. Messages are
+ stored in Internet standard CF LF form, so the file size of
+ the message file equals the size of the message.
+
+ mx is somewhat inefficient; the entire directory must be read
+ and each file stat()'d. We found it intolerable for a
+ moderate sized mailbox (2000 messages) and have more or less
+ abandoned it.
+
+. mh This is supported for compatibility with the past. This is
+ the format used by the old mh program.
+
+ mh is very inefficient; the entire directory must be read
+ and each file stat()'d, and in order to determine the size
+ of a message, the entire file must be read and newline
+ conversion performed.
+
+ mh is deficient in that it does not support any permanent
+ flags or keywords; and has no means to store UIDs (because
+ the mh "compress" command renames all the files, that's
+ why).
+
+. news This is an export of the local filesystem's news spool, e.g.
+ /var/spool/news. Access to mailboxes in news format is read
+ only; however, message "deleted" status is preserved in a
+ .newsrc file in the user's home directory. There is no other
+ status or keywords.
+
+ news is very inefficient; the entire directory must be
+ read and each file stat()'d, and in order to determine the
+ size of a message, the entire file must be read and newline
+ conversion performed.
+
+ news is deficient in that it does not support permanent flags
+ other than deleted; does not support keywords; and has no
+ expunge.
+
+
+Soapbox on File/Message Formats
+
+ If it sounds from the above descriptions that we're not putting
+too much effort into file/message formats, you are correct.
+
+ There's a general reason why file/message formats are a bad idea.
+Just about every filesystem in existance serializes file creation and
+deletions because these manipulate the free space map. This turns out
+to be an enormous problem when you start creating/deleting more than a
+few messages per second; you spend all your time thrashing in the
+filesystem.
+
+ It is also extremely slow to do a text search through a
+file/message format mailbox. All of those open()s and close()s really
+add up to major filesystem thrashing.
+
+
+What about Cyrus and Maildir?
+
+ Both formats are vulnerable to the filesystem thrashing outlined
+above.
+
+ The Cyrus format used by CMU's Cyrus server (and Esys' server)
+has a special associated flat file in each directory that contains
+extensive data (including pre-parsed ENVELOPEs and BODYSTRUCTUREs)
+about the messages. Put another way, it's a (considerably) more
+featureful form of mx. It also uses certain operating system
+facilities (e.g. file/memory mapping) which are not available on older
+systems, at a cost of much more limited portability than c-client.
+These considerably ameliorate the fundamental problems with
+file/message formats; in fact, Cyrus is halfway to being a database.
+Rather than support Cyrus format in c-client, you should run Cyrus or
+Esys if you want that format.
+
+ The Maildir format used by qmail has all of the performance
+disadvantages of mh noted above, with the additional problem that the
+files are renamed in order to change their status so you end up having
+to rescan the directory frequently to locate the current names
+(particularly in a shared mailbox scenario). It doesn't scale, and it
+represents a support nightmare; it is therefore not supported in the
+official distribution. Maildir support code for c-client is available
+from third parties; but, if you use it, it is entirely at your own
+risk (read: don't complain about how poorly it performs or bugs).
+
+
+So what does this all mean?
+
+ A database (such as used by Exchange) is really a much better
+approach if you want to move away from flat files. mx and especially
+Cyrus take a tenative step in that direction; mx failed mostly because
+it didn't go anywhere near far enough. Cyrus goes much further, and
+scores remarkable benefits from doing so.
+
+ However, a well-designed pure database without the overhead of
+separate files would do even better.
projects / uw-imap / commitdiff