--- /dev/null
+/* ========================================================================
+ * Copyright 1988-2006 University of Washington
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *
+ * ========================================================================
+ */
+
+Last update: 18 December 2006
+
+INTRODUCTION
+
+This file is the descendant of a design document used to specify the
+mix format. An attempt is being made to keep this document more or
+less current with the way the mix format actually works.
+
+
+1. Mix mailbox naming
+
+Mailbox names correspond to directory names; thus mix format mailboxes
+are "dual-use" (lack both \NoInferiors and \NoSelect). This will
+satisfy some long-standing requests.
+
+
+2. Mailbox files
+
+A mix format mailbox is a directory with regular files with filenames
+of:
+ .mixmeta mailbox metadata file
+ .mixindex message index file (message static data)
+ .mixstatus message status file (message dynamic data)
+ .mix######## (where ######### is a <hex8>) secondary message
+ data files.
+ .mix primary message data file (used in experimental
+ versions, supported for compatibility only)
+
+2.1 Metadata, index, and status files
+
+The mailbox metadata, index, and status files contain a sequence of
+CRLF-terminated lines. These files have an update sequence, which is
+a strictly-ascending sequence value. Any time the file is changed,
+the update sequence is increased; this allows easy detection of
+whether the file has been changed by another process. For now, this
+update sequence is a modseq (see below).
+
+2.1.1 Metadata file
+
+The mailbox metadata file is called ".mixmeta". It contains a series
+of CRLF-terminated lines. The first character of the line is a key that
+identifies the payload of the line, and the remainder of the line is the
+payload.
+ Key Payload
+ --- -------
+ S <hex8> ;; update sequence
+ V <hex8> ;; UIDVALIDITY
+ L <hex8> ;; UIDLAST
+ N <hex8> ;; current new message file
+ K [atom 0*(SP atom)] ;; keyword list
+
+All other keys are reserved for future assignment and must be ignored
+(and may be discarded) by software which does not recognize them. The
+mailbox metadata file is rewritten as part of new mail delivery (so
+APPENDUID/COPYUID can work) and when new keywords are added.
+
+2.1.2 Message static index file
+
+The mailbox message static index file is called ".mixindex". It contains
+a series of CRLF-terminated lines. The first character of the line is a
+key that identifies the payload of the line, and the remainder of the line
+is the payload.
+ Key Payload
+ --- -------
+ S <hex8> ;; update sequence
+ : <uid>:<date>:<size>:<file>:<pos>:<isiz>:<hsiz>
+ ;; per-message record
+
+The per-message records contain the following data:
+ <uid> = <hex8> ;; message UID
+ <date> = <yyyymmddhhmmss+zzzz> ;; internal date
+ <size> = <hex8> ;; rfc822.size
+ <file> = <hex8> ;; message data file (0 = .mix file)
+ <pos> = <hex8> ;; message position in file
+ <isiz> = <hex8> ;; message internal data size
+ <hsiz> = <hex8> ;; header size (offset to body)
+
+All other keys, and subsequent fields in per-message records, are
+reserved for future assignment and must be ignored (and may be
+discarded) by software which does not recognize them. The mailbox
+metadata file is appended by new mail delivery and rewritten by
+expunge "burping", and otherwise is not altered.
+
+2.1.3 Message dynamic status file
+
+The mailbox message dynamic status file is called ".mixstatus". It contains
+a series of CRLF-terminated lines. The first character of the line is a
+key that identifies the payload of the line, and the remainder of the line
+is the payload.
+ Key Payload
+ --- -------
+ S <hex8> ;; update sequence
+ : <uid>:<uf>:<sf>:<mod>: ;; per-message record
+
+The per-message records contain the following data:
+ <uid> = <hex8> ;; message UID
+ <keys> = <hex8> ;; keyword flags
+ <flag> = <hex4> ;; system flags
+ <mod> = <hex8> ;; date/time last modified (modseq)
+
+All other keys, and subsequent fields in per-message records, are
+reserved for future assignment and must be ignored (and may be
+discarded) by software which does not recognize them. The mailbox
+dynamic idex file is rewritten by flag changes (or any future change
+that alters dynamic data) and is re-read when a session sees that the
+mtime has changed (atime and ctime are not used).
+
+The modseq is an unsigned 32-bit date/time, along with a guarantee
+that this value can not go backwards. It currently corresponds to the
+time from time(); however, since it is unsigned, it won't run out until
+the year 2106. In the future, this may be used as a basic for implementing
+the IMAP CONDSTORE extension.
+
+2.2 Message data files
+
+A mix message file is a regular file with filename starting with
+".mix" followed by a <hex8> suffix which indicates the file number. It
+contains a series of CRLF-terminated lines. By special dispensation, the
+filename ".mix" is used for file number 0, which was used in experimental
+versions of mix as a "primary" file (this concept no longer exists).
+
+A file number is set to the current modseq when it is created. If a copy
+or append causes the file to exceed the compiled-in file size limit, a new
+file is started and the metadata is updated accordingly.
+
+Preceeding each message is per-message record with the following format:
+ Key Payload
+ --- -------
+ ;; per-message record
+ : :<code>:<uid>:<date>:<size>:
+
+The per-message records contain the following data:
+ <code> = "msg" ;; fixed code
+ <uid> = <hex8> ;; message UID
+ <date> = <yyyymmddhhmmss+zzzz> ;; internal date
+ <size> = <hex8> ;; rfc822.size
+The message data begins on the next line
+
+Subsequent fields are reserved for future assignment and must be ignored.
+
+
+3. New mail delivery
+
+To deliver a new message, it is necessary to share lock the destination
+metadata file, then get an exclusive lock on the destination index and
+status files. Once this is done, the new message data is appended to the
+new message file. The metadata (UIDLAST value), index, and status
+files are all updated to add the new message.
+
+Then all the destination mailbox files are closed.
+
+
+4. Mailbox pinging
+
+The index and status files are share locked. Initially, sequences are
+remembered as zero, so at open time they are always "altered".
+
+The sequence from the index file is checked; if it is altered the index
+file is read and processed as follows:
+ . If expunge is permitted, then any messages that are not in the index
+ are reported as having been expunged via mm_expunged().
+ . new messages are announced via mm_exists()/mm_recent().
+
+Next, the sequence from the status file is checked. If it is altered,
+the status file is read and the status updated for any message which is
+new or has an altered modseq in the status file. Altered modseq messages
+are announced via mm_flags().
+
+Then the index and status files are closed.
+
+
+4. Flag alteration
+
+The status file is exclusive locked.
+
+The sequence from the status file is checked. If it is altered, the
+status file is read and the status updated for any message which is
+new or has an altered modseq in the status file. Altered modseq
+messages are announced via mm_flags().
+
+The alterations are then applied for all requested messages, updating
+the modseq for each requestedmessage which changes flags as a result
+of the alteration (alterations which do not result in a change do not
+alter the modseq). Then the status file is rewritten with a new
+sequence, but only if flags of at least one message was changed.
+
+Then the status file is closed.
+
+
+5. Checkpoint and expunge
+
+Checkpoint is identical to expunge, however it skips the step of expunging
+deleted messages.
+
+The index and status files are locked exclusive. If expunging, all
+deleted messages are expunged from the index and announced via
+mm_expunged(). The message data is notremoved at this time.
+
+If a checkpoint was requested, or if any messages were expunged, or if
+it remembered that a "burp" was needed, then:
+ . the metadata file is locked exclusive. If this fails, remember that
+ a burp is needed. Otherwise perform a burp:
+ . calculate the file byte ranges occupied by expunged messages
+ . for each file needing "burping", open and slide down subsequent file
+ data on top of the expunged messages
+ . update the index and status files
+
+Then the index and status files are closed.
+
+5.1 More details on expunging and "burping"
+
+Shared expunge presents a problem due to the requirements of the IMAP
+protocol. You can't "burp" away a message until you are certain that
+no sharers have a pointer to any longer. Consequently, for the nonce
+"burping" out expunged data be defered to an exclusive expunge as in
+mbx format.
+
+If shared burping is ever implemented, then care will be needed not to
+burp data that a session still relies upon. It's easy enough to burp
+the index files; just create new index files, deleting the old, and
+require that you look for a new one appearing at mailbox ping time
+(when it's safe). The data files are a problem, since we
+intentionally don't want to keep them open and do want to avoid quota
+problems by overwriting in place. Also, when you burp you have to
+change the pointers in the index file.
+
+Bottom line: shared burping is too hairy right now, so the first
+version will do exclusive-only burping and not worry about it. If
+shared burping is really needed, then that routine will need to be
+rewritten.
+
+Shared burping has been a problem for every other IMAP server. Most
+get it wrong, and cause terrible confusion to clients (including
+client crashes).
+
+
+6. Message data file file roll out strategy
+
+The current new message file is finalized, and a new one started, when
+an append or copy is done that would cause the file to grow to larger
+than a preconfigured size (MIXDATAROLL). A multi-message copy or
+append is written into its entirety to a single new message file. In
+the case of multi-copy, the new message file is switched when the sum
+of the sizes of all messages to be copied would cause the current new
+message file to exceed MIXDATAROLL. In the case of multi-append, only
+the first message is considered; this is due to technical limitations.
+
+7. Error detection
+
+Mix detects bad data in the metadata, index, and status files; and
+declares the stream dead. It does not unilaterally reassign
+UIDVALIDITY the way that the flat file formats do.
+
+When mix reads a header from the message file, it also reads the
+per-message record and verifies that there is a per-message record there.
+This is a simple test for message file corruption. It doesn't declare
+the stream dead; it simply issues an error message and returns a
+zero-length string for the message header. This makes it possible for
+the user to fix the mailbox simply by deleting and expunging any messages
+that are in this state.
+
+
+8. Reconstruct tool
+
+[None of this is implemented yet.]
+
+The layout of these files is designed to make the reconstruct tool be
+as simple as possible. Much of the need for the reconstruct tool is
+eliminated since the mix format has a much more limited scope of
+writing than the flat file formats; thus there is "less collateral
+damage."
+
+If the metadata file is lost or corrupted, then all keywords are lost;
+if the mailbox has any keywords used in the .mixstatus file, it'll be
+necessary to create some placeholder names. Otherwise, a new
+UIDVALIDITY can be assigned, and a good UIDLAST value calculated by
+the reconstruct tool. Since this file is very small, it's not likely
+to be damaged.
+
+If the index file is lost or corrupted, it is possible to reconstruct
+it with no loss by reading all the data files. However, this could
+cause expunged but not yet burped messages to reappear.
+
+If the status file is lost or corrupted, then flags are lost and
+will revert to a default state of no flags set. Just deleting the
+corrupted file is good enough.
+
+The reconstruct tool can use the per-message record in the message
+file to locate messages if the recorded sizes and/or messages are
+corrupt. If that happens, it will need to rebuild the index file
+(with associated changes to the metadata file to change the
+UIDVALIDITY). That should probably be a manual operation and not be
+part of the default operation or auto-reconstruct.
+
+
+9. Locking strategy
+
+The mix format does not use the traditional c-client /tmp file locking.
+
+The metadata file is open and locked whenever the mailbox is open.
+Normally this is a shared lock, but it will be upgraded to exclusive
+if the mailbox is expunged. As a guard (since there is no true
+lock-upgrade/downgrade on UNIX), the index exclusive lock must be
+acquired first before upgrading to exclusive.
+
+The index file is shared locked when reading the index, and exclusive
+locked (and read) when appending new messages to the index or when
+expunging (note that expunging also requires an exclusive lock on
+metadata). Normally, the index file is not open or locked.
+
+The status file is shared locked when reading status, and exclusive
+locked (and read) when updating status. Normally, the status file is
+not open or locked.
+
+It isn't necessary to lock any of the data files as long as we only
+have exclusive burping.
+
+
+10. Memory usage
+
+The mix format returns a file stringstruct, which is the modern
+c-client behavior. This prevents imapd from growing to enormous sizes
+due to a godzillagram (how it affects other programs depends upon what
+they do with the returned stringstruct).
+
+
+11. Future extensions
+
+Cached ENVELOPE, BODYSTRUCTURE. Cyrus does, and this will eliminate
+most of the reason to access the data files. Possibly cached overviews,
+ala NNTP, instead?
+
+
+Support for ANNOTATION.
+
+
+12. RENAME issues
+
+Mix currently makes no attempt to address the IMAP RENAME problem.
+This occurs when a mailbox is deleted, and another mailbox is renamed
+with that name in place, no attempt is made to reassign UIDVALIDITY
+for this mailbox and all the inferior mailboxes. This potentially can
+cause problems for a disconnected-use client that has cached status
+for the old mailbox which had that name.
+
+The RENAME problem is a well known flaw in the IMAP protocol. Few
+servers correctly handle it (among other things, not only do all the
+UIDVALIDITY values have to be changed but this has to be done
+atomically!). It was a mistake to add RENAME into IMAP, but it's much
+too late to remove it now.
projects / uw-imap / commitdiff