Entries by Rocco Rutte

Today's topic is more of a religious nature: Licensing.



Until now, all files have been just licensed under the GPL. We now switched to a mixed system which makes sense for us.



The new licenses we use include: GPL (of course), LGPL and public domain.



To start with the latter: we took some minimal implementations from mailing lists or other publicly available archives and silently used them. To tag them as GPL is not correct as by appearing publicly we consider these to be public domain already. Also, we put the examples of the manual and some tool scripts under it. The GPL is fine but for some simple scripts this is just overkill.



LGPL. In the very end we want to at least have the core part under LGPL. The reason is quite simple: there're no real algorithms but rather wrapper functions putting libraries (which are themselves under LGPL) behind a convenient API for use in muttng. As the underlying libraries are LGPL to have proprietary software benefit from it, why shouldn't benefits from our abstraction layer be possible? Currently only the files we wrote completely from scratch are put under it. Files containing portions of old code are left under GPL, of course.



GPL. Right now the libmuttng part contains just too many fractions of old code so it cannot be simply put under LGPL. We hope we have time to replace these parts step by step so we can replace GPL-dependencies of our new code by new LGPL-code to, in the end, make libmuttng licensed under LGPL, too. The muttng part (i.e. all tools and applications) will remain GPL as "just an example of how to use the libraries."



As it turns out, there always are dicussions when (partial) license switches appear. So everyone is invited to comment on it.



If somebody discovers parts of old code now tagged as being LGPL, please notify us as this is a serious mistake and license violation not done intentionally.

Today's topic isn't that exciting after all: IPv6.



I just committed an update of the Connection class and the URL parser to work nicely with IPv6. However, there's one thing to note so that I write this blog entry.



What changed (somewhat) is the basic URL syntax. It still remains as:

proto[s]://[user[:password]]@host[:port][/path]

...but for IPv6 the host part is somewhat treated specially if it's an IPv6 address rather than a domain name (in which case the DNS resolver will do the job together with the IPv6 stack of the operating system.)



IPv6 addresses have to be enclosed in [] since they contain colons (':') which are used to detect whether and what port is embedded in the URL in case it's non-standard for the protocoll.



Also, please keep in mind that all parts of the URL are expected to be encoded. This is important for IPv6's link-local addresses with KAME-derived IPv6 stacks such as found on the BSD systems. With KAME, an interface may be embedded like so:

fe80::1%lo0

which is the link-local address for the loopback device fe80::1 reachable via lo0 (of course, just ::1 is shorter but this serves only as an example for non-KAME users.)



In short, the percent sign ('%') is used in URLs for marking an encoded sequence so that itself has to be encoded as %25. A correct example IMAP URL to loopback for user joe with password secret using a secure connection to port 4711 would thus be:


imaps://joe:secret@[fe80::1%25lo0]:4711/

or shorter:

imaps://joe:secret@[::1]:4711/



Also, the code handling connect(2) has been replace by a portable and protocol-independent one so that IPv4 and IPv6 are expected to work without problems. As I didn't setup IPv6-enabled local servers and as I don't/won't get IPv6 connectivity, I couldn't test very much. Please report problems if you encounter any.



Testing can be done with the muttng-query tool like so:

$ muttng-query -d 5 pop[s]://ipv6host/

or

$ muttng-query -d 5 nntp[s]://ipv6host/group.

Today's topic for the rewrite features is: binaries.



As some may have seen from the doxygen documentation and/or the source, several binaries are being built and will be installed. There'll be one main muttng binary and several others being a symlink to it. Upon startup, the binary name is detected an muttng behaves differently. This article briefly explains what these binaries will do as my documentation patch is not yet committed.



Of course, muttng(1) will remain the main interactive client with a curses-based interface. This is also the main binary being installed.



muttng-conf(1) will be, inspired by the postconf(1) command, a command for configuration diagnostics. Currently, the following items are already supported: print variable names and values and print function bindings. For these and all future groups of items, there are the following three modes planned: (1) just print everything as-is, (2) print only changed items and (3) as an add-on to the other two modes, support an "annotated" mode. "Annotated" means that it prints comments with defaults and other descriptions along the main output. Output will be printed in the muttngrc syntax so that the output can be used as a configuration file itself.



muttng-mailx(1) will be a true mailx(1) replacement. Now that there is a command-line option to enable mailx-compatibility. However, it doesn't make sense as scripts or the like must be ported to it. We will turn this into a true replacement so that it, for example, can be used with mechanisms such as Debian's alternates. This will be a really cool feature as all tools will use the users configuration. For example, with it all the hooks and other features being available for the mailx-replacement, users can setup cronjobs and their mail delivery very flexibly.



muttng-query(1) is a temporary addition and it depends on user feedback whether it will stay or not. As the user interface(s) are last on our "roadmap" (because we want to ensure the two library layers are working properly before we start to use them), we need something to test libmuttng with. The query tool will provide a command-line way of dealing with any type of folder. Also it will support a colon mode similar to that of gpg(1) so that output can be parsed by scripts easily. Planned features are listing which mailboxes have new mail, maybe also print which mail is new, maybe search accross folders, etc. Before its addition yesterday some testing code was implemented in muttng(1) where it doesn't fit so I simply created this tool. As said, we're doing a rewrite and are interested in feedback and critics of any type so it depends on users whether this tool will stay and what features it'll offer.



muttng-sync(1) will be a tool supporting bi-directional syncing of arbitrary folders. Possible uses range from simple offline IMAP to syncing two Maildir trees with a workstation and a laptop. As it'll also be built upon libmuttng which will provide transparent support for different mailbox types, it'll be theoretically possible to mirror remote newsgroups to a local Maildir or MBOX file as well as a remote IMAP folder. Not only because users requested an offline IMAP feature (which muttng-sync(1) will be the generalized solution to), but also to test whether the abstraction we make in libmuttng, this tool serves well a real-world test environment.



Any feedback is welcome...

Hereby I start a new series of articles called "Insight Rewrite" which will give some details about this work or will work. Today's topic is: debugging.



Debugging support is essential as with unit testing we can only verify that a single function does what it's supposed to but not really/completely whether they're also used the right way.



Debugging will be available, as before, via the -d command line switch for all tools as well as via the $debug_level option. The reason behind the config option is that now users can freely turn debugging on and off during runtime as well as only change the verbosity level.



Before there were only a few debug files kept which is a "feature" removed in the rewrite. Now debug files will remain until they're deleted by users. Also, the naming is different: files will be named as: ~/.[scope].[pid].[id].log. These fields have the following meanings: "[scope]" is either "libmuttng" or one of "muttng-conf", "muttng-sync", etc (yes, debugging is done to different files for the libmuttng library and the actual application). "[pid]" is the current process' ID number. As debugging may be turned on and off during runtime, the "[id]" field starts off with "1" and will be subsequently increased everytime debugging is turned on again.



New is also the fact that each of the debug levels from 1 to 5 has it's own meaning, i.e. we don't choose a message's debug level by our current mood any longer. For example, level 1 is for all module registration and de-registration, level 2 for parsing, 3 for signals and events, etc.



As before, each line will contain the caller besides the message. New is that lines are space-indented to more quickly find things as debug files may grow quite large.



As some may have wondered why we organized the inheritance for libmuttng as well as applications the way we did: the debugging support is implemented in libmuttng and publicy visible (and thus useable) by applications so that the base classes LibMuttng and Muttng for both layers contain such a debug object. As a consequence, all classes have debug support. The level change is connected via the configuration manager and a signal to achieve separation.



As today and yesterday I ported a number of core/ functions required by the configuration parser, and finally added a first one simply printing each line read to the debug file, brave users can already get, compile and run the muttng-conf binary and see the files created in $HOME.



Next up on my todo list are all the unit tests still missing for these functions added so it depends on how much spare time I have whether we'll have a configuration parser to finish the year with.



(Btw, we developers will meet at the Chaos Communication Congress in Berlin at the end of the year and applied for its hackcenter so maybe we can meet users and/or write the parser.)

For the last couple of days and weeks, there isn't much going on with mutt-ng for several reasons.





First, we, the core developers, have come to the conclusion that a "next generation" mutt really needs a rewrite or better: a (documented!) design made with current and future features in mind. An example which is just impossible to implement right now easily in an elegant fashion is transparent syncing of different folders like remote to local IMAP, remote to remote IMAP, local to (NFS-)local Maildir, etc. With the re-design ideas we have so far, this will be easy as it will be built on top of a "mail storage abstraction."





Second, we have come to the conclusion that when adding features, we spend most of the time trying to understand the complexity behind the current code to find the proper place for the change (sometimes we were even stuck.) Thus, some of the things we already changed only improves the situation from a user's point of view, but adds more complexity making it harder to maintain the code base in the feature.





A third and last reason for why we need a rewrite is simply the fact that we included many patches already. The main problem we experienced is that these patches are just add-ons, that is, they add features but for compatibility are not allowed to improve the underlaying design or interfaces to implement a feature more easily. Also, many patches are standalone and don't really integrate or work well with one another. We found it too difficult to actually integrate all of the foreign code, re-design everything "in-place" and thus make everything fit into a consistent mail client.





Andreas, the maintainer, has opened a branch with early ideas which are subject to change and which will be implemented as soon as they turn out to be the way to go.





Everybody is invited to join and start discussions on the developer's list.





As I maintain an automated blog for the mutt-ng subversion repository, you can also stay up-to-date on the rewrite branch via the following URL:


http://user.cs.tu-berlin.de/~pdmef/muttng.cgi?c=muttng-rewrite.


There you'll also find various RSS, Atom and even MBOX newsfeeds.

As readers of the mailing lists and/or ChangeLogs may have noticed, some time ago I added an ad-hoc stylesheet to transform the DocBook/XML based manual into LaTeX.





As the manual itself isn't valid and plain DocBook but contains some goodies to make it more consistent and easier to write, so does the LaTeX (stylesheet) file. That means it's not a general DocBook to LaTeX conversion solution.





At our Berlios site, there're two links given: one for a PDF generated by pdflatex(1) and one PostScript file generated by latex(1).





The Makefile in the source's doc subdirectory contains two targets to build them: manual-ng.pdf and manual-ng.ps whereby for the latter the breakurl.sty package is required which doesn't seem to be installed by default everywhere.





I'm writing all this because I already started an attempt to continue with LaTeX instead of SGML back then. It would be really nice if people/users could provide input (in any way) on which direction to go...

Within the past few days, many minor problems which can reproduceably crash mutt-ng were fixed. As part of the re-organization of configuration variable handling, there likely are huge problems with regex variable handling etc.





Also, the DelSp=yes handling was incorrect for empty lines as numerous people have reported.





Please upgrade to a revision after 501 as soon as possible.





New packages will be available soon.

Revision 472 contains three more pre-defined variables: $muttng_pwd (for the current working directory), $muttng_folder_name and $muttng_folder_path for refering to the currently opened mailbox.



This really rocks in connection with a very simple folder-hook:


folder-hook . 'source ~/.mutt/score-$muttng_folder_name'


with which we now have a single command to read in scores when entering a folder. This may be especially useful in connection with newsgroups to separate and modularize the config as lots of scoring rules really slown things down.


For more of such useful info to be queried/used at runtime, just contact us...

Peter J. Holzer kindly contributed a replacement for the format=flowed handler which has been further extended to match the update as of RfC 3676. For example, muttng now does space-stuffing when sending messages and handles DelSp. For both fetaures there're still some unsolved questions or interpretations of the standard to make though...

Within the last days the internal handling (and partially parsing) of config variables has changed greatly. As a side-effect, the following two features are now available: (1) not only environment variables but all config variables can be used via the usual $foo notation and (2) users can freely define variables by prefixing them with $user_. These can be used just everywhere including the expansion, etc. See the manual for more verbose examples. In connection to that, a draft feature has been added which is still subject to change: via some $muttng_ variables, muttng provides static and read-only information to be used in configuration (paths, version information and currently the header caching backend). If there's positive feedback, this will be kept and expanded to have portable config files defined only once.

Also, more usability function has been and/or will be added soon. Muttng already can remind of possibly forgotten files to attach (if promised in the message's text) and will soon strip signatures when preparing replies...

Already since June, 14th 2005 there's a port in mail/mutt-ng. I've sent the maintainer a patch which made it in on August 29th. Using the updated port not only a more recent version is available, also qdbm as header caching backend supporting compression can be chosen.