mutt-ng development blog

mutt-ng isn't dead!

nospam@example.com (krennmair) — Sat, 28 Oct 2006 13:51:01 +0200

As some of you may have noticed, mutt-ng development came to a halt quite a while ago. The problem was that mutt-ng, in its original form, wasn't really maintainable anymore. We had huge issues merging changes that happened in mutt, as well as newer versions of the patches that we had integrated into mutt-ng. This led to a total lack of motivation to push mutt-ng development forward, and eventually all efforts were suspended.

Recently, I collected my thoughts about what could have been done better in the whole development, and my conclusion was that a fork can't survive without a certain user community, and the mutt-ng community never really got that big. Another big issue was the lack of the right tools, or at least the knowledge about them, e.g. we had absolutely no patch management.

The consequence of all that was that I decided to "restart" the whole project, but with different goals. The new mutt-ng isn't supposed to be a fork anymore, but instead a well-integrated set of patches against the latest version of mutt. I experimented a little bit with different patch management tools, and decided to use quilt in combination with a Subversion repository. For those who don't know quilt: it manages a stack of patches, and you can push and pop them. The patches themselves are stored in a subdirectory called "patches". When you're a certain patch, you can edit some files, and then refresh the patch so that your changes are reflected in the corresponding patch file. To change a patch to a newer version, all you need to do is to pop as far as the patch before the patch to be replaced, copy the new patch over the old patch, and push it. If there are any conflicts, force the push, correct the conflicts, and refresh the patch. Then push the rest of the patches, which usually works with no or little extra work. This means that managing a relatively big set of patches is quite simple, while quilt manages all the "dirty" stuff.

So far, I managed to reintegrate most of the patches we had in mutt-ng, and also to port some of the mutt-ng-specific features and bugfixes. The new mutt-ng isn't as complete as the old one, but its pretty usable, and the most important things like sidebar and NNTP support work already. If you want to have a look at the current state, read this description on how to apply these patches. As stated above, you will need quilt in order to use it.

Feedback is welcome, and I would also like to know if this new development model is desired and accepted by the community.

Insight Rewrite: Licensing

nospam@example.com (Rocco Rutte) — Thu, 19 Jan 2006 12:03:46 +0100

Today's topic is more of a religious nature: Licensing.

Until now, all files have been just licensed under the GPL. We now switched to a mixed system which makes sense for us.

The new licenses we use include: GPL (of course), LGPL and public domain.

To start with the latter: we took some minimal implementations from mailing lists or other publicly available archives and silently used them. To tag them as GPL is not correct as by appearing publicly we consider these to be public domain already. Also, we put the examples of the manual and some tool scripts under it. The GPL is fine but for some simple scripts this is just overkill.

LGPL. In the very end we want to at least have the core part under LGPL. The reason is quite simple: there're no real algorithms but rather wrapper functions putting libraries (which are themselves under LGPL) behind a convenient API for use in muttng. As the underlying libraries are LGPL to have proprietary software benefit from it, why shouldn't benefits from our abstraction layer be possible? Currently only the files we wrote completely from scratch are put under it. Files containing portions of old code are left under GPL, of course.

GPL. Right now the libmuttng part contains just too many fractions of old code so it cannot be simply put under LGPL. We hope we have time to replace these parts step by step so we can replace GPL-dependencies of our new code by new LGPL-code to, in the end, make libmuttng licensed under LGPL, too. The muttng part (i.e. all tools and applications) will remain GPL as "just an example of how to use the libraries."

As it turns out, there always are dicussions when (partial) license switches appear. So everyone is invited to comment on it.

If somebody discovers parts of old code now tagged as being LGPL, please notify us as this is a serious mistake and license violation not done intentionally.

mutt-ng at 22C3

nospam@example.com (krennmair) — Mon, 26 Dec 2005 11:20:40 +0100

From December 27th to December 30th 2005, the 22nd Chaos Communication Congress (short: 22C3), a congress on "technology, society and utopia", also known as the "European Hacker Conference", is taking place in Berlin, Germany. Chaos Communication Congress proved itself to be a wonderful place to hack, to collaborate and to exchange ideas, and to meet new and interesting people.

This year, we decided to register seats for the mutt-ng project at the Hackcenter, where most of the hacking is about to takeplace. So, if you want to meet us, talk to us, or contribute code and/or ideas, just stop by. Nico Golde and I (Andreas Krennmair) will be there from the 26th in the evening on, and Rocco Rutte will most likely join us on the 28th. We hope to get as much as possible done at the Congress, especially the ongoing mutt-ng rewrite will have our attention. See you in Berlin!

Insight Rewrite: IPv6

nospam@example.com (Rocco Rutte) — Thu, 22 Dec 2005 13:22:41 +0100

Today's topic isn't that exciting after all: IPv6.

I just committed an update of the Connection class and the URL parser to work nicely with IPv6. However, there's one thing to note so that I write this blog entry.

What changed (somewhat) is the basic URL syntax. It still remains as:

proto[s]://[user[:password]]@host[:port][/path]

...but for IPv6 the host part is somewhat treated specially if it's an IPv6 address rather than a domain name (in which case the DNS resolver will do the job together with the IPv6 stack of the operating system.)

IPv6 addresses have to be enclosed in [] since they contain colons (':') which are used to detect whether and what port is embedded in the URL in case it's non-standard for the protocoll.

Also, please keep in mind that all parts of the URL are expected to be encoded. This is important for IPv6's link-local addresses with KAME-derived IPv6 stacks such as found on the BSD systems. With KAME, an interface may be embedded like so:

fe80::1%lo0

which is the link-local address for the loopback device fe80::1 reachable via lo0 (of course, just ::1 is shorter but this serves only as an example for non-KAME users.)

In short, the percent sign ('%') is used in URLs for marking an encoded sequence so that itself has to be encoded as %25. A correct example IMAP URL to loopback for user joe with password secret using a secure connection to port 4711 would thus be:

imaps://joe:secret@[fe80::1%25lo0]:4711/

or shorter:

imaps://joe:secret@[::1]:4711/

Also, the code handling connect(2) has been replace by a portable and protocol-independent one so that IPv4 and IPv6 are expected to work without problems. As I didn't setup IPv6-enabled local servers and as I don't/won't get IPv6 connectivity, I couldn't test very much. Please report problems if you encounter any.

Testing can be done with the muttng-query tool like so:

$ muttng-query -d 5 pop[s]://ipv6host/

or

$ muttng-query -d 5 nntp[s]://ipv6host/group.

Insight Rewrite: Binaries

nospam@example.com (Rocco Rutte) — Sat, 17 Dec 2005 16:05:15 +0100

Today's topic for the rewrite features is: binaries.

As some may have seen from the doxygen documentation and/or the source, several binaries are being built and will be installed. There'll be one main muttng binary and several others being a symlink to it. Upon startup, the binary name is detected an muttng behaves differently. This article briefly explains what these binaries will do as my documentation patch is not yet committed.

Of course, muttng(1) will remain the main interactive client with a curses-based interface. This is also the main binary being installed.

muttng-conf(1) will be, inspired by the postconf(1) command, a command for configuration diagnostics. Currently, the following items are already supported: print variable names and values and print function bindings. For these and all future groups of items, there are the following three modes planned: (1) just print everything as-is, (2) print only changed items and (3) as an add-on to the other two modes, support an "annotated" mode. "Annotated" means that it prints comments with defaults and other descriptions along the main output. Output will be printed in the muttngrc syntax so that the output can be used as a configuration file itself.

muttng-mailx(1) will be a true mailx(1) replacement. Now that there is a command-line option to enable mailx-compatibility. However, it doesn't make sense as scripts or the like must be ported to it. We will turn this into a true replacement so that it, for example, can be used with mechanisms such as Debian's alternates. This will be a really cool feature as all tools will use the users configuration. For example, with it all the hooks and other features being available for the mailx-replacement, users can setup cronjobs and their mail delivery very flexibly.

muttng-query(1) is a temporary addition and it depends on user feedback whether it will stay or not. As the user interface(s) are last on our "roadmap" (because we want to ensure the two library layers are working properly before we start to use them), we need something to test libmuttng with. The query tool will provide a command-line way of dealing with any type of folder. Also it will support a colon mode similar to that of gpg(1) so that output can be parsed by scripts easily. Planned features are listing which mailboxes have new mail, maybe also print which mail is new, maybe search accross folders, etc. Before its addition yesterday some testing code was implemented in muttng(1) where it doesn't fit so I simply created this tool. As said, we're doing a rewrite and are interested in feedback and critics of any type so it depends on users whether this tool will stay and what features it'll offer.

muttng-sync(1) will be a tool supporting bi-directional syncing of arbitrary folders. Possible uses range from simple offline IMAP to syncing two Maildir trees with a workstation and a laptop. As it'll also be built upon libmuttng which will provide transparent support for different mailbox types, it'll be theoretically possible to mirror remote newsgroups to a local Maildir or MBOX file as well as a remote IMAP folder. Not only because users requested an offline IMAP feature (which muttng-sync(1) will be the generalized solution to), but also to test whether the abstraction we make in libmuttng, this tool serves well a real-world test environment.

Any feedback is welcome...

Insight Rewrite: Debugging

nospam@example.com (Rocco Rutte) — Thu, 15 Dec 2005 17:07:33 +0100

Hereby I start a new series of articles called "Insight Rewrite" which will give some details about this work or will work. Today's topic is: debugging.

Debugging support is essential as with unit testing we can only verify that a single function does what it's supposed to but not really/completely whether they're also used the right way.

Debugging will be available, as before, via the -d command line switch for all tools as well as via the $debug_level option. The reason behind the config option is that now users can freely turn debugging on and off during runtime as well as only change the verbosity level.

Before there were only a few debug files kept which is a "feature" removed in the rewrite. Now debug files will remain until they're deleted by users. Also, the naming is different: files will be named as: ~/.[scope].[pid].[id].log. These fields have the following meanings: "[scope]" is either "libmuttng" or one of "muttng-conf", "muttng-sync", etc (yes, debugging is done to different files for the libmuttng library and the actual application). "[pid]" is the current process' ID number. As debugging may be turned on and off during runtime, the "[id]" field starts off with "1" and will be subsequently increased everytime debugging is turned on again.

New is also the fact that each of the debug levels from 1 to 5 has it's own meaning, i.e. we don't choose a message's debug level by our current mood any longer. For example, level 1 is for all module registration and de-registration, level 2 for parsing, 3 for signals and events, etc.

As before, each line will contain the caller besides the message. New is that lines are space-indented to more quickly find things as debug files may grow quite large.

As some may have wondered why we organized the inheritance for libmuttng as well as applications the way we did: the debugging support is implemented in libmuttng and publicy visible (and thus useable) by applications so that the base classes LibMuttng and Muttng for both layers contain such a debug object. As a consequence, all classes have debug support. The level change is connected via the configuration manager and a signal to achieve separation.

As today and yesterday I ported a number of core/ functions required by the configuration parser, and finally added a first one simply printing each line read to the debug file, brave users can already get, compile and run the muttng-conf binary and see the files created in $HOME.

Next up on my todo list are all the unit tests still missing for these functions added so it depends on how much spare time I have whether we'll have a configuration parser to finish the year with.

(Btw, we developers will meet at the Chaos Communication Congress in Berlin at the end of the year and applied for its hackcenter so maybe we can meet users and/or write the parser.)

Update and background information on the rewrite

nospam@example.com (krennmair) — Mon, 12 Dec 2005 13:12:01 +0100

As some of you probably saw already, mutt-ng is currently in a rewrite phase. I would like to give some insight on why we started the rewrite and what's actually going on. When we were trying to improve mutt-ng with more and more features, we came across the problem that it was very difficult to actually identify and fix bugs, and how changes on some parts of the code would influence the overall behaviour and probably would add new bugs to the code.

So we came to the conclusion that the only feasible way that would make it easier for us in the future would be a well-documented rewrite, as we had troubles seeing overall design goals, in fact, there was no documentation at all for the original mutt(-ng) design. So first, we started drawing UML diagrams of the overall design, using ArgoUML. These designs can be found in our Subversion repository. As soon as we had a reasonably usable design, we started with the implementation in C++, which is currently going on. Why C++? Because it's language that makes it reasonably easy to provide a clean mutt-ng implementation that reflects the UML design and makes it possible to take advantage of object-oriented features while retaining compatibility with existing mutt-ng code pieces, making it relatively simple to reuse existing (pre-rewrite) mutt-ng code.

As mentioned before, one focus is on documentation. Rocco put some effort into documentation building: complete developer documentation of the source code can be automatically generated. Even better: the documentation is generated once a day from the most current source code, and put online here. The user documentation is auto-generated. More general information about the rewrite can be found on the mutt-ng rewrite page. This is also something where anybody who wants to contribute can do so: the user documentation and the rewrite pages are created in such a way that it's simple to translate it to another language. So, if you want to contribute to mutt-ng, just translate the user documentation to your native language. So far, English and German are available. Although there isn't much to document yet, we think that providing the documentation in as many languages as possible makes it easier to get a grip on mutt-ng, to understand it and to get them contribute by themselves.

Besides documentation, there is another important goal on the rewrite: as much of the mutt-ng functionality as possible should be automatically testable. The buzzword for this is unit testing. The tool of choice for this is unit++, a very C++-ish unit testing support library. To meet our own expectations of high-quality code, we wrote unit tests for the core library and the mutt-ng library (yes, very modular design) practically in parallel, a development method that proved to be useful especially in big projects: as soon as a test breaks, you know that something has gone wrong, so continuously developing and testing makes it easy to detect hidden bugs. For refactoring, this is also very useful, as the unit test is part of the specification, and you can easily test your code whether it conforms to the previously defined (and expected) behaviour. We think that this will very valuable for future development.

If you're interested in contributing to the mutt-ng rewrite, don't hesitate to submit patches. But please don't forget to provide useful documentation (a look at the header files shows what we mean) and unit tests. Documentation makes your code easier to understand, and the unit tests show that your code works, so providing both raises the chance of getting your patch included.

In the future, we hope to be able to provide more and especially more often background information on the rewrite status.

Rewrite in progress

nospam@example.com (Rocco Rutte) — Tue, 08 Nov 2005 07:59:55 +0100

For the last couple of days and weeks, there isn't much going on with mutt-ng for several reasons.

First, we, the core developers, have come to the conclusion that a "next generation" mutt really needs a rewrite or better: a (documented!) design made with current and future features in mind. An example which is just impossible to implement right now easily in an elegant fashion is transparent syncing of different folders like remote to local IMAP, remote to remote IMAP, local to (NFS-)local Maildir, etc. With the re-design ideas we have so far, this will be easy as it will be built on top of a "mail storage abstraction."

Second, we have come to the conclusion that when adding features, we spend most of the time trying to understand the complexity behind the current code to find the proper place for the change (sometimes we were even stuck.) Thus, some of the things we already changed only improves the situation from a user's point of view, but adds more complexity making it harder to maintain the code base in the feature.

A third and last reason for why we need a rewrite is simply the fact that we included many patches already. The main problem we experienced is that these patches are just add-ons, that is, they add features but for compatibility are not allowed to improve the underlaying design or interfaces to implement a feature more easily. Also, many patches are standalone and don't really integrate or work well with one another. We found it too difficult to actually integrate all of the foreign code, re-design everything "in-place" and thus make everything fit into a consistent mail client.

Andreas, the maintainer, has opened a branch with early ideas which are subject to change and which will be implemented as soon as they turn out to be the way to go.

Everybody is invited to join and start discussions on the developer's list.

As I maintain an automated blog for the mutt-ng subversion repository, you can also stay up-to-date on the rewrite branch via the following URL:

http://user.cs.tu-berlin.de/~pdmef/muttng.cgi?c=muttng-rewrite.

There you'll also find various RSS, Atom and even MBOX newsfeeds.

Scoring in mutt-ng

nospam@example.com (krennmair) — Wed, 19 Oct 2005 09:12:42 +0200

In revision 553, we fine-tuned the scoring functionality of mutt-ng in order to make it more usable, especially for reading newsgroups. Basically, the fine-tuning consists of enabling patterns to match against the realname of headers containing mail addresses, e.g. From:, Cc:, To:, etc., which was deactivated before for whatever reason.

Continue reading "Scoring in mutt-ng"

Better documentation

nospam@example.com (Rocco Rutte) — Sun, 09 Oct 2005 15:39:42 +0200

As readers of the mailing lists and/or ChangeLogs may have noticed, some time ago I added an ad-hoc stylesheet to transform the DocBook/XML based manual into LaTeX.

As the manual itself isn't valid and plain DocBook but contains some goodies to make it more consistent and easier to write, so does the LaTeX (stylesheet) file. That means it's not a general DocBook to LaTeX conversion solution.

At our Berlios site, there're two links given: one for a PDF generated by pdflatex(1) and one PostScript file generated by latex(1).

The Makefile in the source's doc subdirectory contains two targets to build them: manual-ng.pdf and manual-ng.ps whereby for the latter the breakurl.sty package is required which doesn't seem to be installed by default everywhere.

I'm writing all this because I already started an attempt to continue with LaTeX instead of SGML back then. It would be really nice if people/users could provide input (in any way) on which direction to go...

Bug fixes

nospam@example.com (Rocco Rutte) — Sun, 11 Sep 2005 20:06:37 +0200

Within the past few days, many minor problems which can reproduceably crash mutt-ng were fixed. As part of the re-organization of configuration variable handling, there likely are huge problems with regex variable handling etc.

Also, the DelSp=yes handling was incorrect for empty lines as numerous people have reported.

Please upgrade to a revision after 501 as soon as possible.

New packages will be available soon.

More pre-defined variables providing more flexibility

nospam@example.com (Rocco Rutte) — Wed, 31 Aug 2005 23:31:42 +0200

Revision 472 contains three more pre-defined variables: $muttng_pwd (for the current working directory), $muttng_folder_name and $muttng_folder_path for refering to the currently opened mailbox.

This really rocks in connection with a very simple folder-hook:

folder-hook . 'source ~/.mutt/score-$muttng_folder_name'

with which we now have a single command to read in scores when entering a folder. This may be especially useful in connection with newsgroups to separate and modularize the config as lots of scoring rules really slown things down.

For more of such useful info to be queried/used at runtime, just contact us...

Improved RfC 3676 aka f=f support

nospam@example.com (Rocco Rutte) — Tue, 30 Aug 2005 21:14:48 +0200

Peter J. Holzer kindly contributed a replacement for the format=flowed handler which has been further extended to match the update as of RfC 3676. For example, muttng now does space-stuffing when sending messages and handles DelSp. For both fetaures there're still some unsolved questions or interpretations of the standard to make though...

More flexibility

nospam@example.com (Rocco Rutte) — Tue, 30 Aug 2005 21:11:26 +0200

Within the last days the internal handling (and partially parsing) of config variables has changed greatly. As a side-effect, the following two features are now available: (1) not only environment variables but all config variables can be used via the usual $foo notation and (2) users can freely define variables by prefixing them with $user_. These can be used just everywhere including the expansion, etc. See the manual for more verbose examples. In connection to that, a draft feature has been added which is still subject to change: via some $muttng_ variables, muttng provides static and read-only information to be used in configuration (paths, version information and currently the header caching backend). If there's positive feedback, this will be kept and expanded to have portable config files defined only once.

Also, more usability function has been and/or will be added soon. Muttng already can remind of possibly forgotten files to attach (if promised in the message's text) and will soon strip signatures when preparing replies...

FreeBSD Port available.

nospam@example.com (Rocco Rutte) — Tue, 30 Aug 2005 21:06:50 +0200

Already since June, 14th 2005 there's a port in mail/mutt-ng. I've sent the maintainer a patch which made it in on August 29th. Using the updated port not only a more recent version is available, also qdbm as header caching backend supporting compression can be chosen.