Erlang Software Framework

I’ve been looking at the way the FSM is handling the responses for the IMAP server for quite some time. In many cases it works wonderfully, but there are some cases that it does not do what is needed for RFC compliance.

The primary case of this is when two or more clients are connected to the same mailbox and/or when new mail is delivered to a mailbox that is selected. The second of those two scenarios is more important, but the first is a main design feature that I have always intended for the server.

The solution seems to be to create a mailbox server that handles the responses and potentially handles all interaction with the mailbox as well. This will allow for the IMAP server to keep a queue of messages for individual clients so that they will be able to receive all of the message that they are interested in and potentially reduce the access to the message store as well.

The most notable feature that is required of an IMAP server that I cannot implement with the current design in the NOOP commands ability to send untagged messages with status updates. When a mailbox has an event that changes the status of a message or the mailbox I have no real way to collecting the responses and delivering them to all of the clients that may be interested.

While I have known about this problem for a while, it has now become a large enough issue that I need to reconsider this part of the design before moving forward. I’ve been making notes on the limitation of the IMAP server in it’s current configuration and I think I’m going to take a bit of time to redesign some areas of the server and implement the mailbox / response server to take care of many of these issues.

I most likely and going to redesign some of the record I’ve been using and find better ways to handle the UID and UIDVALIDITY features in the IMAP server while I am working on this problem. All of these things need to be addressed at some point and I think taking care of them now will help the server int he long run. (or this could all be a way that I can avoid working on the FETCH command)

I haven’t been able to code as much as I would have liked over the past two weeks due to some personal issue that came up. This puts me about three weeks behind where I wanted to be with the IMAP server and still counting.

As of this morning I have one command that I need to fully test; COPY / UID COPY. The code looks finished, but I haven’t gotten to testing it in outlook express yet.

I have one command that needs lots of work still; FETCH / UID FETCH. For anyone who followed me when I was writing the IMAP client you know how much I love the FETCH command. It is a huge command that is more complicated then it needs to be, but I think I am about 40% through it and I have the design pattern down now. It’s pretty much a matter of sitting down and and implementing the rest.

That leaves 4 commands that I am putting off till the next version; STARTTLS, AUTHENTICATE, APPEND and SEARCH / UID SEARCH . I may change my mind on APPEND and include it in this version, but I don’t think so.

After that there is still a lot of work to be done to make the IMAP server RFC compliant. At this point I am intending to make it functional and go for RFC compliance later.

I’m still working with Outlook Express as my test client and I’m hoping to have the test server working (not finished or RFC compliant) within the next two weeks.

I was working on the IMAP server this morning and I realized that I had an annoying pattern that I needed to generalize. Throughout the code I have been using the state of the FMS to figure out which is the correct store to get information from and then calling a specific command from that store. Depending on the actual process that is doing this it can take between 2 and 4 lines of code.

I started to develop an abstraction layer inside the gen_store, so that you can now call the command you are using from gen_store, giving it the data needed and the state data, and it will figure out the correct store to use to call the command from the state data.

The way the information is being processed does not change, but the way the user calls the data does. The user no longer need to figure out which is the correct store, it only calls the function from gen_store and lets gen_store figure it out. This simplifies the user process down to one line of code instead of as many as 4 and only minimally increases the size of the gen_store module. 

 I’ll be re-factoring this into the code as I go along, but I am looking forward to simplifying this task quite a bit.

This afternoon I had a great conversation with a few of my friends that are starting some projects and are interested in using Erlang at the core of them. I’m not going to get too much into what the projects are, but they are around working with Microsoft technologies.

That makes Erlang a wonderful language to use especially when looking at Active Directory, which is based on LDAP. Erlang has a built in ASN.1 library that the really hard work of working with LDAP out of the picture. I know there are a few other people out there working on similar projects.

This conversation also had me thinking about integrating Outlook more tightly into ErlMail. To that end I may try to write a set of MAPI server/client modules into ErlMail. This will be a low priority task, unless I get a lot of interest in it. Plus I’m way behind on the IMAP server right now anyway and I have a few other project that I want to spend my programming time on before this.

I always get great feedback via email when I put up posts like this and it helps me keep focused on what others think are good technologies to have written in Erlang. So if you have any thoughts about this let me know.

I’ve been enjoying using Google Code and subversion in general for ErlMail. So much so that I decided that I wanted to put up all of my current code so that I can access it all the same way. This was mostly motivated by how easy it was to take my laptop to a WiFi hot-spot and continue working when I needed to be away from my apartment this morning. Now I’ll be able to work on any of my current code so long as I have Internet access.

 For those who are interested:

ErlDir: http://erldir.googlecode.com (Mostly just DNS encoding/decoding modules)
ErlMail: http://erlmail.googlecode.com (This is the most active project at the moment)
ErlVoIP: http://erlvoip.googlecode.com (This project has not been started yet)
ErlWeb: http://erlweb.googlecode.com (Not completely sure of the status if this code, ERML works great)

This morning I started testing the IMAP server in ERlMail-0.0.6 with Outlook Express. This is the first IMAP client I intend to test and it was the easiest to setup without messing up my own email

I’m not sure if my interpretations of the RFCs are way off or if Outlook Express is not following the RFCs. I hadn’t seen much references to double quotes (”) around strings, but Outlook Express seems to like putting double quotes around all strings. In the RFCs I only found a mention of them in LIST and LSUB commands. In any case, I’ve spent more time then I care to admit to figuring out how to implement code that handles the double quotes without side effects. So far so good.

The really great news is that I can LIST and LSUB folders, SUBSCRIBE and UNSUBSCRIBE folders and CREATE and DELETE folders as well. (At least I can now that I’m handling double quotes) I’m hoping that several other commands are working already as well too, but I haven’t gotten around to testing them yet; or rather I need to get the FETCH command working before Outlook Express will do much more.

 For anyone who is interested in watching the development more closely or testing with a different IMAP client you can connect with the following settings:

Server: imap.erlmail.net
User Name: simpleenigma@erlsoft.net
Password: erlmail 

This account is for testing purposes only (read: lots of spam) and the server is only up and running while I am in active development. Which usually means the sun is up in California