Erlang Software Framework

I’m looking into working with a few developers on several of the projects in the Erlang Internet Framework. To that end I’m starting to work with subversion and I have uploaded the latest code for ErlMail-0.0.6 up to Google Code.

http://erlmail.googlecode.com

I’m still working with figuring out exactly how subversion works and how to incorporate it into my own daily routine, but this will definitely open up some doors for collaboration with others and that is more important then anything else.

I’m hoping to remember to commit my changes at least once a day and I intend to make sure that the trunk at least compiles every time I work with it.

I also intend to put ErlDir up when I re-factor it at some point in November and ErlWeb, which is mostly ERML at this point, will go up at some point as well. If you want to contribute code to any of these project email me and I’ll upload them sooner.

I’ve been working on the IMAP server in ErlMail-0.0.6 as my primary project since I release ErlMail-0.0.5 last week. I think I’m about a third through the server and for what I’ve been doing I think it’s going pretty fast.

 The main server is based off of my SMTP server for the TCP communications and the basic file structure. Although in the past few days I’ve been implementing some new strategies that I will need to put back into the SMTP server as well. Most of that code is around the extensions to the IMAP (or SMTP) protocols. I created a modules called IMAPD_EXT that is run when the IMAP server does not recognize a command. This modules looks at the config file to see the list of known extensions and then verifies if they extension is loaded in the path on the current IMAP server. The extensions are listed in the config file as “Module1:Funcation1 Module2:function2 Function3″ When the module name is left off the default of imapd_ext is used for the module name. With this setup the IMAP server will be able to be extended without recompiling the ErlMail code. By adding an extension in the config file and making sure it is located in the search path that the Erlang run-time uses anyone will be able to extend the IMAP (or SMTP) server with minor effort. (Depending on the extension of course)

I’ve also been working a lot with the modular message store. This has already changed significantly since ErlMail-0.0.5, which was the first version to have the message store in it. I’m finding that the IMAP server needs to have a decent amount of it’s logic in the message store, since the way the files are being accessed in each message store is so different. Most notably the Mnesia message store and the Maildir++ message store are light years away from each other in the logic they use to return the same results from each function in the module.

In effect this means I am writing about 25% of the IMAP server in each message store and I’m planning on having two to four message stores in the next release. This was more effort then I was expecting, but the results are more then worth it.

I’ve gotten another report of a bug in the IMAP client. Actually it the same bug, but it’s manifesting itself slightly differently now. I’m thinking this particular bug my require a rewrite of the IMAP grammar files for both LEEX and YECC. I may have to put this off till the next version of ErlMail, as this will be quite time consuming.

I’m hoping to get the next version of ErlMail out in the first half of November now. Hopefully I’ll get plenty of time to spend on it and it will be out sooner then that.

I just uploaded ERlMail-0.0.5 into the downloads section of the website. This is a major reworking of the previously released modules and the first release of the generalized message store (gen_store) along with the DETS messages store (dets_store).

 The SMTP server module has been completely reworked and the TCP communications problems have been fixed. The server has also been expanded to use the dets_store module to save the messages that it receives and check for users.

The SMTP client has been reworked, but it is not a major change.

The IMAP client has been completely rewritten. Most notably the TCP communications and the error handling have been significantly updated. This version of the IMAP client is most likely not directly compatible with the previous versions. Some testing and integration of the new return values will be needed for anyone using a previous version.

The next version will include the initial release of an IMAP server along with a mnesia_store and hopefully a maildir_store. I still have hopes of releasing ErlMail-0.0.6 during this month, but it would be more realistic to see it in the first half of November.

I’ve been working with Erlang as my primary programming language for about three years now. (Don’t quote me on that, it might only be two years) In any case I learn new techniques almost every day and with everything that I learn the language itself seems to get simpler and the number of lines of code I need to accomplish something goes down.

I was really noticing this on my implementation of the Flash Media server, which uses an odd proprietary protocol that was create by Macromedia (now adobe). I was implementing a section of code that was parsing the header for the packet. My first rendition was nearly 100 lines of code. When I was showing this to a colleague of mine he expressed that the way I had coded it was exactly the way he would have as well, which I consider standard practices for most programmers.

 This same section of code is now down to 7 lines of code, it is easier to read and I suspect it is much faster if for no other reason that is has fewer conditional statements in my code. The simplicity of the newer version of this function came from Erlang’s pattern matching in general and the amazing way that Erlang performs binary pattern matching. I was able to write patterns instead of conditional statements and set variables within the patterns. I understand that python has some features like this as well, but I have never used python or any other language that create code that is so succinct.

After realizing what had happened with this one function I started looking at some of my other code from the past few months to try and find similar patterns that I might be able to reduce. I found plenty of the in the SMTP and IMAP clients from ErlMail and I plan on releasing the new version of ErlMail-0.0.5 Friday or Saturday of this week. (I want to add some more documentation into the code so I remember what was going on and other people can use it as well, other then that it is done) 

After I write the first version of the IMAP server I am working on next I plan on revisiting the DNS server code I had been working on. Since the DNS protocol is a binary protocol the pattern matching that I learned while working on the Flash Media Server will almost directly translate into the DNS server. I looked at the code in one of the modules before writing this post and I saw three different functions that I could reduce with just 1 or 2 minutes of scanning the code. I’m planning on paying more attention to how many lines of code I started with and how many I am left with after re-factoring and I’ll publish that some time in the future. I hope to be able to get to that at some point in November or December, unless I get stuck on the IMAP server and need a break from it

I was hoping to be releasing ErlMail-0.0.5 today, but I’ve been obsessed with re-factoring the IMAP client finite state machine (FSM) and I haven’t even started on the SMTP client yet.

I found a few errors in the LEEX and YECC scripts that I had built for the IMAP client and I was in the process of moving the TCP communication inside the FSM instead of in it’s own files when I started to see some better ways of handing the processing. I’m making the error reporting much more robust and I’m fixing some small glitches in the FSM on the way. I’m also adding in more code to make sure you cannot perform commands when you are no in the proper state to perform them, along with error message that tell you that you can’t do that and why.

The great news about re-factoring both the IMAP and SMTP client is that if you have been using the function calls from the main modules (IMAPC or SMTPC) and not the FSM (not IMAPC_FSM or SMPTC_FSM) then you will not have to change your code for anything except perhaps better error message.

I’m about half way done with the IMAP client and the SMTP client is much simpler; plus I already know how I want to change it. The SMTP server and the DETS message store are working well and stable enough for an initial release. If things work out well ErlMail-0.05 will be release early next week.

Technorati Tags: ErlMail, SMTP, IMAP, Erlang

I put up a demo server for ErlMail that I an using for testing purposes. Right now it does a complete SMTP conversation and then discards the email message. I have a few of my retired domain names pointing their MX records to the demo server so that I have plenty of email messages (read: spam) to play with. The demo server is at smtp.erlsoft.net if you want to telnet in to port 25 and play with it. You can also send email to simpleenigma@erlsoft.net for test messages that will be eventually used for testing, those message will be used for testing and otherwise ignored.

I’m working on the message store at the moment, which will allow ErlMail to store the messages for users in domains. I’m implementing a modular message store so that it can be easily replaced. The message store is basically a gen_store behavior that you define your own functions for. The ErlMail server can have different stores for users, domains and messages, so it is going to be possible to have users and domains in Mnesia and the messages on the file system in a maildir store. I am currently working with a DETS store to start with, but before I release the complete store I plan on having  DETS, ETS, Mnesia, MySQL and Maildir stores and I might even implement an XML store.

I’m hoping to release ErlMail-0.0.5 with the revised SMTP server and the DETS message store sometime this week.