Erlang Software Framework

The idea of the Erlang Internet Framework, or EIF, was created at the beginning of 2007 therefore it is fitting to do a status update and ponder on the future of the project one year after it initial conception.

The EIF was originally created to be a way to combine all of the projects that I had been working on into one focused project, since then it has grown and several projects have been added. The original projects beyond the EIF were ErlMail, ErlWeb and ErlDir. Each of these projects are still in early development stages and they have been joined by ErlVoIP, a Voice over IP server, and ErlMedia, a Flash Media Server written in Erlang.

ErlMedia/ErlyVideo

The code base for my flash media server, which I was calling ErlMedia has been joined with ErlyVideo, which is another Flash Media Server written in Erlang. Since my initial work on the ErlMedia/ErlyVideo in August and September I haven’t been able to spend much more time on the project and the other team members have done a wonderful job of mending the problems that were plaguing the code I was working with and taking it much further then I could have done alone.

I am very hopefully that I will be able to spend a decent amount of time working on this project in early 2008, after the next release of ErlMail.

ErlMail

ErlMail has been the focus of my attention for most of 2007. The distinct lack of a complete email package, including library modules for both client and server, has been my inspiration for this projects since it’s inception. In my personal opinion, ErlMail has the most advanced email modules of anything currently available as open source in Erlang.

For the past several months I have been working diligently on the IMAP server in ErlMail. I knew this server was going to be difficult to create, but I had severely underestimated the amount of time it was going to take to make an IMAP server operational. I am down to the last command, the FETCH command, in the IMAP server before I release a stable version of it. It is not going to be RFC compliant in the next version and there are still a few commands that I will need to include after the next version as well, most notably the search command.

The feature that I think is the most notable in the ErlMail package is the modular message store. The modular message store allows users to create their own message store modules and simply change the user configuration file for ErlMail to use them. This feature already garners more interest then any other single feature in the ErlMail package and with my future directions, which you will read later, it will become even more important.

ErlWeb

ErlWeb, which is a web server and associated modules, got one official release in 2007. The most notable piece of this release is the tag based scripting language called Erlang Markup Language, or ERML. My intention with ERML is to create a way that people who do not know how to program in Erlang can still get the benifits of Erlang. The most recent shift in the design of ERML has been to add a way to extend ERML with user create modules that are not part of the ErlWeb package. This is being done now with the other packages in the EIF.

For instance, a module in ErlMail called erml_erlmail will have all ERML commands that are associated with email commands for use in the ErlWeb application. This allows for users to create their own ERML commands, written in Erlang, and include them into ErlWeb without recompiling the system. In the case of ErlMail, this also allows all of the email logic to be contained in one application while still interacting with the ErlWeb application. This feature appeared in the code base in December of 2007 and I consider it as important to ErlWeb as the modular message store is to ErlMail.

Similar files have been created for other EIF projects; such as erml_erldir and erml_erlvoip which are located in their respective applications.

Other Projects

The rest of the projects included in the EIF have not moved forward much in 2007. ErlDir did get an official release which is mostly library modules that handle packet encoding/decoding and compression/decompression and ErlVoIP has been announced although work on this project has not begun yet.

Google Code

During the second half of 2007 I moved all open source projects that I have onto Google Code. While this was a move on my part to allow me to be more portable with my development environment and create a pseudo backup system to ease my own mind this also allows for other people to join the projects and contribute their own code if they wish. I have one person who is helping me document (and correct my spelling errors) most of the projects already and I am open to the idea of other people joining the work to see these projects move forward faster then I could accomplish alone.

Future of EIF

I see a bright future for the EIF in 2008; starting with a new release of ErlMail in the first few weeks of the year. After that there are a few changes to the core idea of the system that I intend to implement. These changes reflect new innovations that were not around during the EIF initial conception and large needs that have arisen since then as well.

The EIF Director

I’m not sure I have ever described the main project for the EIF itself. The EIF project is intended to be a manager application that keeps track of the versions of the other applications and which nodes have what servers running. So while each application in the EIF will be independent, the EIF application will be the glue that holds them all together.

Some of the features that are planned for the future of the EIF application itself are:

• Upgrading itself and other EIF applications
• Monitoring which nodes are running what servers
• Load balancing traffic across all nodes
• Fail-over on node crashes; allowing a user connection to move from one node to another without the user noticing

The EIF application would handle much of the communications and some of the state data for each connection, and then the server processes on each node would do the work that each connection is requesting. The design goal of this is to allow nodes to be added or removed into the system without users noticing any interruptions. This would include the ability to move a long session that a user has, on say an IMAP server, from one node to another without the user needing to reconnect. I am also hoping to implement this into ErlWeb so that comet connections could be moved from one node to another without the user noticing that a node has been removed from the cluster.

Amazon EC2/S3/SimpleDB integration

The Amazon web services, or AWS, announcements, that have been on going for several months, have been unexpected and potentially a huge boost to the design on the EIF. I will be taking some time to look at these services and creating a road map to completely integrate these services into the EIF at every level that is reasonable.

The first use of the AWS will be in ErlMail. I intend to create a modular message store that incorporates the AWS as the backend, most likely using the SimpleDB for user and domain information and the S3 storage for the actual messages.

After the ErlMail message store is integrated with the AWS all other project will be designed to work with the EIF Director on EC2. With the flexibility of the EC2 cloud any number of Amazon Machine Images, or AMIs, could be made with different server configurations. Then as the EIF director noticed a burst in traffic or number of connections the EIF director could initiate a new instance of an EC2 AMI. When the load has lowered enough the EIF director could move sessions onto the oldest running instances (without users noticing) and disconnect from instances that are no longer needed.

The core configuration of the EIF on AWS could start out as one instance with the EIF director and all other servers on the same instance and as load increases the EIF director will expand or contract the needed nodes. As more nodes are added the EIF director will move session that are active on itself onto other nodes allowing the EIF director to focus it’s resources on controlling traffic flow and balancing connections. Eventually as load decreases the EIF director would take the sessions back onto itself to minimize the number of instances back down to one instance.

If implemented well this could create a system where no matter what type of services you want to provide you would be able to use the AWS to create the project and if the need arises the project could be moved to a dedicated hosting facility without the users noticing the move and zero downtime. Once you are on your own hardware when you manually add new nodes to your system the EIF director will immediately start making use of them.

I see this as a boon to potential startups as the will be able to grow and move resources around in real-time without needing to incur major expenses up front. Once they are making money the can build their own infrastructure at their own pace and potentially still use the AWS when the need more capacity to their own infrastructure.

Conclusion

While 2007 was a great start to the EIF and all of the applications, 2008 looks to be even better. I’m sure there will be more shifts in direction and changes made to the ideas behind the system as the start to get implemented, but the core goal of making the most flexible, stable and powerful Internet development environment will always be there.

I’ve been redesigning some of the core elements of the message store and the way the IMAP server handles responses for the last week. In that time I think I’ve come up with some solutions to problems that I thought were inconveinces, but not really problems at the time.

After looking at the way I was starting the gen_server for the message store I started to realize that my configuration file was completely wrong. I had created a text file that had one command per line and needed to be parsed and fixed to be used. Default values were difficult and if the user excluded a mandatory line the server would crash

I’ve now moved the default configuration into the .app file located in the ebin directory and the user can specify a .config file on the command line to override any settings they wish. I’ve also built in a function that lets me search for a key and define a default value if the key is not found.

This also lead me to look at how the processes were starting and stopping. Using appmon I found that the client connections were not being closed properly when the STMP QUIT the IMAP LOGOUT commands were issued. I’m not terminating the FSM properly when either of these commands are issued.

Originally I had the SMTP and IMAP servers started independently; now they are started by the erlmail app and there are configuration settings to determine if they are started by default or not.

I’m moving some of the logic from the modular message store into one place; erlmail_store.erl. This is a gen_server that will handle the message store functions and knows which module to call for specific functions. Each of the modular message store modules will have less logic in them; making them easier to create and the other ErlMail modules will call erlmail_store when they need to interact with the store. In the long run this means less redundant code and more control over how the modular message store components interact with the rest of the server.

The IMAP response server is in place and operating, but it now needs access to the message store and those features haven’t been developed yet. The two servers will go hand-in-hand and for a while I will be developing the two in tandem.

It feels like I haven’t added much functionality lately, but the system is much more stable and more robust from the improvements I have been doing.

I’ve been looking at the way the FSM is handling the responses for the IMAP server for quite some time. In many cases it works wonderfully, but there are some cases that it does not do what is needed for RFC compliance.

The primary case of this is when two or more clients are connected to the same mailbox and/or when new mail is delivered to a mailbox that is selected. The second of those two scenarios is more important, but the first is a main design feature that I have always intended for the server.

The solution seems to be to create a mailbox server that handles the responses and potentially handles all interaction with the mailbox as well. This will allow for the IMAP server to keep a queue of messages for individual clients so that they will be able to receive all of the message that they are interested in and potentially reduce the access to the message store as well.

The most notable feature that is required of an IMAP server that I cannot implement with the current design in the NOOP commands ability to send untagged messages with status updates. When a mailbox has an event that changes the status of a message or the mailbox I have no real way to collecting the responses and delivering them to all of the clients that may be interested.

While I have known about this problem for a while, it has now become a large enough issue that I need to reconsider this part of the design before moving forward. I’ve been making notes on the limitation of the IMAP server in it’s current configuration and I think I’m going to take a bit of time to redesign some areas of the server and implement the mailbox / response server to take care of many of these issues.

I most likely and going to redesign some of the record I’ve been using and find better ways to handle the UID and UIDVALIDITY features in the IMAP server while I am working on this problem. All of these things need to be addressed at some point and I think taking care of them now will help the server int he long run. (or this could all be a way that I can avoid working on the FETCH command)

I haven’t been able to code as much as I would have liked over the past two weeks due to some personal issue that came up. This puts me about three weeks behind where I wanted to be with the IMAP server and still counting.

As of this morning I have one command that I need to fully test; COPY / UID COPY. The code looks finished, but I haven’t gotten to testing it in outlook express yet.

I have one command that needs lots of work still; FETCH / UID FETCH. For anyone who followed me when I was writing the IMAP client you know how much I love the FETCH command. It is a huge command that is more complicated then it needs to be, but I think I am about 40% through it and I have the design pattern down now. It’s pretty much a matter of sitting down and and implementing the rest.

That leaves 4 commands that I am putting off till the next version; STARTTLS, AUTHENTICATE, APPEND and SEARCH / UID SEARCH . I may change my mind on APPEND and include it in this version, but I don’t think so.

After that there is still a lot of work to be done to make the IMAP server RFC compliant. At this point I am intending to make it functional and go for RFC compliance later.

I’m still working with Outlook Express as my test client and I’m hoping to have the test server working (not finished or RFC compliant) within the next two weeks.

I was working on the IMAP server this morning and I realized that I had an annoying pattern that I needed to generalize. Throughout the code I have been using the state of the FMS to figure out which is the correct store to get information from and then calling a specific command from that store. Depending on the actual process that is doing this it can take between 2 and 4 lines of code.

I started to develop an abstraction layer inside the gen_store, so that you can now call the command you are using from gen_store, giving it the data needed and the state data, and it will figure out the correct store to use to call the command from the state data.

The way the information is being processed does not change, but the way the user calls the data does. The user no longer need to figure out which is the correct store, it only calls the function from gen_store and lets gen_store figure it out. This simplifies the user process down to one line of code instead of as many as 4 and only minimally increases the size of the gen_store module. 

 I’ll be re-factoring this into the code as I go along, but I am looking forward to simplifying this task quite a bit.

This afternoon I had a great conversation with a few of my friends that are starting some projects and are interested in using Erlang at the core of them. I’m not going to get too much into what the projects are, but they are around working with Microsoft technologies.

That makes Erlang a wonderful language to use especially when looking at Active Directory, which is based on LDAP. Erlang has a built in ASN.1 library that the really hard work of working with LDAP out of the picture. I know there are a few other people out there working on similar projects.

This conversation also had me thinking about integrating Outlook more tightly into ErlMail. To that end I may try to write a set of MAPI server/client modules into ErlMail. This will be a low priority task, unless I get a lot of interest in it. Plus I’m way behind on the IMAP server right now anyway and I have a few other project that I want to spend my programming time on before this.

I always get great feedback via email when I put up posts like this and it helps me keep focused on what others think are good technologies to have written in Erlang. So if you have any thoughts about this let me know.

I’ve been enjoying using Google Code and subversion in general for ErlMail. So much so that I decided that I wanted to put up all of my current code so that I can access it all the same way. This was mostly motivated by how easy it was to take my laptop to a WiFi hot-spot and continue working when I needed to be away from my apartment this morning. Now I’ll be able to work on any of my current code so long as I have Internet access.

 For those who are interested:

ErlDir: http://erldir.googlecode.com (Mostly just DNS encoding/decoding modules)
ErlMail: http://erlmail.googlecode.com (This is the most active project at the moment)
ErlVoIP: http://erlvoip.googlecode.com (This project has not been started yet)
ErlWeb: http://erlweb.googlecode.com (Not completely sure of the status if this code, ERML works great)

This morning I started testing the IMAP server in ERlMail-0.0.6 with Outlook Express. This is the first IMAP client I intend to test and it was the easiest to setup without messing up my own email

I’m not sure if my interpretations of the RFCs are way off or if Outlook Express is not following the RFCs. I hadn’t seen much references to double quotes (”) around strings, but Outlook Express seems to like putting double quotes around all strings. In the RFCs I only found a mention of them in LIST and LSUB commands. In any case, I’ve spent more time then I care to admit to figuring out how to implement code that handles the double quotes without side effects. So far so good.

The really great news is that I can LIST and LSUB folders, SUBSCRIBE and UNSUBSCRIBE folders and CREATE and DELETE folders as well. (At least I can now that I’m handling double quotes) I’m hoping that several other commands are working already as well too, but I haven’t gotten around to testing them yet; or rather I need to get the FETCH command working before Outlook Express will do much more.

 For anyone who is interested in watching the development more closely or testing with a different IMAP client you can connect with the following settings:

Server: imap.erlmail.net
User Name: simpleenigma@erlsoft.net
Password: erlmail 

This account is for testing purposes only (read: lots of spam) and the server is only up and running while I am in active development. Which usually means the sun is up in California

I’m looking into working with a few developers on several of the projects in the Erlang Internet Framework. To that end I’m starting to work with subversion and I have uploaded the latest code for ErlMail-0.0.6 up to Google Code.

http://erlmail.googlecode.com

I’m still working with figuring out exactly how subversion works and how to incorporate it into my own daily routine, but this will definitely open up some doors for collaboration with others and that is more important then anything else.

I’m hoping to remember to commit my changes at least once a day and I intend to make sure that the trunk at least compiles every time I work with it.

I also intend to put ErlDir up when I re-factor it at some point in November and ErlWeb, which is mostly ERML at this point, will go up at some point as well. If you want to contribute code to any of these project email me and I’ll upload them sooner.

I’ve been working on the IMAP server in ErlMail-0.0.6 as my primary project since I release ErlMail-0.0.5 last week. I think I’m about a third through the server and for what I’ve been doing I think it’s going pretty fast.

 The main server is based off of my SMTP server for the TCP communications and the basic file structure. Although in the past few days I’ve been implementing some new strategies that I will need to put back into the SMTP server as well. Most of that code is around the extensions to the IMAP (or SMTP) protocols. I created a modules called IMAPD_EXT that is run when the IMAP server does not recognize a command. This modules looks at the config file to see the list of known extensions and then verifies if they extension is loaded in the path on the current IMAP server. The extensions are listed in the config file as “Module1:Funcation1 Module2:function2 Function3″ When the module name is left off the default of imapd_ext is used for the module name. With this setup the IMAP server will be able to be extended without recompiling the ErlMail code. By adding an extension in the config file and making sure it is located in the search path that the Erlang run-time uses anyone will be able to extend the IMAP (or SMTP) server with minor effort. (Depending on the extension of course)

I’ve also been working a lot with the modular message store. This has already changed significantly since ErlMail-0.0.5, which was the first version to have the message store in it. I’m finding that the IMAP server needs to have a decent amount of it’s logic in the message store, since the way the files are being accessed in each message store is so different. Most notably the Mnesia message store and the Maildir++ message store are light years away from each other in the logic they use to return the same results from each function in the module.

In effect this means I am writing about 25% of the IMAP server in each message store and I’m planning on having two to four message stores in the next release. This was more effort then I was expecting, but the results are more then worth it.

I’ve gotten another report of a bug in the IMAP client. Actually it the same bug, but it’s manifesting itself slightly differently now. I’m thinking this particular bug my require a rewrite of the IMAP grammar files for both LEEX and YECC. I may have to put this off till the next version of ErlMail, as this will be quite time consuming.

I’m hoping to get the next version of ErlMail out in the first half of November now. Hopefully I’ll get plenty of time to spend on it and it will be out sooner then that.