Erlang Software Framework

The idea of the Erlang Internet Framework, or EIF, was created at the beginning of 2007 therefore it is fitting to do a status update and ponder on the future of the project one year after it initial conception.

The EIF was originally created to be a way to combine all of the projects that I had been working on into one focused project, since then it has grown and several projects have been added. The original projects beyond the EIF were ErlMail, ErlWeb and ErlDir. Each of these projects are still in early development stages and they have been joined by ErlVoIP, a Voice over IP server, and ErlMedia, a Flash Media Server written in Erlang.

ErlMedia/ErlyVideo

The code base for my flash media server, which I was calling ErlMedia has been joined with ErlyVideo, which is another Flash Media Server written in Erlang. Since my initial work on the ErlMedia/ErlyVideo in August and September I haven’t been able to spend much more time on the project and the other team members have done a wonderful job of mending the problems that were plaguing the code I was working with and taking it much further then I could have done alone.

I am very hopefully that I will be able to spend a decent amount of time working on this project in early 2008, after the next release of ErlMail.

ErlMail

ErlMail has been the focus of my attention for most of 2007. The distinct lack of a complete email package, including library modules for both client and server, has been my inspiration for this projects since it’s inception. In my personal opinion, ErlMail has the most advanced email modules of anything currently available as open source in Erlang.

For the past several months I have been working diligently on the IMAP server in ErlMail. I knew this server was going to be difficult to create, but I had severely underestimated the amount of time it was going to take to make an IMAP server operational. I am down to the last command, the FETCH command, in the IMAP server before I release a stable version of it. It is not going to be RFC compliant in the next version and there are still a few commands that I will need to include after the next version as well, most notably the search command.

The feature that I think is the most notable in the ErlMail package is the modular message store. The modular message store allows users to create their own message store modules and simply change the user configuration file for ErlMail to use them. This feature already garners more interest then any other single feature in the ErlMail package and with my future directions, which you will read later, it will become even more important.

ErlWeb

ErlWeb, which is a web server and associated modules, got one official release in 2007. The most notable piece of this release is the tag based scripting language called Erlang Markup Language, or ERML. My intention with ERML is to create a way that people who do not know how to program in Erlang can still get the benifits of Erlang. The most recent shift in the design of ERML has been to add a way to extend ERML with user create modules that are not part of the ErlWeb package. This is being done now with the other packages in the EIF.

For instance, a module in ErlMail called erml_erlmail will have all ERML commands that are associated with email commands for use in the ErlWeb application. This allows for users to create their own ERML commands, written in Erlang, and include them into ErlWeb without recompiling the system. In the case of ErlMail, this also allows all of the email logic to be contained in one application while still interacting with the ErlWeb application. This feature appeared in the code base in December of 2007 and I consider it as important to ErlWeb as the modular message store is to ErlMail.

Similar files have been created for other EIF projects; such as erml_erldir and erml_erlvoip which are located in their respective applications.

Other Projects

The rest of the projects included in the EIF have not moved forward much in 2007. ErlDir did get an official release which is mostly library modules that handle packet encoding/decoding and compression/decompression and ErlVoIP has been announced although work on this project has not begun yet.

Google Code

During the second half of 2007 I moved all open source projects that I have onto Google Code. While this was a move on my part to allow me to be more portable with my development environment and create a pseudo backup system to ease my own mind this also allows for other people to join the projects and contribute their own code if they wish. I have one person who is helping me document (and correct my spelling errors) most of the projects already and I am open to the idea of other people joining the work to see these projects move forward faster then I could accomplish alone.

Future of EIF

I see a bright future for the EIF in 2008; starting with a new release of ErlMail in the first few weeks of the year. After that there are a few changes to the core idea of the system that I intend to implement. These changes reflect new innovations that were not around during the EIF initial conception and large needs that have arisen since then as well.

The EIF Director

I’m not sure I have ever described the main project for the EIF itself. The EIF project is intended to be a manager application that keeps track of the versions of the other applications and which nodes have what servers running. So while each application in the EIF will be independent, the EIF application will be the glue that holds them all together.

Some of the features that are planned for the future of the EIF application itself are:

• Upgrading itself and other EIF applications
• Monitoring which nodes are running what servers
• Load balancing traffic across all nodes
• Fail-over on node crashes; allowing a user connection to move from one node to another without the user noticing

The EIF application would handle much of the communications and some of the state data for each connection, and then the server processes on each node would do the work that each connection is requesting. The design goal of this is to allow nodes to be added or removed into the system without users noticing any interruptions. This would include the ability to move a long session that a user has, on say an IMAP server, from one node to another without the user needing to reconnect. I am also hoping to implement this into ErlWeb so that comet connections could be moved from one node to another without the user noticing that a node has been removed from the cluster.

Amazon EC2/S3/SimpleDB integration

The Amazon web services, or AWS, announcements, that have been on going for several months, have been unexpected and potentially a huge boost to the design on the EIF. I will be taking some time to look at these services and creating a road map to completely integrate these services into the EIF at every level that is reasonable.

The first use of the AWS will be in ErlMail. I intend to create a modular message store that incorporates the AWS as the backend, most likely using the SimpleDB for user and domain information and the S3 storage for the actual messages.

After the ErlMail message store is integrated with the AWS all other project will be designed to work with the EIF Director on EC2. With the flexibility of the EC2 cloud any number of Amazon Machine Images, or AMIs, could be made with different server configurations. Then as the EIF director noticed a burst in traffic or number of connections the EIF director could initiate a new instance of an EC2 AMI. When the load has lowered enough the EIF director could move sessions onto the oldest running instances (without users noticing) and disconnect from instances that are no longer needed.

The core configuration of the EIF on AWS could start out as one instance with the EIF director and all other servers on the same instance and as load increases the EIF director will expand or contract the needed nodes. As more nodes are added the EIF director will move session that are active on itself onto other nodes allowing the EIF director to focus it’s resources on controlling traffic flow and balancing connections. Eventually as load decreases the EIF director would take the sessions back onto itself to minimize the number of instances back down to one instance.

If implemented well this could create a system where no matter what type of services you want to provide you would be able to use the AWS to create the project and if the need arises the project could be moved to a dedicated hosting facility without the users noticing the move and zero downtime. Once you are on your own hardware when you manually add new nodes to your system the EIF director will immediately start making use of them.

I see this as a boon to potential startups as the will be able to grow and move resources around in real-time without needing to incur major expenses up front. Once they are making money the can build their own infrastructure at their own pace and potentially still use the AWS when the need more capacity to their own infrastructure.

Conclusion

While 2007 was a great start to the EIF and all of the applications, 2008 looks to be even better. I’m sure there will be more shifts in direction and changes made to the ideas behind the system as the start to get implemented, but the core goal of making the most flexible, stable and powerful Internet development environment will always be there.

This afternoon I had a great conversation with a few of my friends that are starting some projects and are interested in using Erlang at the core of them. I’m not going to get too much into what the projects are, but they are around working with Microsoft technologies.

That makes Erlang a wonderful language to use especially when looking at Active Directory, which is based on LDAP. Erlang has a built in ASN.1 library that the really hard work of working with LDAP out of the picture. I know there are a few other people out there working on similar projects.

This conversation also had me thinking about integrating Outlook more tightly into ErlMail. To that end I may try to write a set of MAPI server/client modules into ErlMail. This will be a low priority task, unless I get a lot of interest in it. Plus I’m way behind on the IMAP server right now anyway and I have a few other project that I want to spend my programming time on before this.

I always get great feedback via email when I put up posts like this and it helps me keep focused on what others think are good technologies to have written in Erlang. So if you have any thoughts about this let me know.

I’ve been enjoying using Google Code and subversion in general for ErlMail. So much so that I decided that I wanted to put up all of my current code so that I can access it all the same way. This was mostly motivated by how easy it was to take my laptop to a WiFi hot-spot and continue working when I needed to be away from my apartment this morning. Now I’ll be able to work on any of my current code so long as I have Internet access.

 For those who are interested:

ErlDir: http://erldir.googlecode.com (Mostly just DNS encoding/decoding modules)
ErlMail: http://erlmail.googlecode.com (This is the most active project at the moment)
ErlVoIP: http://erlvoip.googlecode.com (This project has not been started yet)
ErlWeb: http://erlweb.googlecode.com (Not completely sure of the status if this code, ERML works great)

I’ve been working with Erlang as my primary programming language for about three years now. (Don’t quote me on that, it might only be two years) In any case I learn new techniques almost every day and with everything that I learn the language itself seems to get simpler and the number of lines of code I need to accomplish something goes down.

I was really noticing this on my implementation of the Flash Media server, which uses an odd proprietary protocol that was create by Macromedia (now adobe). I was implementing a section of code that was parsing the header for the packet. My first rendition was nearly 100 lines of code. When I was showing this to a colleague of mine he expressed that the way I had coded it was exactly the way he would have as well, which I consider standard practices for most programmers.

 This same section of code is now down to 7 lines of code, it is easier to read and I suspect it is much faster if for no other reason that is has fewer conditional statements in my code. The simplicity of the newer version of this function came from Erlang’s pattern matching in general and the amazing way that Erlang performs binary pattern matching. I was able to write patterns instead of conditional statements and set variables within the patterns. I understand that python has some features like this as well, but I have never used python or any other language that create code that is so succinct.

After realizing what had happened with this one function I started looking at some of my other code from the past few months to try and find similar patterns that I might be able to reduce. I found plenty of the in the SMTP and IMAP clients from ErlMail and I plan on releasing the new version of ErlMail-0.0.5 Friday or Saturday of this week. (I want to add some more documentation into the code so I remember what was going on and other people can use it as well, other then that it is done) 

After I write the first version of the IMAP server I am working on next I plan on revisiting the DNS server code I had been working on. Since the DNS protocol is a binary protocol the pattern matching that I learned while working on the Flash Media Server will almost directly translate into the DNS server. I looked at the code in one of the modules before writing this post and I saw three different functions that I could reduce with just 1 or 2 minutes of scanning the code. I’m planning on paying more attention to how many lines of code I started with and how many I am left with after re-factoring and I’ll publish that some time in the future. I hope to be able to get to that at some point in November or December, unless I get stuck on the IMAP server and need a break from it

I just out out the code for ErlDir-0.0.1, which basically can only compress and decompress DNS packets. I’m releasing this code under the ‘release early and release often’ theory of software development, even though I haven’t worked on this code in a few months.

This release is the framework for future releases and really don’t have much more functionality then encoding and decoding packets. When I say it like that it makes it sound like it was easy to create this much functionality, it wasn’t and I’m pretty proud of this in it’s own right.

In any case, I hope this helps someone somewhere and shows that I’m actually getting some code out again

EIF, ErlDir, DNS

Technorati Tags: EIF, ErlDir, DNS

ErlDir will be the package for holding the DNS and LDAP protocols and a few miscellenaous pieces that don;t quite fit in other places.

I already have the DNS packet encoding, decoding, compression and decompression routines written for the DNS server. I’ve even done some very simple communications work with it, although since DNS is based on UDP instead of TCP I’m still in the process of learning what the correct communications patterns are.

I’m planning on the DNS server being an authoritative and a caching DNS server along with the capability of doing DNS based black lists to enhance ErlMail.

The LDAP server will be using the built in ASN.1 modules (Thank you Erlang) for most of the hard work. The rest is mostly writing the data storage and communications. I intend that the LDAP server will become the central repository for many different type of information through all the applications that use the Erlang Internet Framework.

I personally plan on using LDAP for the user and configuration information for ErlMail and ErlWeb which will simplify the process of creating, removing and changing users that will use a web based email client.

Erlang, ErlDir, DNS, LDAP

Technorati Tags: Erlang, ErlDir, DNS, LDAP