The idea of the Erlang Internet Framework, or EIF, was created at the beginning of 2007 therefore it is fitting to do a status update and ponder on the future of the project one year after it initial conception.

The EIF was originally created to be a way to combine all of the projects that I had been working on into one focused project, since then it has grown and several projects have been added. The original projects beyond the EIF were ErlMail, ErlWeb and ErlDir. Each of these projects are still in early development stages and they have been joined by ErlVoIP, a Voice over IP server, and ErlMedia, a Flash Media Server written in Erlang.

ErlMedia/ErlyVideo

The code base for my flash media server, which I was calling ErlMedia has been joined with ErlyVideo, which is another Flash Media Server written in Erlang. Since my initial work on the ErlMedia/ErlyVideo in August and September I haven’t been able to spend much more time on the project and the other team members have done a wonderful job of mending the problems that were plaguing the code I was working with and taking it much further then I could have done alone.

I am very hopefully that I will be able to spend a decent amount of time working on this project in early 2008, after the next release of ErlMail.

ErlMail

ErlMail has been the focus of my attention for most of 2007. The distinct lack of a complete email package, including library modules for both client and server, has been my inspiration for this projects since it’s inception. In my personal opinion, ErlMail has the most advanced email modules of anything currently available as open source in Erlang.

For the past several months I have been working diligently on the IMAP server in ErlMail. I knew this server was going to be difficult to create, but I had severely underestimated the amount of time it was going to take to make an IMAP server operational. I am down to the last command, the FETCH command, in the IMAP server before I release a stable version of it. It is not going to be RFC compliant in the next version and there are still a few commands that I will need to include after the next version as well, most notably the search command.

The feature that I think is the most notable in the ErlMail package is the modular message store. The modular message store allows users to create their own message store modules and simply change the user configuration file for ErlMail to use them. This feature already garners more interest then any other single feature in the ErlMail package and with my future directions, which you will read later, it will become even more important.

ErlWeb

ErlWeb, which is a web server and associated modules, got one official release in 2007. The most notable piece of this release is the tag based scripting language called Erlang Markup Language, or ERML. My intention with ERML is to create a way that people who do not know how to program in Erlang can still get the benifits of Erlang. The most recent shift in the design of ERML has been to add a way to extend ERML with user create modules that are not part of the ErlWeb package. This is being done now with the other packages in the EIF.

For instance, a module in ErlMail called erml_erlmail will have all ERML commands that are associated with email commands for use in the ErlWeb application. This allows for users to create their own ERML commands, written in Erlang, and include them into ErlWeb without recompiling the system. In the case of ErlMail, this also allows all of the email logic to be contained in one application while still interacting with the ErlWeb application. This feature appeared in the code base in December of 2007 and I consider it as important to ErlWeb as the modular message store is to ErlMail.

Similar files have been created for other EIF projects; such as erml_erldir and erml_erlvoip which are located in their respective applications.

Other Projects

The rest of the projects included in the EIF have not moved forward much in 2007. ErlDir did get an official release which is mostly library modules that handle packet encoding/decoding and compression/decompression and ErlVoIP has been announced although work on this project has not begun yet.

Google Code

During the second half of 2007 I moved all open source projects that I have onto Google Code. While this was a move on my part to allow me to be more portable with my development environment and create a pseudo backup system to ease my own mind this also allows for other people to join the projects and contribute their own code if they wish. I have one person who is helping me document (and correct my spelling errors) most of the projects already and I am open to the idea of other people joining the work to see these projects move forward faster then I could accomplish alone.

Future of EIF

I see a bright future for the EIF in 2008; starting with a new release of ErlMail in the first few weeks of the year. After that there are a few changes to the core idea of the system that I intend to implement. These changes reflect new innovations that were not around during the EIF initial conception and large needs that have arisen since then as well.

The EIF Director

I’m not sure I have ever described the main project for the EIF itself. The EIF project is intended to be a manager application that keeps track of the versions of the other applications and which nodes have what servers running. So while each application in the EIF will be independent, the EIF application will be the glue that holds them all together.

Some of the features that are planned for the future of the EIF application itself are:

  • Upgrading itself and other EIF applications
  • Monitoring which nodes are running what servers
  • Load balancing traffic across all nodes
  • Fail-over on node crashes; allowing a user connection to move from one node to another without the user noticing

The EIF application would handle much of the communications and some of the state data for each connection, and then the server processes on each node would do the work that each connection is requesting. The design goal of this is to allow nodes to be added or removed into the system without users noticing any interruptions. This would include the ability to move a long session that a user has, on say an IMAP server, from one node to another without the user needing to reconnect. I am also hoping to implement this into ErlWeb so that comet connections could be moved from one node to another without the user noticing that a node has been removed from the cluster.

Amazon EC2/S3/SimpleDB integration

The Amazon web services, or AWS, announcements, that have been on going for several months, have been unexpected and potentially a huge boost to the design on the EIF. I will be taking some time to look at these services and creating a road map to completely integrate these services into the EIF at every level that is reasonable.

The first use of the AWS will be in ErlMail. I intend to create a modular message store that incorporates the AWS as the backend, most likely using the SimpleDB for user and domain information and the S3 storage for the actual messages.

After the ErlMail message store is integrated with the AWS all other project will be designed to work with the EIF Director on EC2. With the flexibility of the EC2 cloud any number of Amazon Machine Images, or AMIs, could be made with different server configurations. Then as the EIF director noticed a burst in traffic or number of connections the EIF director could initiate a new instance of an EC2 AMI. When the load has lowered enough the EIF director could move sessions onto the oldest running instances (without users noticing) and disconnect from instances that are no longer needed.

The core configuration of the EIF on AWS could start out as one instance with the EIF director and all other servers on the same instance and as load increases the EIF director will expand or contract the needed nodes. As more nodes are added the EIF director will move session that are active on itself onto other nodes allowing the EIF director to focus it’s resources on controlling traffic flow and balancing connections. Eventually as load decreases the EIF director would take the sessions back onto itself to minimize the number of instances back down to one instance.

If implemented well this could create a system where no matter what type of services you want to provide you would be able to use the AWS to create the project and if the need arises the project could be moved to a dedicated hosting facility without the users noticing the move and zero downtime. Once you are on your own hardware when you manually add new nodes to your system the EIF director will immediately start making use of them.

I see this as a boon to potential startups as the will be able to grow and move resources around in real-time without needing to incur major expenses up front. Once they are making money the can build their own infrastructure at their own pace and potentially still use the AWS when the need more capacity to their own infrastructure.

Conclusion

While 2007 was a great start to the EIF and all of the applications, 2008 looks to be even better. I’m sure there will be more shifts in direction and changes made to the ideas behind the system as the start to get implemented, but the core goal of making the most flexible, stable and powerful Internet development environment will always be there.