Erlang Software Framework

I’ve been looking at how I want to implement some of my own ideas into web services in the near future and one thing that I keep coming back to is user authentication. Currently my preferred way of handing this will be using OpenId, but I have not found any Erlang projects that are using this yet. So I’m starting one at:

http://erlopenid.googlecode.com

I haven’t even started working on the code for this yet, so if anyone is interested in this project and wants to get in at the beginning email me and I’ll add you as a project member.

I’m going to be spending some time over the next few days looking over the spec and seeing how I think this should be created.

Update: I started a Google Group for the project at http://groups.google.com/group/erlopenid Please join us in talking about all things Erlang related to OpenId.

One of the biggest downsides to using INETS, the built in web server in the Erlang runtime, is that it does not do multi homing straight out of the box. Because of this single migging feature I had though I would either rewrite the web server, or I’d end up using mochiweb. My goal with the Erlang Internet Framework is to have as few dependencies on other people code as I can, so finding a way to achieve multi homing without using (the really great) mochiweb and without rewriting the whole server was a big concern for me.

After playing around with different ideas for the better part of a day I came across a suitable work around, at least for me. I created a module called mod_multi and configured a server on port 80 (and 443) that only uses this one module. mod_multi basically will take any commands that it gets, figure out the host name, look to see if any other INETS servers on the local machine are running an instance with the server_name set to the Host Name and then it request the same URI on the loopback interface and on the port that is being used by the server that it has found.

There are a few assumptions that I make in order for this to work. The port for the internal web servers is always set to 0, this gives a random port number, and the bind_address is always set to 127.0.0.1, which is the loopback interface. Since the internal communications is being handled on the loopback interface I’m not as concerned with checking if it is going to use SSL, so all internal web servers have no need for SSL at this time. If I chage this in the future I’ll be sure to add support for internal SSL.

This is more of a proof of concept at this point, basically proving that mod_multi can perform the tasks needed to create multi homing on INETS. Although the downside to it is that it is a classic man in the middle attack template. I suppose to some extent any proxy server has all the code built in to create a man in the middle attack as well, but I just never realized exactly how easy it would to create one.

Ultimately this is destined to be part of my effort to turn EIF into a cloud based (read: AWS based) infrastructure application. now that I have gotten past this point, I’m going to look into a mod_s3 module that will allow a web server to grab it’s templates from S3 if it does not already have them and setup a way to update the template if they have changed. That way you could upload all of your HTML or ERML files to S3 and each time a page is accesses from a web server (within a reasonable amount of time) on EC2 it would check for updates and stay current.

With the thought of this being on S3, I may implement a way to allow other servers to be involved. That way one external IP address could accept all web connections and redirect them to other EC2 instances for processing. In fact I had been thinking about doing this on several protocols, including SMTP and IMAP; which would give one IP address as the face of all connections and you could have an army of servers behind it doing all of the tasks.

I got an email this morning that made me realize how long it had been since I had updated this site. I’ve been a little bit distracted since I got my book deal. Originally I submitted a proposal to write a book on Erlang, but through a series of events I’m actually writing a book on on of my other passions; yoga. You can find more details and eventual updates about the book at http://www.yogaforhackers.com

So now that you all know why I’ve been off the grid for a while, I have a new project to announce as well. I’ve started to create my modules for the Amazon Web Services, which I am calling EIFAWS and it is hosted on Google code at http://eifaws.googlecode.com . I know that there is already one project doing AWS modules in Erlang, but  I wanted some specific things in the code to make sure that it was compatible with the EIF, so I decided to create my own.

The EIFAWS project has at least one feature that I haven’t seen yet in the other AWS libraries, which is version support. I’m intended to create specific modules for each version which will allows users to continue to use an older version of an AWS API until they are ready to switch.

I am currently working on the EC2 modules, and to go along with them I am going to launch a public AMI later today. (Right after I finish the EC2 code to make it public.) I have two AMIs one 32-bit and the other 64-bit based on Fedora Core 9. I’ve gone through and disabled just about all of the processes and removed some packages that I know useless on EC2 or I will not want to use, like sendmail and gpm. I also installed the latest Erlang RPM from the FC9 repo.

The 32-bit AMI will be ami-01b55168 and the 64-bit AMI is ami-30db5959. Also my amazon account number is 920563975665 in case you will want to search for other AMIs in the future. All of my AMIs will end up in a /erlsoft_ami/ folder on S3 as well.

These AMIs are meant to playing around with EC2 and Erlang and will be improved upon as I learn more of what I need to do and how I want to do things. They may be removed at some point int he future, when I have new and better AMIs for everyone to work with

And one last note on the EIFAWS project. One of my test projects that I intend to write will be a FTP server front end for S3 that runs on EC2. The hope would be that with this as a front end, anyone with an FTP client would be able to access S3 and since it would be running on EC2 the FTP/S3 bandwidth charges would not apply, only bandwidth charges to and from EC2. Basically I didn’t liek any of the S3 explorer type packages that I saw and thought an FTP front end would open up S3 to many more uses. So well see where that ends up.

In my last post I mentioned that I was working on a new way to extend the ERML command in ErlWeb so that they can be extended with a behavior in a different application. I’ve had the idea stuck in my head since then and I needed to get something operational so that I could focus on something else for a while.

Last week I rewrote the command processing so that is looks to the configuration file to see which ERML command modules are potentially available. After that it checks to see if the module is on the server, if it is a gen_erml behavior and finds out what functions meet the requirements to make then ERML commands; for now that means they have to start with erml and have 2 arguments.

I decided to start working with the existing commands and I got them up and running in short order. They are in the erml_core and erml_tags files. Eventually that will all be in the erml_core file and the erml_tag file will go away.

Today I decided that I needed to get the ERML commands working from a different module. I had already setup ERML command modules for all of the EIF projects and since I have the most work done on ErlMail it made the most sense to start there.

Sending email from a web page is a pretty common activity and with the interaction of the EIF applications this was always destine to be a feature to ERML. This is now a reality.

In the ErlMail application the ERML command file is erml_erlmail. There is a function called ermlsmtp that will now take information about the IP address to comment, who the message is from, who it is to, the subject of the message and then it will create a MIME message and send the message using the smtpc:sendmail command.

As of about 10 minutes ago I successfully sent a message from a web page that was processed through the ERML engine, it sent an email message to my test account on an SMTP server running ErlMail. After that I was able to see the message in the folder using an IMAP connection to the same ErlMail server. Since the IMAP server is not complete I was not able to see the body of the message, but that is more about the IMAP server then ERML at the moment.

Through this process I started to find some utilities that are going to be needed to write successful ERML commands. I have started to put them into the gen_erml module. I’m going to continue this process of refining the way ERML commands are created as I continue to implement new ERML command through the EIF.

Then the big news of this post is that the new ERML engine works and it is expandable without recompiling ErlWeb. It does still require more configuration then I am happy with, but that will get worked out over time.

The idea of the Erlang Internet Framework, or EIF, was created at the beginning of 2007 therefore it is fitting to do a status update and ponder on the future of the project one year after it initial conception.

The EIF was originally created to be a way to combine all of the projects that I had been working on into one focused project, since then it has grown and several projects have been added. The original projects beyond the EIF were ErlMail, ErlWeb and ErlDir. Each of these projects are still in early development stages and they have been joined by ErlVoIP, a Voice over IP server, and ErlMedia, a Flash Media Server written in Erlang.

ErlMedia/ErlyVideo

The code base for my flash media server, which I was calling ErlMedia has been joined with ErlyVideo, which is another Flash Media Server written in Erlang. Since my initial work on the ErlMedia/ErlyVideo in August and September I haven’t been able to spend much more time on the project and the other team members have done a wonderful job of mending the problems that were plaguing the code I was working with and taking it much further then I could have done alone.

I am very hopefully that I will be able to spend a decent amount of time working on this project in early 2008, after the next release of ErlMail.

ErlMail

ErlMail has been the focus of my attention for most of 2007. The distinct lack of a complete email package, including library modules for both client and server, has been my inspiration for this projects since it’s inception. In my personal opinion, ErlMail has the most advanced email modules of anything currently available as open source in Erlang.

For the past several months I have been working diligently on the IMAP server in ErlMail. I knew this server was going to be difficult to create, but I had severely underestimated the amount of time it was going to take to make an IMAP server operational. I am down to the last command, the FETCH command, in the IMAP server before I release a stable version of it. It is not going to be RFC compliant in the next version and there are still a few commands that I will need to include after the next version as well, most notably the search command.

The feature that I think is the most notable in the ErlMail package is the modular message store. The modular message store allows users to create their own message store modules and simply change the user configuration file for ErlMail to use them. This feature already garners more interest then any other single feature in the ErlMail package and with my future directions, which you will read later, it will become even more important.

ErlWeb

ErlWeb, which is a web server and associated modules, got one official release in 2007. The most notable piece of this release is the tag based scripting language called Erlang Markup Language, or ERML. My intention with ERML is to create a way that people who do not know how to program in Erlang can still get the benifits of Erlang. The most recent shift in the design of ERML has been to add a way to extend ERML with user create modules that are not part of the ErlWeb package. This is being done now with the other packages in the EIF.

For instance, a module in ErlMail called erml_erlmail will have all ERML commands that are associated with email commands for use in the ErlWeb application. This allows for users to create their own ERML commands, written in Erlang, and include them into ErlWeb without recompiling the system. In the case of ErlMail, this also allows all of the email logic to be contained in one application while still interacting with the ErlWeb application. This feature appeared in the code base in December of 2007 and I consider it as important to ErlWeb as the modular message store is to ErlMail.

Similar files have been created for other EIF projects; such as erml_erldir and erml_erlvoip which are located in their respective applications.

Other Projects

The rest of the projects included in the EIF have not moved forward much in 2007. ErlDir did get an official release which is mostly library modules that handle packet encoding/decoding and compression/decompression and ErlVoIP has been announced although work on this project has not begun yet.

Google Code

During the second half of 2007 I moved all open source projects that I have onto Google Code. While this was a move on my part to allow me to be more portable with my development environment and create a pseudo backup system to ease my own mind this also allows for other people to join the projects and contribute their own code if they wish. I have one person who is helping me document (and correct my spelling errors) most of the projects already and I am open to the idea of other people joining the work to see these projects move forward faster then I could accomplish alone.

Future of EIF

I see a bright future for the EIF in 2008; starting with a new release of ErlMail in the first few weeks of the year. After that there are a few changes to the core idea of the system that I intend to implement. These changes reflect new innovations that were not around during the EIF initial conception and large needs that have arisen since then as well.

The EIF Director

I’m not sure I have ever described the main project for the EIF itself. The EIF project is intended to be a manager application that keeps track of the versions of the other applications and which nodes have what servers running. So while each application in the EIF will be independent, the EIF application will be the glue that holds them all together.

Some of the features that are planned for the future of the EIF application itself are:

• Upgrading itself and other EIF applications
• Monitoring which nodes are running what servers
• Load balancing traffic across all nodes
• Fail-over on node crashes; allowing a user connection to move from one node to another without the user noticing

The EIF application would handle much of the communications and some of the state data for each connection, and then the server processes on each node would do the work that each connection is requesting. The design goal of this is to allow nodes to be added or removed into the system without users noticing any interruptions. This would include the ability to move a long session that a user has, on say an IMAP server, from one node to another without the user needing to reconnect. I am also hoping to implement this into ErlWeb so that comet connections could be moved from one node to another without the user noticing that a node has been removed from the cluster.

Amazon EC2/S3/SimpleDB integration

The Amazon web services, or AWS, announcements, that have been on going for several months, have been unexpected and potentially a huge boost to the design on the EIF. I will be taking some time to look at these services and creating a road map to completely integrate these services into the EIF at every level that is reasonable.

The first use of the AWS will be in ErlMail. I intend to create a modular message store that incorporates the AWS as the backend, most likely using the SimpleDB for user and domain information and the S3 storage for the actual messages.

After the ErlMail message store is integrated with the AWS all other project will be designed to work with the EIF Director on EC2. With the flexibility of the EC2 cloud any number of Amazon Machine Images, or AMIs, could be made with different server configurations. Then as the EIF director noticed a burst in traffic or number of connections the EIF director could initiate a new instance of an EC2 AMI. When the load has lowered enough the EIF director could move sessions onto the oldest running instances (without users noticing) and disconnect from instances that are no longer needed.

The core configuration of the EIF on AWS could start out as one instance with the EIF director and all other servers on the same instance and as load increases the EIF director will expand or contract the needed nodes. As more nodes are added the EIF director will move session that are active on itself onto other nodes allowing the EIF director to focus it’s resources on controlling traffic flow and balancing connections. Eventually as load decreases the EIF director would take the sessions back onto itself to minimize the number of instances back down to one instance.

If implemented well this could create a system where no matter what type of services you want to provide you would be able to use the AWS to create the project and if the need arises the project could be moved to a dedicated hosting facility without the users noticing the move and zero downtime. Once you are on your own hardware when you manually add new nodes to your system the EIF director will immediately start making use of them.

I see this as a boon to potential startups as the will be able to grow and move resources around in real-time without needing to incur major expenses up front. Once they are making money the can build their own infrastructure at their own pace and potentially still use the AWS when the need more capacity to their own infrastructure.

Conclusion

While 2007 was a great start to the EIF and all of the applications, 2008 looks to be even better. I’m sure there will be more shifts in direction and changes made to the ideas behind the system as the start to get implemented, but the core goal of making the most flexible, stable and powerful Internet development environment will always be there.

I’ve been redesigning some of the core elements of the message store and the way the IMAP server handles responses for the last week. In that time I think I’ve come up with some solutions to problems that I thought were inconveinces, but not really problems at the time.

After looking at the way I was starting the gen_server for the message store I started to realize that my configuration file was completely wrong. I had created a text file that had one command per line and needed to be parsed and fixed to be used. Default values were difficult and if the user excluded a mandatory line the server would crash

I’ve now moved the default configuration into the .app file located in the ebin directory and the user can specify a .config file on the command line to override any settings they wish. I’ve also built in a function that lets me search for a key and define a default value if the key is not found.

This also lead me to look at how the processes were starting and stopping. Using appmon I found that the client connections were not being closed properly when the STMP QUIT the IMAP LOGOUT commands were issued. I’m not terminating the FSM properly when either of these commands are issued.

Originally I had the SMTP and IMAP servers started independently; now they are started by the erlmail app and there are configuration settings to determine if they are started by default or not.

I’m moving some of the logic from the modular message store into one place; erlmail_store.erl. This is a gen_server that will handle the message store functions and knows which module to call for specific functions. Each of the modular message store modules will have less logic in them; making them easier to create and the other ErlMail modules will call erlmail_store when they need to interact with the store. In the long run this means less redundant code and more control over how the modular message store components interact with the rest of the server.

The IMAP response server is in place and operating, but it now needs access to the message store and those features haven’t been developed yet. The two servers will go hand-in-hand and for a while I will be developing the two in tandem.

It feels like I haven’t added much functionality lately, but the system is much more stable and more robust from the improvements I have been doing.

I’ve been looking at the way the FSM is handling the responses for the IMAP server for quite some time. In many cases it works wonderfully, but there are some cases that it does not do what is needed for RFC compliance.

The primary case of this is when two or more clients are connected to the same mailbox and/or when new mail is delivered to a mailbox that is selected. The second of those two scenarios is more important, but the first is a main design feature that I have always intended for the server.

The solution seems to be to create a mailbox server that handles the responses and potentially handles all interaction with the mailbox as well. This will allow for the IMAP server to keep a queue of messages for individual clients so that they will be able to receive all of the message that they are interested in and potentially reduce the access to the message store as well.

The most notable feature that is required of an IMAP server that I cannot implement with the current design in the NOOP commands ability to send untagged messages with status updates. When a mailbox has an event that changes the status of a message or the mailbox I have no real way to collecting the responses and delivering them to all of the clients that may be interested.

While I have known about this problem for a while, it has now become a large enough issue that I need to reconsider this part of the design before moving forward. I’ve been making notes on the limitation of the IMAP server in it’s current configuration and I think I’m going to take a bit of time to redesign some areas of the server and implement the mailbox / response server to take care of many of these issues.

I most likely and going to redesign some of the record I’ve been using and find better ways to handle the UID and UIDVALIDITY features in the IMAP server while I am working on this problem. All of these things need to be addressed at some point and I think taking care of them now will help the server int he long run. (or this could all be a way that I can avoid working on the FETCH command)

I was working on the IMAP server this morning and I realized that I had an annoying pattern that I needed to generalize. Throughout the code I have been using the state of the FMS to figure out which is the correct store to get information from and then calling a specific command from that store. Depending on the actual process that is doing this it can take between 2 and 4 lines of code.

I started to develop an abstraction layer inside the gen_store, so that you can now call the command you are using from gen_store, giving it the data needed and the state data, and it will figure out the correct store to use to call the command from the state data.

The way the information is being processed does not change, but the way the user calls the data does. The user no longer need to figure out which is the correct store, it only calls the function from gen_store and lets gen_store figure it out. This simplifies the user process down to one line of code instead of as many as 4 and only minimally increases the size of the gen_store module. 

 I’ll be re-factoring this into the code as I go along, but I am looking forward to simplifying this task quite a bit.

This afternoon I had a great conversation with a few of my friends that are starting some projects and are interested in using Erlang at the core of them. I’m not going to get too much into what the projects are, but they are around working with Microsoft technologies.

That makes Erlang a wonderful language to use especially when looking at Active Directory, which is based on LDAP. Erlang has a built in ASN.1 library that the really hard work of working with LDAP out of the picture. I know there are a few other people out there working on similar projects.

This conversation also had me thinking about integrating Outlook more tightly into ErlMail. To that end I may try to write a set of MAPI server/client modules into ErlMail. This will be a low priority task, unless I get a lot of interest in it. Plus I’m way behind on the IMAP server right now anyway and I have a few other project that I want to spend my programming time on before this.

I always get great feedback via email when I put up posts like this and it helps me keep focused on what others think are good technologies to have written in Erlang. So if you have any thoughts about this let me know.

I’ve been enjoying using Google Code and subversion in general for ErlMail. So much so that I decided that I wanted to put up all of my current code so that I can access it all the same way. This was mostly motivated by how easy it was to take my laptop to a WiFi hot-spot and continue working when I needed to be away from my apartment this morning. Now I’ll be able to work on any of my current code so long as I have Internet access.

 For those who are interested:

ErlDir: http://erldir.googlecode.com (Mostly just DNS encoding/decoding modules)
ErlMail: http://erlmail.googlecode.com (This is the most active project at the moment)
ErlVoIP: http://erlvoip.googlecode.com (This project has not been started yet)
ErlWeb: http://erlweb.googlecode.com (Not completely sure of the status if this code, ERML works great)