Erlang Software Framework

In my last post I mentioned that I was working on a new way to extend the ERML command in ErlWeb so that they can be extended with a behavior in a different application. I’ve had the idea stuck in my head since then and I needed to get something operational so that I could focus on something else for a while.

Last week I rewrote the command processing so that is looks to the configuration file to see which ERML command modules are potentially available. After that it checks to see if the module is on the server, if it is a gen_erml behavior and finds out what functions meet the requirements to make then ERML commands; for now that means they have to start with erml and have 2 arguments.

I decided to start working with the existing commands and I got them up and running in short order. They are in the erml_core and erml_tags files. Eventually that will all be in the erml_core file and the erml_tag file will go away.

Today I decided that I needed to get the ERML commands working from a different module. I had already setup ERML command modules for all of the EIF projects and since I have the most work done on ErlMail it made the most sense to start there.

Sending email from a web page is a pretty common activity and with the interaction of the EIF applications this was always destine to be a feature to ERML. This is now a reality.

In the ErlMail application the ERML command file is erml_erlmail. There is a function called ermlsmtp that will now take information about the IP address to comment, who the message is from, who it is to, the subject of the message and then it will create a MIME message and send the message using the smtpc:sendmail command.

As of about 10 minutes ago I successfully sent a message from a web page that was processed through the ERML engine, it sent an email message to my test account on an SMTP server running ErlMail. After that I was able to see the message in the folder using an IMAP connection to the same ErlMail server. Since the IMAP server is not complete I was not able to see the body of the message, but that is more about the IMAP server then ERML at the moment.

Through this process I started to find some utilities that are going to be needed to write successful ERML commands. I have started to put them into the gen_erml module. I’m going to continue this process of refining the way ERML commands are created as I continue to implement new ERML command through the EIF.

Then the big news of this post is that the new ERML engine works and it is expandable without recompiling ErlWeb. It does still require more configuration then I am happy with, but that will get worked out over time.

The idea of the Erlang Internet Framework, or EIF, was created at the beginning of 2007 therefore it is fitting to do a status update and ponder on the future of the project one year after it initial conception.

The EIF was originally created to be a way to combine all of the projects that I had been working on into one focused project, since then it has grown and several projects have been added. The original projects beyond the EIF were ErlMail, ErlWeb and ErlDir. Each of these projects are still in early development stages and they have been joined by ErlVoIP, a Voice over IP server, and ErlMedia, a Flash Media Server written in Erlang.

ErlMedia/ErlyVideo

The code base for my flash media server, which I was calling ErlMedia has been joined with ErlyVideo, which is another Flash Media Server written in Erlang. Since my initial work on the ErlMedia/ErlyVideo in August and September I haven’t been able to spend much more time on the project and the other team members have done a wonderful job of mending the problems that were plaguing the code I was working with and taking it much further then I could have done alone.

I am very hopefully that I will be able to spend a decent amount of time working on this project in early 2008, after the next release of ErlMail.

ErlMail

ErlMail has been the focus of my attention for most of 2007. The distinct lack of a complete email package, including library modules for both client and server, has been my inspiration for this projects since it’s inception. In my personal opinion, ErlMail has the most advanced email modules of anything currently available as open source in Erlang.

For the past several months I have been working diligently on the IMAP server in ErlMail. I knew this server was going to be difficult to create, but I had severely underestimated the amount of time it was going to take to make an IMAP server operational. I am down to the last command, the FETCH command, in the IMAP server before I release a stable version of it. It is not going to be RFC compliant in the next version and there are still a few commands that I will need to include after the next version as well, most notably the search command.

The feature that I think is the most notable in the ErlMail package is the modular message store. The modular message store allows users to create their own message store modules and simply change the user configuration file for ErlMail to use them. This feature already garners more interest then any other single feature in the ErlMail package and with my future directions, which you will read later, it will become even more important.

ErlWeb

ErlWeb, which is a web server and associated modules, got one official release in 2007. The most notable piece of this release is the tag based scripting language called Erlang Markup Language, or ERML. My intention with ERML is to create a way that people who do not know how to program in Erlang can still get the benifits of Erlang. The most recent shift in the design of ERML has been to add a way to extend ERML with user create modules that are not part of the ErlWeb package. This is being done now with the other packages in the EIF.

For instance, a module in ErlMail called erml_erlmail will have all ERML commands that are associated with email commands for use in the ErlWeb application. This allows for users to create their own ERML commands, written in Erlang, and include them into ErlWeb without recompiling the system. In the case of ErlMail, this also allows all of the email logic to be contained in one application while still interacting with the ErlWeb application. This feature appeared in the code base in December of 2007 and I consider it as important to ErlWeb as the modular message store is to ErlMail.

Similar files have been created for other EIF projects; such as erml_erldir and erml_erlvoip which are located in their respective applications.

Other Projects

The rest of the projects included in the EIF have not moved forward much in 2007. ErlDir did get an official release which is mostly library modules that handle packet encoding/decoding and compression/decompression and ErlVoIP has been announced although work on this project has not begun yet.

Google Code

During the second half of 2007 I moved all open source projects that I have onto Google Code. While this was a move on my part to allow me to be more portable with my development environment and create a pseudo backup system to ease my own mind this also allows for other people to join the projects and contribute their own code if they wish. I have one person who is helping me document (and correct my spelling errors) most of the projects already and I am open to the idea of other people joining the work to see these projects move forward faster then I could accomplish alone.

Future of EIF

I see a bright future for the EIF in 2008; starting with a new release of ErlMail in the first few weeks of the year. After that there are a few changes to the core idea of the system that I intend to implement. These changes reflect new innovations that were not around during the EIF initial conception and large needs that have arisen since then as well.

The EIF Director

I’m not sure I have ever described the main project for the EIF itself. The EIF project is intended to be a manager application that keeps track of the versions of the other applications and which nodes have what servers running. So while each application in the EIF will be independent, the EIF application will be the glue that holds them all together.

Some of the features that are planned for the future of the EIF application itself are:

• Upgrading itself and other EIF applications
• Monitoring which nodes are running what servers
• Load balancing traffic across all nodes
• Fail-over on node crashes; allowing a user connection to move from one node to another without the user noticing

The EIF application would handle much of the communications and some of the state data for each connection, and then the server processes on each node would do the work that each connection is requesting. The design goal of this is to allow nodes to be added or removed into the system without users noticing any interruptions. This would include the ability to move a long session that a user has, on say an IMAP server, from one node to another without the user needing to reconnect. I am also hoping to implement this into ErlWeb so that comet connections could be moved from one node to another without the user noticing that a node has been removed from the cluster.

Amazon EC2/S3/SimpleDB integration

The Amazon web services, or AWS, announcements, that have been on going for several months, have been unexpected and potentially a huge boost to the design on the EIF. I will be taking some time to look at these services and creating a road map to completely integrate these services into the EIF at every level that is reasonable.

The first use of the AWS will be in ErlMail. I intend to create a modular message store that incorporates the AWS as the backend, most likely using the SimpleDB for user and domain information and the S3 storage for the actual messages.

After the ErlMail message store is integrated with the AWS all other project will be designed to work with the EIF Director on EC2. With the flexibility of the EC2 cloud any number of Amazon Machine Images, or AMIs, could be made with different server configurations. Then as the EIF director noticed a burst in traffic or number of connections the EIF director could initiate a new instance of an EC2 AMI. When the load has lowered enough the EIF director could move sessions onto the oldest running instances (without users noticing) and disconnect from instances that are no longer needed.

The core configuration of the EIF on AWS could start out as one instance with the EIF director and all other servers on the same instance and as load increases the EIF director will expand or contract the needed nodes. As more nodes are added the EIF director will move session that are active on itself onto other nodes allowing the EIF director to focus it’s resources on controlling traffic flow and balancing connections. Eventually as load decreases the EIF director would take the sessions back onto itself to minimize the number of instances back down to one instance.

If implemented well this could create a system where no matter what type of services you want to provide you would be able to use the AWS to create the project and if the need arises the project could be moved to a dedicated hosting facility without the users noticing the move and zero downtime. Once you are on your own hardware when you manually add new nodes to your system the EIF director will immediately start making use of them.

I see this as a boon to potential startups as the will be able to grow and move resources around in real-time without needing to incur major expenses up front. Once they are making money the can build their own infrastructure at their own pace and potentially still use the AWS when the need more capacity to their own infrastructure.

Conclusion

While 2007 was a great start to the EIF and all of the applications, 2008 looks to be even better. I’m sure there will be more shifts in direction and changes made to the ideas behind the system as the start to get implemented, but the core goal of making the most flexible, stable and powerful Internet development environment will always be there.

I just uploaded the initial release of ErlWeb. This release holds the base version of a web scripting language that I am calling Erlang Markup Language. This is a tag based language that should be very easy to use, so the learning curve of the average web developer should be very short. In a future release I’ll be introducing the ability to use any Erlang module on the web server through ERML, which will give the language all of the power of Erlang

Currently ERML works by including the mod_erml file in your INET config file. After that any file that ends in a .erml file extension will be processed for any ERML tags. Once I’m done with this post I’m going to start working on some very simple documentation to let other people get setup and start using the language. I hope to have a rough draft of that done over the weekend.

I’m pretty excited about ERML and this project in general. I’m planning on using it in my own future developments, so I’ll need to keep developing ERML to make sure that I have all of the feature I need. The code for ERML is not the prettiest code I’ve ever written and it doesn’t have as much documentation as I’d like either, but I needed to get it in a working condition to give to someone else tomorrow, so the cleanup will happen before I release 0.0.2.

Note: I’ve only tested this on a Linux server, it might have file operation bugs on windows.

Update: I put together a very simple document on how to use ERML commands, you’ll notice that documentation is not my strong suit.

ERML, Erlang Markup Language, ErlWeb, EIF

Technorati Tags: ERML, Erlang Markup Language, ErlWeb, EIF

I’ve been working on the Erlang Markup Language for most of the coding time I’ve had this weekend and I’m making some great progress. I started off trying to figure out how I was going to traverse the XML tree I had created using xmerl, after reading the documentation I was more confused about what I wanted to do then I was when I started.

I took to reading through the xmerl code and I found the xmerl_lib modules, which has a mapxml/2, foldxml/2 and mapfoldxml/3 functions that after reading them thoroughly them they turned out to be the greater part of the work I was planning on doing today This bring me back to the biggest problem I have with Erlang; the documentation. In my opinion, the three greatest functions in the xmerl library were the ones I found today and they are not mentioned in the documentation anywhere … But i digress …

After figuring out that I was going to need to use mapfoldxml/3 so be able to modify the structure of the document and accumulate changes in the state at the same time I was off to the races.

The first challenge was to create a ermlset command to set variables. I wanted this command to set the variables and to let the variables change at any point later on the page. (I’ve been enjoying the irony of writing a scripting language that lets you change variables values in a language that does not) Then after the variable is set I needed to remove the command from the output of the apple. I ended up replace the command with a generic ermlcommand that I remove all of the references to ermlcommand after processing.

Once I had my ermlset command working I moved along to the ermlloop command. I’m planning on having several loops, including a index or counter loop, a list loop and a conditional loop. I started with the index loop, just displaying some text a number of times. I got that working with a few glitches with out too much trouble and then I implemented some recursion into it, which managed to solve some of my glitches. (I’ve never figured out why, but for some reason my code runs better after I implement recursion. It’s happened about 4 times now, anyway …)

After I built int he recursion I was able to perform ermlset command within the loop and I have nested ermlloop statements … not bad for a days work.

Next, is working on the if/then/else statements, hopefully that will work just as easily.

ERML,Erlang,ErlWeb,Markup Language

Technorati Tags: ERML, Erlang, ErlWeb, Markup Language

With the projects that I am working on it has become more pressing that I get the Erlang Markup Language up and running in some form or fashion sooner rather then later. As it turns out I’m the current bottleneck when it comes to development and so I needed to get this part working.

I looked into implementing ERML into INETS and YAWS and choose to implement into INETS initially. Took me about a days worth of work to get the mod_erml module working with INETS, but most of that was figuring out how INETS needed the module to work.

Now that I have the mod_erml working I’m working on the actual markup language itself. I have a parser that is based in parsetools that I created a few months ago to break the original .erml files into tokens, so now I just need to expand the parser a bit and process the tokens.

Sounds simple enough, right?

The parts that are needed the most and I haven’t figured how I want to implement are the decision tree functions and the loops. Most of the rest of ERML is pretty simple after I get those down.

If things go well, I’m hoping to release a broken buggy version of ERML in the first half of August, is anyone is interested please leave comments and I’ll put out more info on my progress

Technorati Tags: ERML, Erlang, Markup Language, Erlang, ErlWeb

ErlWeb is the package that contains the HTTP and FTP protocols and a few other goodies.

The HTTP protocol will be served by the Erlang Web Server (EWS) which is going to be an HTTP/1.1 compliant web server with as many standard extension as I can mange to create. I already have the scanning and parsing routines down for HTTP messages and I have the base code done for the communications side as well.

While all of that can be used now with the built in HTTP server and/or YAWS the part that is motivating me to create EWS is a built in markup language that I am developing called Erlang Markup Language (ERML). This language will be similar to other markup languages like HTML, but it will have the scripting capabilities of a server side language and be extendable by any erlang module on the system.

ERML will be compiled into BEAM files so that they are easily run by any Erlang system and a caching system will detect when a file has changed and it will re-compile the file into a new BEAM file and reload the newer version into the system. This will allow for easier updating of web sites and faster development times.

I believe that ERML will eventually create a group of programmers that know how to use ERML, but may not know how to write programs in Erlang. This will hopefully expand the overall Erlang user base and open up new markets, including writing Erlang modules that are specifically meant to be used in ERML web based applications.

I have written two prototypes of the scanning and parsing routing for ERML and I’m not quite happy with them yet. I have extensive notes on how the languages is going to be implemented and I just need to find the time to create it

Erlang, ErlWEb, ERML, Erlang Markup Language

Technorati Tags: Erlang, ErlWEb, ERML, Erlang Markup Language