TLS 1.0 in Erlang, no OpenSSL

I was borrowing ideas from Erlang/OTP for my X Erlang implementation as a matter of routine. Sometimes the shortcut did not work because I would not understand Erlang/OTP sources or would think they are too convoluted.

Erlang/OTP approach to crypto functions is a disappoinment. It uses external OpenSSL library through a port driver. Erlang has nice binary parsing, big numbers, hash function BIFs etc and all these are replicated in OpenSSL. Plus OpenSSL itlself is three times bigger than whole X Erlang.

Now I have a working TLS library with all cryptographic primitives implemented in Erlang. No OpenSSL or anything of its ilk. The exception are MD5 and SHA1 functions which are BIFs done in C. I barely believe when my TLS server makes its way through a jungle of TLS handshake calculations and starts shoveling application data.

I tesify, and give my oath, that TLS is implementable using RFC 2246 and RFC 2104 (and, of course, Wikipedia).

Actionscript talks Erlang

Continuing the discussion on how to make Erlang more Ajax friendly, I would like to share the format I use for representation of Erlang terms in my Flex/Actionscript-based X Erlang IDE.
The format codenamed as ASON (just like JSON, but for Actionscript) allows representation of every possible Erlang term in Actionscript (again funs are somewhat of a hassle) and is built upon Actonscript's support for inline XML syntax (e4x).

1. Atoms, numbers and strings encoded similarily

<value atom="abc" />
<value string="def" />
<value number="123" />

1. Lists and tuples use XML nesting

<list>
<value numbe="1" />
<value numbe="2" />
<value numbe="3" />
</list>

<list />
// []

<tuple>
<value numbe="1" />
<value numbe="2" />
<value numbe="3" />
</tuple>

3. Binaries use XML text entity

<binary>010203ff</binary>
// means <<1,2,3,255>>

4. Pids, ports and references are alikes

<pid node="local" serial="1" creation="0" />
<port node="local" serial="1" creation="0" />
<ref node="local" serial="1" creation="0" />

Now, I can call an Erlang function from Actionscript like this:

Erlang.apply("lists:nth", <list><value number="2" />
<list><value atom="a" /><value atom="b" /><value atom="c" /></list>
</list>, onResult);
function onResult(success:Number, result:XML):void
{
Alert.show("lists:nth(2, [a,b,c]) returns "+result.@atom+".");
}

Of course, there are easier ways to retrieve the second element of the list in Actionscript.

Moved to Flex/Actionscript

Why it was never mentioned that dojo toolkit is reimplementation of Flex framework? It would probably saved me many months of poking in dark corners of Javascript with mediocre results.

Today, one month after the 'discovery' my X Erlang IDE has a syntax highlighting editor, a build system, a process viewer, an Erlang console, etc running in Flash Player:



What I am still missing sorely is function call completion. The exact positioning of a small floating window near the cursor is nearly impossible to implement reliably in Javascript. Flex/Actionscript does this and many other things elegantly, is identical in all browsers and is present in 90% of computers around the globe.

I have to mention that it is not possible to process right mouse button click with Flex/Actionscript. There is always, they say, something.

Javascript talks Erlang

When extending my web-based IDE for X Erlang I stumbled on the fact that I am doing much the same thing over and over again adding a small snippets of code on the server and corresponding integration-only additions to Javascript on the client. Many times the obtained result would be an Erlang term represented by opaque string for the user (me) not the software to understand and interpret.

The next big step was standardizing the way Javascript (JSON) objects are generated from a subset of possible Erlang terms. For instance, [{name:"Judas"}, {occupation:"priest"}] would be converted to { name: 'Judas', occupation: 'priest' } on the Javascript side. So far so good, but the encoding scheme would work only in one direction - from Erlang-based server to the web browser.

The enligtment came later when I recognized that it is not at all necessary to add a new server stub for each request the web application may want to send. The web application may just call Erlang functions directly by their names, provide properly-encoded parameters and expect results as JSON messages. The question was the two-way encoding between Erlang terms and JSON objects. Here comes the scheme:

1. Erlang strings, including empty string "", are represented by Javascript strings

"" -> ""

abc" -> "abc"

2. Atom 'true' and 'false' become Javascript boolean values

'true' -> true

'false' -> false

3. Numbers are just numbers

123 -> 123

123.5 -> 123.5

4. Erlang atoms become objects with a single property "atom"

'abc' -> {atom: "abc"}

5. Erlang binaries become similar objects with a single property, this time named "binary"

<<1,2,255>> -> {binary: "0102ff"}

Binary data is encoded into string, each byte represented by two hex digits

(more compact representations can be imagined)

6. PIDs, References and Ports are encoded as objects too

#Ref<1.2.3> -> {reference: 2, node: 1, creation: 3}

#Port<1.2.3> -> {port: 2, node: 1, creation: 3}

<1.2.3> -> {pid: 2, node: 1, creation: 3}

7. Tuples become objects with integer field names

{T1,T2,T3} -> {1: T1, 2: T2, 3: T3}

8. Lists (which happened not to be strings) are the easiest

[T1,T2,T3] -> [T1, T2, T3]

The described encoding allows to represent any Erlang term as a Javascript value (but not vice-versa). The small open is funs. Today, they are encoded as {fun: name} but this representation is not reversible. Besides it dificult to understand how to reconstract fun object in Erlang (binary_to_term()?).

The additional sugar to the above scheme is record descriptions. For known records, the representation of tuples could be made simpler and more manageable. For instance, the record 'req' is known to have 'module', 'function' and 'args' fields. Then, the Erlang tuple {req,Mod,Fun,Args} is represented by more concise Javascript object with explicit field names: {record: "req", module: Mod, function: Fun, args: Args}. Record description work both ways, during encoding and decoding.

The implementation of the described representation allowed my dojo-based web client to call Erlang functions on the server and pass complex structures back and forth with little or no hassle.

Now, the Erlang console which evaluated expressions using erl_parse and erl_eval stores the resulting bindings on the client as Javascript objects and passes them to subsequent calls as a context not heeding their alien nature.

Web-based IDE for X Erlang

In one of my previous posts I described a console application that allows the user to interact with X Erlang instance. As the logical next step I wanted to make it more a 'Visual Studio' than a 'DOS prompt'.

After two months of petty bickering I finally settled to use dojo for portability among browsers and ready-to-go widget goodies. Currently, the application allows management of the in-memory filesystem, editing of Erlang code using CodeMirror control and compilation of the code. It even jumps to the correct line in the source 

code when the error message is clicked in the compiler output pane!

The progress is stalled by the capabilities of the X Erlang itself. Now it is not possible to 'pause' a process to observe its internals.

The current version looks like this:

X Erlang shell application

To demonstrate X Erlang system I quickly put together a simple web server which serves static files and runs scripts. On top of it I implemented a simple shell application which allows to enter and evaluate arbitrary Erlang expressions from a web browser window. First I put the system running on an Amazon EC2 server but then I put it down after all my friend had a look at how it works.

The typical shell session looks like this:

X Erlang v0.2 is here, type '?' for help

> test:primes(100).
[2,3,5,7,11,13,17,19,23,29,31,37,41,43,47,53,59,61,67,71,73, 79,83,89,97]

> [{X,Y} X <- test:primes(100), Y <- test:primes(100), Y =:= X+2].
[{3,5}, {5,7}, {11,13}, {17,19}, {29,31}, {41,43}, {59,61}, {71,73}]

> code:all_loaded().
[lists,test,dict,orddict,queue,sets,erl_parse,regexp,gen_tcp,inet,io_lib_format,erl_lint,dashboard,stdio,gb_sets, io_lib,erlang,io,packages,io_lib_pretty,prim_erlang,code,erl_eval,string,init,erl_internal,files,ordsets,efile, erl_scan,spur,error_handler,x_internal]

$_

The blue characters are entered by the user, the rest is the output of the system. The first
request is to calculate a few prime numbers. Every programming language is demonstrated using prime number routine. Why should I be different? The second request is trickier. The entered expression finds all 'twin primes' less than 100. As you noticed X Erlang supports list comprehensions (but not yet binary comprehensions). The last expression is a maintenance job. It shows which modules are loaded.

Nothing to fancy but it works. It took me a day or two to program a robust shell application on top of a web server. It may take even less for other programmers using other versions of Erlang.

X Erlang as a userland OS

Erlang system has many features of an operating system. The most obvious is processes, the more subtle is modules.

X Erlang takes a few steps further to the autonomous userland OS than OTP does. In X Erlang the operating system is never searched when the system needs to load a module. Two dozens of modules are 'embedded' into the X Erlang executable. There is also an internal storage for module images. These images may be loaded on request. If the referenced module is not embedded and it is not found in the internal storage then {undef,...} is thrown. The file system of the underlying OS is never touched in the process.

To calm down the readers I need to ascertain them that there is a file system in X Erlang. It is internal and in-memory, simplistic, yet hierarchical. In X Erlang file:write_file("temp1.txt", <<1,2,3>>) creates an in-memory 'file' (or rather a named binary). The external file system of the underlying OS is still accessible though 'efile' interface. Compiler works on files and emits output files to the internal file system only.

In X Erlang you need exactly one executable file copied to a freshly installed operating system to launch your application. No runtime, no libraries, a single file typically of less than 2 MB size.