metajs — a new language?

No. :-)

This is a javascript extension, that only use comments to provide meta-features. I named my project files as *.metajs only to let clear: it may be a valid javascript code, but will not work as expected. This looks like a good idea, right? The features are code inclusion from another file and macros. The compiler is a BASH function, but lets talk about the features in detail first.

Why i did it? Because I was going crazy scrolling my source code with more then thousand lines, and with weird expressions that i wont put in variables or functions, for may js13kGames entry Gravity.

Including another file

When we want to minify a project code we need it all in one file, but, when we have some more "laborious", is a good idea to modularize it in some files. This explain why the including feature. The include does not only copy the content, it also put it all in a closure. This approach is also better then including many files inside the html, through <script>, because its content scope is protected and its declared variables will be valid only for this module and its sub-modules. As i said, it allows sub-modules, because it works recursively.

Well, lets see my real case, the game Gravity. I have this files:
audio.metajs draw.metajs gravity.metajs objects-update.metajs riffwave.metajs stars.metajs

Inside of some of this files we find special comments like this, on gravity.metajs:

var bgW = Math.round(bgZ*2.9);
var bgH = bgZ*2;
//include stars

(window.onresize = function() {

This "//include X" will search for a file named X.metajs and will replace the comment with its content, inside a closure.

Compiling gravity.metajs (the main file), we get:

(function() { // Start gravity
  ... gravity code ...
  (function() { // Start audio
    ... audio code ...
    var RIFFWAVE;
    (function() { // Start riffwave
      ... riffwave code ...
    })(); // End riffwave
    ... audio code ...
  })(); // End audio
  ... gravity code ...
  (function() { // Start stars
    ... stars code ...
  })(); // End stars
  ... gravity code ...
  (function() { // Start objects-update
    ... objects-update code ...
  })(); // End objects-update
  ... gravity code ...
  (function() { // Start draw
    ... draw code ...
  })(); // End draw
  ... gravity code ...
})(); // End gravity

If you know the RIFFWAVE.js, you can see this method is good to include third part projects, but when the js lib provide features as "global" variables, by only declaring it on the root scope of the file, you must change a detail: remove the "var" before the variable declaration inside the module and declare it on the parent scope, as you see on the simplified code above.

This feature also helps the minifyer as I could see while implementing it. The closures allows the minifyer to reset the variable naming and this is also good for the zip algorithm.

An open issue: the first implementation does not supports modules that includes it ancestors, because that will enter in a infinite loop.

Macros

Another long term wanted feature is macros. A simple comment define a macro and that will be valid for the entry file, and for the included sub-modules. You define a macro like this:

//macro myConst: 123
//macro sum: ( parseFloat(@1) + parseFloat(@2) )
//macro size: ( @1.length || 0 )

Then you can use like this:

someDiv.style.width = myConst * zoom + "px";
total = sum(someDiv.style.width, otherDiv.style.width) * 3;
for (var i=0; i<size(obj); i++) doSomething(obj[i]);

And the result will be:

someDiv.style.width = 123 * zoom + "px";
total = ( parseFloat(someDiv.style.width) + parseFloat( otherDiv.style.width) ) * 3;
for (var i=0; i<( obj.length || 0 ); i++) doSomething(obj[i]);

The parenthesis used on parametrized macros are not mandatory, but, as you can see on the second line of the usage example, the result is placed directly over the macro call without any "protection". In most cases were you define an expression in a macro you may need to protect it in a parenthesis to do not fight with toe operators precedence.

By this way, macros can extend js like this:

//macro forEach: for(var item,_i=0; item=@1[_i],typeof(item)!='undefined'; _i++)
var list = [123, false, 'hello', null, 'world'];
forEach(list) { console.log(item) }

That works nice, but doing this you don't have a valid javascript anymore. If you want sugar syntax you must consider other language that compiles to javascript, like CoffeeScript. (And i must to say: I love CoffeScript!)

How macros can help me?

If you don't use it for evil, that works pretty much like variables and functions. The first benefit is that you don't need to put parenthesis to your macro call if it does not requires parametrization and it still being some dynamic thing, different from a simple variable.

Macros are know for giving efficiency on C programs. Things goes faster when you don't need to allocate functions and other things in memory, but we can't handle the complexity without some abstraction feature, so you can try to put macros where you would put functions. But that is useful for javascript too? I always believe the answer is "yes", but surely the impact may not to be the same. So, i decide to test it. I implement some silly test to ensure this idea:

I did wrote this files: test.metajs and test.js, with exactly the same algorithm and data, but one with macros and the other with variables and functions.

Then i run this scripts in the browser ans in node.js, with this results:

Node.js

Arithmetic with vars run 1000000000 times, in 35.315 seconds.
Arithmetic with literals run 1000000000 times, in 27.654 seconds.
Calling functions run 100000000 times, in 13.531 seconds.
Calling parametrized macros run 100000000 times, in 13.528 seconds.

Firefox

Arithmetic with vars run 1000000000 times, in 13.677 seconds.
Arithmetic with literals run 1000000000 times, in 13.676 seconds.
Calling functions run 100000000 times, in 12.951 seconds.
Calling parametrized macros run 100000000 times, in 11.777 seconds.

Chromium

Arithmetic with vars run 1000000000 times, in 39.675 seconds.
Arithmetic with literals run 1000000000 times, in 33.426 seconds.
Calling functions run 100000000 times, in 14.232 seconds.
Calling parametrized macros run 100000000 times, in 13.886 seconds.

The values vary on each running, but they do not change so much. When I see some big variation, it's something lower then 10%. So... in this silly test we see no important efficiency gain on using macros. This may be related to the compiler optimizations or to the nature of javascript. I don't know... Maybe there are some cases where macros will show its power, but i'm less interested in macros to efficiency, and more about the final size reduction.

Size reduction?

Well... That is obvious, if we are talking about the compiled code size, it depends on the macro/value length relation. But, if we are talking about the zip compressed final size, it's harder to answer, and to give a real example. So, i will only remember you about "data entropy". In some cases you have some steps repeating over and over in a module, like that all "ctx.bP() ... ctx.cP()" on my draw.metajs. You may have the same idea as i had: replace they with a abstraction function. I got a file with less chars, but the final zip file size grows up. I did try hard, the zip size floats, but never got smaller then the size without repeated steps. That happens because the data entropy. If you have big sub strings repeating N times, the compressor only need to point this as a "word" on each N instance. That is cheap. On the other hand, if you have small sub strings repeating N times, plus a new big string, you will need more or less the same space as before with more some space to this extra word (the abstraction function). You have less chars, but more information. That also happens with variables, which required more creativity to reduce the final size.

So, macros can (not ever, but often) reduce your code complexity and the entropy. We can't be sure before test, so a good feature for metajs compiler could be the parameterization of the macro replacement: that must works as a standard macro or must define a function? So you can test how it influence the zip's final size.

Implementation

The metajs compiler is, today, a (not so much) simple BASH function hardly dependent on sed. Please, see the metajs.sh file. In that file we see the function generateJS with this steps:

build a sed script, by searching for "//macro" definitions, and for each one write a replacer sed expression;
print the closure start;
print line by line of the macro replaced content;
if the line contains "//include" , replace it by the generateJS processed content of the referenced file; (The generateJS recursive call also receives the current sed script, so macros from a ancestor module are valid inside descendants.)
print the closure end.

Simple right? I don't know if that will have a future, but if it happens, the first step will be rewrite in javascript. Here is the project page: https://gitlab.com/aurium/metajs, if you have some idea or proposal, please, comment here or issue the project.