Javascript compression / minimize

[skiddy56]

I'm writing an application that's pushing the limits of the 65K that we have to work with.  I haven't had this restriction since I worked on the Commodore 64K (1982?).  But I'm not complaining.  I think it's unbelievable that we can write programs to run on a watch.  I'm having a ball writing, testing and rewriting and retesting and rewriting...

I wasn't being honest when I said it was pushing the 65K limit.  I've gone well over it, several times!  I'm very familiar with the "out of memory error".  In fact, we're best friends. 🙂

During the rewrites, I figured I could use any of a number of tools out there to compress javascript code.  Little did I know, they don't work with the flavor of Javascript we're using. 😞

I did, however, find a php program that will minimize the script.  It works really well to remove all white space and, especially for me, all the comments.  The program can be pulled from git:

https://github.com/matthiasmullie/minify

If anyone knows of a compression tool that works for us, I'd be interested to here about it.  I ended up rolling my own, somewhat specific, compression tool. I saved a bunch of memory.  Who would have thought that descriptive variable names was a bad thing...

Rich

[Gondwana]

Thanks for the info! I'll have to watch (ahem) this; I love self-documenting code.

FWIW, the CPU power of the Ionic is equivalent to a desktop CPU from about 1995 (120MHz single core), so I suppose we should try to adopt coding conventions from those days (especially when using interpreted code).

[Cmspooner]

I tried to use uglify on my code and it didn't save me that much. active ram. The biggest gains I made where movide data structures into functions so that return smaller chunks of data, see the history on my schedule.js file.

 I also made some some gains moving my DOM references inside of the functions that use them.

[gaperton]

Fitbit SDK uses RollupJS, it should already perform JS sources minification during the build. If it's not, it's a good idea to request it from the dev team (minification is a standard part of the modern JS building pipeline and it's trivial to turn on).

Speaking about the heap memory consumption, keep in mind that every floating point number is allocated in the heap separately and takes 8 bytes + 4 bytes for the value. It means that while "let x = 1" will take 4 bytes, "let x = 1.5" will allocate 12.

https://github.com/jerryscript-project/jerryscript/blob/master/docs/04.INTERNALS.md

Thus, for instance, arrays of floats should be avoided because they allocate 3x more memory than the regular array. Float32Array is for the rescue.

[gaperton]

You could save even more if you will use minutes from the beginning of the day instead of strings ("military time" as a number will do as well).

[
{name: "Before School", start: 395, end: 455},
...
]

Reason is that a short integer (29-bit and less) is being packed inside of the 32-bit JerryScript value, while the string is being allocated in the heap with a pointer stored inside of the value.

You should save even more memory if you will prefer object of arrays to the array of objects:

{
    names : [ "Before School", "Warning Bell", ... ],
    starts : [ 395, 455, ... ],
    ends : [ 455, ... ]
}

And you will use twice less memory if you will use Int16Array instead of the regular Array. Int16Array can't be resized, though. But it's okay for many scenarios.

Read about the details of the JerryScript/Fitbit OS memory management here:
https://wiki.tizen.org/images/5/52/04-JerryScript_ECMA_Internal_and_Memory_Management.pdf

[Cmspooner]

If I hit the memory ceiling again  I will definitely start going that way. I want to keep that file as non-programmer friendly as possible to make it easier for people to build their own schedules. 

One day I will make it so you can build a a schedule in the app itself... But I have too much other stuff to do before that. 

[skiddy56]

I like that solution.  Very simple.  I love simple. 🙂

[skiddy56]

Ok, so replies don't go to the message you actually reply to.  They just go to the bottom of the list.  Good to know.

I was responding to the use of a function to replace a static array.  It's so simple.  And so easy to implement.

Hi Strider,

I noticed the floating point number problem.  I used a little javascript routine I downloaded from http://code.iamkate.com/javascript/finding-the-memory-usage-of-objects/ created by Kate Morley.

Pass in the object and it returns the approximate size in bytes.

I wasn't sure what to do about the floating point problem, now that I know there's a Float32Array I'll take a look at ALL the data types. 

Thanks for that and for the links to the Jerryscript documentation.

BTW, I'm using the Ionic Multi-screen app template that you created.  Thanks for that too. 

Rich

[Cmspooner]

The use of functions to store data in JavaScript came from my work with Espruino, an embedded JavaScript on esp8266 and the like.

I seem to only write JavaScript on tiny embedded devices, not on web pages like it was intended  

[gaperton]

> The use of functions to store data in JavaScript came from my work with Espruino, an embedded JavaScript on esp8266 and the like.

In JerryScript, it doesn't completely prevent VM from preallocating assets. From the docs:

---
JerryScript does not have a global string table for literals, but stores them into the Literal Store. During the parsing phase, when a new literal appears with the same identifier that has already occurred before, the string won't be stored once again, but the identifier in the Literal Store will be used. If a new literal is not in the Literal Store yet, it will be inserted.
---

Thus, string literals in this example will be preallocated in the Literal Store before dayToSchedule function will have any chance to be called. However, an object itself (which is a linked list of property pairs) consumes about 8 bytes per property. So, this code here delays an allocation of ~64 bytes to the moment when it will be actually needed (and it can be reclaimed later by the CG when it's not needed).

function dayToSchedule(){                
    return {"Sunday": "No School",
          "Monday": "Normal",
          "Tuesday": "Normal",
          "Wednesday": "Normal",
          "Thursday": "Normal",
          "Friday": "Normal",
          "Saturday": "No School"
    }
}

This code, however, completely avoids this 64 bytes allocation:

function dayToSchedule( day ){                
    switch( day ){
        case "Sunday": return "No School";
        case "Monday": return "Normal";
        case "Tuesday": return "Normal";
        case "Wednesday": return "Normal";
        case "Thursday": return "Normal";
        case "Friday": return "Normal";
        case "Saturday": return "No School;
    }
}

[Cmspooner]

Ohhh, Interesting...I will definitely make some changes along those lines! 

[skiddy56]

It makes me wish I had some static arrays that I could change...  I will keep it in mind for the future. 

[gaperton · Accepted solution]

Guys, it seems that these optimizations with wrapping objects in a functions don't really work with JerryScript. Quite an opposite, they make things worse. Just made some tests, there are no doubts. It's an antipattern.

https://github.com/gaperton/ionic-views/blob/master/docs/optimization-guidelines.md#do-not-use-funct...

[skiddy56]

@gaperton That explains a lot.  My code consists of a lot of small functions.  I'll try a version with the functions merged and see what happens to the memory.  Your article should be a must read for anyone developing for the Fitbit.  Nice job.  Thanks for sharing.

[skiddy56]

I just reread your article, especially this statement:

...Function's bytecode takes more memory than the preallocated object, and both the heap and the code share the same memory quote.

I think I took that the wrong way.  Initially, I read it as any function's bytecode takes a significant amount of memory.  I think what you really said was, in this case, the function's code takes more memory...

Still it begs the question, what other assumptions have I made, coming from a more standard programming environment, are wrong.  Time to go back and throw out my assumptions and re evaluate my code. 

[gaperton]

Yep. Just for the case - there's a way to measure the memory consumption. So, you can easily check if it really helped, or not.

https://dev.fitbit.com/build/reference/device-api/system/

[gaperton]

Yes, I meant something like “in this particular case an object takes 80 bytes, while the function doing the same job takes about 300”.

[gaperton]

> I think what you really said was, in this case, the function's code takes more memory...

Thanks. Changed the article accordingly.

[gaperton]

@skiddy56

>  Initially, I read it as any function's bytecode takes a significant amount of memory.

Apparently, you were right about that too. An empty function takes about 100 bytes. It happens because functions are first-class objects in JS. Thus, not only the byte-code is being generated when we define the function, but the Function object with properties (the same property pairs list) is being created in the heap.

[skiddy56]

Nice to be right even if I didn't mean to be.   Unfortunately I've had some prior success in cutting down on the memory usage by replacing a couple of objects with arrays and functions.  I removed the object by replacing the properties with a global associative array and the object's methods with functions.  I don't have exact numbers, but it seemed to save a considerable amount of memory.

In another scenario, I have a popup/dialog "module" that I was attempting to optimize for memory thinking that the popup might be called when there was limited space available. In the module, I created just one local variable. It went against my normal coding style, but I coded in a bunch of getElementById calls to the same objects. I would normally have created a local variable and referenced the local variable but I was attempting to cut down on the space needed to run the popup by not creating the local variables. I wasn’t worried about the execution time, just the space usage. In reevaluating my assumptions, I decided to try an experiment and go ahead and create the variables for the objects as global constants.
To summarize: The original code has 24 calls to getElementById and no local variables. In the revised routine the 24 calls were replaced with 6 global constants initialized with a call to getElementById, one for each of the objects. The routine then referenced the constants to get the object properties.
For example,

document.getElementById("DIA_btnYes").style.display="none";

 was revised to:

const DIA_Yes=document.getElementById("DIA_btnYes");
DIA_Yes.style.display="none";
...

 So other than the initialization of the 6 constants, both routines have the same number of executable statements.
I ran the program 7 times for each routine and captured and averaged the minimum (load) memory and the peak memory. The results were right along the lines that you predicted. The original code, with no constants, had a lower initial load, but a higher peak load. I wondered why the original code, which had virtually no local variables, had a higher peak load. My suspicion is that each call to getElementById, placed a variable on the stack. So even though I wasn’t creating the local variable, it was being created for me.
I then used a compression routine to compress variable names and remove comments and white space and again ran each routine 7 times. I was curious if I would really see a difference. It’s a tiny program so I wasn’t expecting too much but I do tend to get a little verbose in my variable names.
As expected, the compressed versions did save some memory. The relative results were the same with the original routine having the best initial load memory while the revised routine had a lower peak memory.
Interestingly the compression routines were able to save 219 bytes for the revised version but only 48 bytes for the original version. The difference is the compression routine couldn’t compress the 24 calls to getElementById but could compress the 24 calls for the revised version since it was using a variable name. For example all references to DIA_Yes were compressed to A0.

Based on the results, I'm going with the revised version.  Although the original had a lower initial load, the peak load was higher.  And further, the revised version was able to be compressed and had a yet lower peak load.