sabato 2 febbraio 2013

GWT Benchmarks: GWT+JsVM vs JavaVM

What is GWT?

GWT (Google Web Toolkit) is a tool for compiling Java into JavaScript code; with GWT you can write both your web application and your server side code using the Java language.


What about GWT performance?

I have worked with GWT in the recent years and a simple question has arisen:

"How does the same Java code perform when running inside the Java virtual machine VS running inside the browser?"

I have decided to do a bunch experiments and write this article after taking in account these facts:
  • GWT developers state that the generated JavaScript code may be better than the handwritten one
  • V8 JavaScript Engine and the other JavaScript engines have been highly improved in the recent years
It also makes sense to measure the perormance of GWT applications running on different browsers, Firefox, Opera, Chrome, Explorer in order to see who sucks and who rules.  

I think that these results may be useful from a distribuited High Perfomance Computing perspective (via web client).


 ..but what is performance?

I'm looking to numeric and data crunching performance; input/output, graphics and multimedia performace are out of topic.
I have measured execution time of the same code compiled with GWT, running inside the browser JavaScript VM(JsVM), and running "natively" inside the Java SE7 Virtual Machine (JavaVM).
Little execution time means high performace, it's quite simple


The Benchmark suite (Internals)

I have developed a little suite (EDIT: sources released) in order to run a bunch of deterministic single threaded benchmarks covering: strings crunching, numerical crunching and data manipulation; The suite performs these benchmarks after a warm up.
The warm up has been configured in order to make sure that all the benchmark code is JITted first.
To be clear, the JsVM compiles the JavaScript after a lot of execution cycles (e.g: 11000) but JavaVM  is going to compile the  bytecode to native code in few cycles.
After the warm up, the benchmark is executed lot of times and the overall execution time is recorded, it's the same approach than taking an average execution time.
The GWT and native benchmark suites share the same Java code and benchmark configurations, only the Main classes are different.
Here is the list and description of the performed benchmarks:
  • CollectionSortBenchmark, Test Java collection sort and shuffle, manipulation.
  • FFTBenchmaark, Computes FFT's of complex, double precision data, GSL code taken from scimark
  • RegexBenchMark, apply a Regex (Javascript compatible) to a text
  • SieveBenchmark, Computes prime numbers into a given numeric range
  • StringCrunchBenchmark, String and StringBuilder manipulation, append, delete  

These benchmarks have been performed twice with different configurations, one time using little data sets, and second time on a huge data sets (big arrays, big lists and so on).
This is intended to track the JsVM and JavaVM behavior when playing with memory demanding applications, yes, this looks like an Good&Old Garbage Collector stress test. ;-)

The Benchmark Setup

The suite has been executed on an asus K51 notebook equipped with:
  • Intel Core2Duo T6600@2.2Ghz
  • 4Gbyte RAM
  • Windows7 Home (x64)
The web benchmark suite is loaded from a glassfish AP running on another server, the last GWT 2.5 SDK has been used to compile the benchmarks into Javascript code (Obfuscated javascript).

Let's see the Results

Here you have a bar graph (Figure.1) generated with (The Great) R; the bar heights represents the benchmark execution time, normalized towards the JavaVM 1.7.0 execution time. One little bar means little execution time and high performance, please note that the y-axis has a logaritmic scale.
The experiment name (x-axis) have a number appended on it, bigger the number, bigger is the data size used: FFTBenchmark5 is less memory demanding than FFTBenchmark6.
Take a look at the graph, the first benchmark on the left is the CollectionSortBenchMark3, the red bar represents the JavaVM (ver.1.7.0_05) execution time, the yellow bar is about the Chrome browser (V8 engine) and is about 2 times slower than native javaVM; the other browsers looks about 10 times slower than java code running "native".
The blue bar refers to Microsoft Internet Explorer version 9 (MSIE).

Littler execution times (tiny bars) means higher performance

Last note about these results, I have had to remove a Sieve benchmark on a greater numeric range (big numeric arrays), it's my fault, I have not tuned right the number of iterations and the javaVM has completed the task in ZERO ms!! Quite strange and annoying but I have had not the time to grasp inside this.
Another note, there are few limitations on the browser about the array size, Firefox has an higher limit but the problem applies too. So, sometimes all the JsVMs suck.


Results in Figure.1 are quite self-explanatory.

From a GWT application standpoint Chrome rules, aka V8 rules.
A java program running inside the Chrome browser is going to perform about 2 times slower, but it's so near to JavaVM, so near.
Another interesting point is Chrome regular expression execution time: Java Regex bechmarks performs far better in Chrome than in JavaVM; You know, the GWT Regex classes are backed directly by low level JsVM functions and they look terrific.
Kudos to Chrome regex implementation.
The Java GWT performance in Opera,Firefox and MSIE sucks a bit, running the code on those browsers is 5 or 10 times slower than running on JavaVM.
Firefox sucks a lot with numeric workloads or crunching big arrays of numeric data, running 40 (or 50) times slower than the JavaVM .
I hardly suspect that the GWT developers devote themselves to Chrome optimization, and for this reason GWT compilation for the other browsers sucks. I have no clue, do you have? :-)

From an HPC standpoint, GWT apps on Chrome performs very well compared to native Java Virtual Machine apps, so, GWT looks like a good tool for writing distributed High Performance Computing web clients.

Comments and criticisms are welcome

16 commenti:

  1. Interesting post, are the sources available? I'd like to compare other Java to JavaScript frameworks.

    1. I'll release the sources.. I hope tomorrow. I think I'll release the code thru Google code or similari.Thanks forma feedback

    2. Sources released, hope the project structure make sense:

  2. I'll release the sources.. I hope tomorrow. I think I'll release the code thru Google code or similari.Thanks forma feedback

  3. Sources release postponed to tomorrow ;-)

  4. Cool stuff.
    It would be interesting to include NaCL in the comparison.

    1. @Unknown NaCl is abit out of topic, I'm focusing on Java and NaCL (I haven't look at this deeply) doesn't fit. Perhaps in a far far far future. :-)

  5. Thank you !!
    It would be interesting to run the benchmarks with ClosureCompiler (-XenableClosureCompiler) and even with -draftCompile maybe.

    1. -draftCompile is not interesting indeed cause it disables javascript optimization and I'm looking for the best aviable performance with production code. ClosureCompiler is a very interesting option and I'll add in future benchmark updates.

    2. I forgot,
      You are welcome :-)

    3. Well,
      yes -draftCompile disables optimization but may be interesting to benchmark gwt without optimization
      to understand 'how much' the optimization is relevant.

      Thank you again.

  6. Actually, we didn't make much effort to optimize for Chrome, because we didn't need to. I think the most we did was change codegen that caused v8 to de-optimize code, but we have done that for all the browsers.

    1. @John, thanks for the info, I was just curious, you know, GWT chrome performance totally annihilated the other browsers. I was doubtful about. So, V8 is simply magnitudes better than others.

      I have understood you are directly involved in GWT developement, amI right?

    2. The reason for chrome's excellent performance is likely due to it's JIT compiler; as your code is running, v8 optimizes the assembly code it uses, similar to how java with the -server flag performs.

    3. thanks James, quite interesting info.

  7. Questo commento è stato eliminato dall'autore.