next »

[Marvin Fröhlich]

Do you see what I mean and would you go in this direction ?

I see, what you mean and it sounds good. The primary goal should be to limit GC-overhead and network-traffic to a minimum, which means to only send primitives over the net, which could perfectly work with your skeleton.

Marvin

['n ddrylliog]

Now maybe we could even work hand-in-hand with the JiBX team itself, as they have very much in common with what I suggest, for example :
- a well-designed and flexible binding XML file format
- expertise (and implementation) of bytecode enhancement
etc.

I've just sent an e-mail to Dennis Sosnoski (Jibx developer) about helping us.

['n ddrylliog]

So here is my mail to Dennis Sosnoski

Hi,

We at Xith3D have been users of Jibx for a few months now and are so
far very, very satisfied with it !

We are currently having a discussion on our forum on implementing a
Networking API for games. Its usecase would be to send Java objects
over the network, with minimal GC overhead, minimal bandwith use,
minimal CPU usage, well, you're an optimization guy, you know the trick.

I suggested we took an approach similar to Jibx's one, with binding
definition files and byte code enhancement. Then I thought maybe your
expertise would be very valuable for us. If you'd be so kind as to
give a little of your time to us 

The forum discussion can be found here :
http://xith.org/forum/index.php/topic,631.0.html

The Xith3D project is a 3D game engine consisting of a scenegraph, a
renderer, tools for sound, content creation, graphical user interface
creation, debugging, maths, and networking (soon). It's BSD-licensed,
hence, open-source. We are making no profit for it and thus we can't
afford to offer you a retribution (unless the members collect some
money).

Greets,
Amos Wenger, one of the Xith3D Lead Developers ( http://www.xith.org/ )

['n ddrylliog]

And his answer (btw, thanks Dennis !)

Hi Amos,

Doing bytecode generation for the conversions is likely to be a fairly
big undertaking. How many types of objects do you plan to handle? You
can fairly easily handcode the serialization/deserialization directly,
using interface methods. That's probably what you'd want to do to start
with even if you plan to go with bytecode generation in the longer term
- that way you'll work out the kinks in your serialization mechanism
first, which will make it simpler when you want to go to bytecode.

On the bright side, serializing/deserializing binary is going to be
easier than working with XML, and you can avoid most of the complexity
of JiBX bindings. Instead, all you'd need is a list of fields to be
serialized for each class. You could even build the custom
serializing/deserializing into the framework rather than needing to
specify it in a binding, by having the binding compiler check if the
user has provided an implementation of the methods and if so skip
modifying that class. I'd suggest that you also consider not using
handles for every object, too - many objects aren't worth handling with
handles, the overhead is more than the gain.

I did look at the forum, but don't want to join up at this point. Feel
free to pass along my comments, though, and if you decide to go this
route I can certainly offer more pointers in the future.

Cheers,

  - Dennis

[Pandaemonium]

Sorry for my absence lately, I've been on vacation in Maui 

On the bright side, serializing/deserializing binary is going to be
easier than working with XML, and you can avoid most of the complexity
of JiBX bindings.

I definitely agree. I think the overhead involved with parsing xml in the network layer will be too high anyway. Direct binary serialization should be the way to go. I've been doing a lot of research in this area and I found that I'm going to use Mina for my networking layer. The performance benchmarks I've seen for it are very impressive and the best part....you can choose your threading model for your gaming server. Also, the ease of use and features make it worth considering (We have been using Mina at the company I work for and it's been very fast and reliable). Darkstar is built on top of Mina so they also must have seen a great value in its implementation. Here's a list of main features http://mina.apache.org/features.html

[Marvin Fröhlich]

I definitely agree. I think the overhead involved with parsing xml in the network layer will be too high anyway. Direct binary serialization should be the way to go. I've been doing a lot of research in this area and I found that I'm going to use Mina for my networking layer. The performance benchmarks I've seen for it are very impressive and the best part....you can choose your threading model for your gaming server. Also, the ease of use and features make it worth considering (We have been using Mina at the company I work for and it's been very fast and reliable). Darkstar is built on top of Mina so they also must have seen a great value in its implementation.

Benchmarks don't always express the GC problem. While general performance can be great, there can still be those annoying hick-ups caused by GC overhead.

I don't know, how Mina works. But if it uses plain old object serialization, there will be GC overhead, which we cannot accept in a real-time game. It is certainly acceptable for round-based games or stuff like that, where the VM actually has time to handle GC.

The way I understood the "jibx-similar" way talked about above, is, that we would use the same technique, but without XML. Just pushing incoming attributes directly to container classes. Well, I do agree, that the same performance might be achievable without bytecode enrichment and bindings, which I would definitely prefer. If we don't need it for performance, we should avoid the complexity and overhead of bytecode enrichment.

Do you know, if Mina works without GC overhead? If it does, we should definitely consider using it.

Marvin

[Pandaemonium]

I don't believe GC is an issue with Mina. A developer is free to chose whatever protocol is desired...whether its text based, binary based, or object based. If an object based protocol is chosen, then GC will probably be an issue. Though, an object based protocol is very useful for rapid prototyping where a proof of concept is more important than performance.

I'm currently rewriting my network layer to use Mina. What I'll do is once I'm done and ready for stress testing I'll run it through a profiler and get some results. I'll list the results and my thoughts here. This might take me a few weeks since I'm getting pounded at work.

['n ddrylliog]

I agree with Marvin here, and with Pandaemonium, so just to sum up :

- Yes with binary we don't have the overhead of XML
- Yes Mina is the way to go
- Yes if performance is okay without bytecode enhancement we'll go without it : though it's an option to keep in mind
- Yes we are waiting for your results, Pandaemonium

About bytecode enhancement.. I don't think it's really that complicated.. (especially in our case, cause with binary it's much much simpler)

[Pandaemonium]

Quick update. I've finally found the time to completely rewrite my networking code to use Mina. I have yet to perform tests. Currently for simplicity, I'm using object serialization since I'm more concerned with creating a prototype and to test any problems with garbage collection. Mina makes it extremely easy to change the message passing format later, if that is what a dev desires. I've been recruited into another dev team at work so my time is more limited as before. I'll still try to get some tests done.

After getting my hands dirty with Mina, I'm starting to wonder if such a library should be integrated into a scenegraph api. Mina is very easy to use, and the documentation for it is decent. If Mina was integrated, Xith would just serve as a wrapper for an already easy to use library. Imo, wrappers should only be used to  simplify or normalize complex modules (to limit how the external module is used).

[Marvin Fröhlich]

After getting my hands dirty with Mina, I'm starting to wonder if such a library should be integrated into a scenegraph api. Mina is very easy to use, and the documentation for it is decent. If Mina was integrated, Xith would just serve as a wrapper for an already easy to use library. Imo, wrappers should only be used to  simplify or normalize complex modules (to limit how the external module is used).

Actually this was the goal of the entire thread . After you have your demo ready, I will have a look at it and start with the xith integration.

Marvin

[Pandaemonium]

Sorry this has taken so long and in kukanani's words:

has gone almost nowhere

I've been really busy, but I have done some more work with Mina in my game to see if it's viable and it still looks promising.

I am not sure how extensive the demo should be. The quickest way would be to simply create a Client and a Server and demonstrate how they would use Mina to communicate to each other. Is this all that needs shown in the demo?

[Marvin Fröhlich]

Sorry this has taken so long and in kukanani's words:

has gone almost nowhere

No problem. We all have things to do .

I've been really busy, but I have done some more work with Mina in my game to see if it's viable and it still looks promising.

Good to know.

I am not sure how extensive the demo should be. The quickest way would be to simply create a Client and a Server and demonstrate how they would use Mina to communicate to each other. Is this all that needs shown in the demo?

Well, sounds like all we need to judge the API. What I am interested in is GC-overhead. You could use a profiler (like YourKit) to monitor garbage collection package and even per class/method. If Mina doesn't produce any garbage and we judge the API as acceptable, Mina is taken .

Marvin

[kukanani]

Sorry this has taken so long and in kukanani's words:

has gone almost nowhere

Sorry, Pandaemonium, I didn't mean any personal offense.  Am I forgiven?

[Pandaemonium]

Sorry, Pandaemonium, I didn't mean any personal offense.  Am I forgiven?

Hehe, no problem. It was just the sort of comment I needed to motivate me to get off my ace and get the net demo done, which I should say, it's done now. I just need to post the files. I'll post them when I get home tonight (not sure what time it'll be, I have tons of Christmas shopping to do still )

One thing though, I went to download both JProfiler and YourKit and noticed that they're only evaluation copies (for free). Is there someone who has a full licensed version that would like to profile the demo files? I hate downloading/installing X-day software.

['n ddrylliog]

Pandaemonium, just write to the e-mail adress "sales" at "yourkit.com" and tell them you're a developer from the open-source Xith3D project, and that, as Amos Wenger asked them several times before, you'd like an Open-source license for working on Xith3D.

They will send you a full-working key, but please be honest : don't use it for commercial projects.