Welcome, Guest. Please login or register.

Login with username, password and session length

 
Advanced search

11855 Posts in 1569 Topics- by 3058 Members - Latest Member: xynewhand

12. February 2012, 03:16:42 AM
Xith3D CommunityXith3D InternalsDeveloper discussion (Moderators: Marvin Fröhlich, 'n ddrylliog)Testcases in general
Pages: [1]
Print
Author Topic: Testcases in general  (Read 941 times)
horati
Global Moderator
Getting respectable
*****
Offline Offline

Posts: 393


View Profile
« on: 18. June 2010, 12:07:59 PM »

Maybe you should have a look at the RenderLoop, as the problem and bug reported is related to the source distribution from SVN. The RenderLoop obviously will cause a dead lock, when started as RUN_IN_SAME_THREAD, which is the case, if I simply called begin() without arguments in my application. Documentation tags and implementation of begin() also differ a little bit and the documentation tag argues the converse of what is really executed in the code. ;-)
...
PS: unfortunately unable to post the bug in the correct forum. However, it seems a general code review of the class RenderLoop and others is advisable, as there are many obvious bugs seen from the source (as there are some others bugs found within different class sources, which I don't remember all in detail. Anyway, I wouldn't use the current source distribution in a production environment)

Actually, I noticed several things like this when I came back to the project last year.  Because all the test cases continued to work and I didn't have any burning reason to check into the code, I never pursued further.  Throughout the code base, there are several places where code and Javadoc no longer match.  I generally agree with Dykstra about this situation: "When code and documentation do not match, both are generally wrong."  It does concern me.

Moreover, the project I wrote that got me to join Xith no longer runs if I update to a recent version.  Obviously, I had to make some updates for the API changes.  That part was very easy; however, I ran into all kinds of problems that were not worth solving.

The sum of these problems (now that they have been reported by someone besides me -- someone who actually needs these problems resolved) makes me believe that we need to enhance the test cases.  In particular, we need the following prioritized items:
  • Some cases that are fully automated in addition to the purely visual cases we have now.  Many of these could be achieved by a few changes to the test case super class plus a test runner that executes all cases with all options in sequence.  It should fail if it detects wildly variant FPS or any exceptions.
  • New cases that are non-visual.  We should test things like boogle's problem: if initialized with X parameters, we expect Y result.  A sub-project needs to be started to come up with all these non-visual tests.  This sub-project would be great for the active users who might feel reluctant to change the regular code base for fear of hurting it.
  • A test runner (included in the sequential runner) that does a pixel by pixel comparison of key frames.  This would allow us to visually inspect the test runs and take screen shots of what we consider correct.  These screen shots would then be included in a new visual inspector test case that would pause the program at frame X where X is measured as frames since program start (so it ignores FPS).  Either frame X should contain the same pixels with every rendering or we need to know why it does not.
Logged

Kevin
"It may not seem like a big deal, but ignorance of character encoding issues leads to insidious code rot akin to y2k."
http://stackoverflow.com/users/3474/sylvarking
Marvin Fröhlich
Xith Lord
Administrator
Guru
*****
Offline Offline

Posts: 4337


May the 4th, be with you...


View Profile
« Reply #1 on: 18. June 2010, 03:03:18 PM »

Actually, I noticed several things like this when I came back to the project last year.  Because all the test cases continued to work and I didn't have any burning reason to check into the code, I never pursued further.  Throughout the code base, there are several places where code and Javadoc no longer match.  I generally agree with Dykstra about this situation: "When code and documentation do not match, both are generally wrong."  It does concern me.

I agree. But it is a lot of work get keep these matching. It is mostly my fault, that it came to this point. Sorry about that. It is even more work to do this now (search the code for unmatching code/doc). I am not sure, if or when I will have to time to do this.

In particular, we need the following prioritized items:
  • Some cases that are fully automated in addition to the purely visual cases we have now.  Many of these could be achieved by a few changes to the test case super class plus a test runner that executes all cases with all options in sequence.  It should fail if it detects wildly variant FPS or any exceptions.
  • New cases that are non-visual.  We should test things like boogle's problem: if initialized with X parameters, we expect Y result.  A sub-project needs to be started to come up with all these non-visual tests.  This sub-project would be great for the active users who might feel reluctant to change the regular code base for fear of hurting it.
  • A test runner (included in the sequential runner) that does a pixel by pixel comparison of key frames.  This would allow us to visually inspect the test runs and take screen shots of what we consider correct.  These screen shots would then be included in a new visual inspector test case that would pause the program at frame X where X is measured as frames since program start (so it ignores FPS).  Either frame X should contain the same pixels with every rendering or we need to know why it does not.

I fully agree (except for the pixel-by-pixel comparison, which will not be possible because of rounding problems). Who wants to do this?

Marvin
Logged
Pages: [1]
Print
Jump to:  

Theme orange-lt created by panic