Bug 197065 - No way to specify command-line arguments in a functional test
No way to specify command-line arguments in a functional test
Status: RESOLVED FIXED
Product: platform
Classification: Unclassified
Component: NB JUnit
7.0.1
All All
: P2 (vote)
: 7.1
Assigned To: Jaroslav Tulach
issues@platform
: API_REVIEW_FAST
Depends on:
Blocks: 197066
  Show dependency treegraph
 
Reported: 2011-03-24 20:22 UTC by tomwheeler
Modified: 2011-05-14 15:25 UTC (History)
4 users (show)

See Also:
Issue Type: ENHANCEMENT
:


Attachments
Patch which allows test configuration to accept startup arguments (9.28 KB, application/octet-stream)
2011-03-28 18:40 UTC, tomwheeler
Details
Revised patch which satisfies Y01, Y02 and Y03 (11.97 KB, application/octet-stream)
2011-04-08 18:31 UTC, tomwheeler
Details
Revised patch which increments version number and updates documentation (13.93 KB, application/octet-stream)
2011-04-12 17:26 UTC, tomwheeler
Details

Note You need to log in before you can comment on or make changes to this bug.
Description tomwheeler 2011-03-24 20:22:17 UTC
I needed to write a functional test for my application in which the UI behaves a certain way when a specific argument is provided on the command line.  The use case is conceptually similar to when you use --open <file> with NetBeans IDE and need a test case to verify that the requested file is indeed opened in the editor.

After looking at the public API of NbModuleSuite and some related classes, I was unable to find a way to specify startup arguments in a functional test.  While one workaround would be to use system properties instead of command line arguments, there are certain things that cannot be passed this way.  JVM arguments are one example of this, while NetBeans' startup parameters (--laf, --branding, --open, --locale, etc.) are another.  

Therefore, this issue requests that support be added to NbModuleSuite to allow a functional test to specify command-line arguments which are used to start the application under test.  I plan to submit a patch to support this in the next several days.
Comment 1 Jaroslav Tulach 2011-03-25 06:03:30 UTC
Submit a patch.
Comment 2 tomwheeler 2011-03-28 18:40:15 UTC
Created attachment 107347 [details]
Patch which allows test configuration to accept startup arguments

I've attached a patch I developed which adds support for startup arguments in the unit test.  It should be pretty straightforward to use, but here is an example:

    public static Test suite() {
        Configuration config = NbModuleSuite.createConfiguration(TestSayHelloMenu.class);
        config = config.addTest("testClickFileSayHelloMenuItem")
            .clusters(".*").enableModules(".*")
            .gui(true)
            .addStartupArgument("--branding", "functionaltestsuite")
            .addStartupArgument("--fontsize", "18");

        return NbModuleSuite.create(config);
    }

I apologize that I do not have a unit test for this.  Several of the unit tests for NbModuleSuiteTest fail on my machine (before applying the patch) and so I cannot create a new unit test with confidence.  I have tested this manually using the code above in conjunction with the minimal suite from #197066 and i4t seems to work fine.
Comment 3 Jaroslav Tulach 2011-04-07 10:20:45 UTC
Y01 Missing @since tag on new method
Y02 Missing test
Y03 Need a change in apichanges.xml

Otherwise ok, and when fixed in new patch, it can be applied. The trunk is currently closed for bugfixes only. Marian, can Tom get this in or does he need to wait for 7.1?
Comment 4 tomwheeler 2011-04-07 15:26:38 UTC
I will make all the changes you request, but out of curiosity, do all the tests in NbModuleSuiteTest pass for you?  Several of them fail on my machine and so I am not sure whether the tests are broken, I have a bad checkout or something's not right about my setup.
Comment 5 Jesse Glick 2011-04-07 16:16:04 UTC
(In reply to comment #4)
> do all the tests in NbModuleSuiteTest pass for you?

For me and for

http://deadlock.netbeans.org/hudson/job/NB-Core-Build/lastStableBuild/testReport/org.netbeans.junit/NbModuleSuiteTest/

> Several of them fail on my machine

Details?

> I have a bad checkout

You are first checking without your patch applied I hope?

hg version
java -version
ant -version
hg clean --all
hg up -c default
hg st
hg log -r .
ant -Dcluster.config=basic build-nozip
ant -f nbjunit/build.xml test

> or something's not right about my setup

If there is anything environmental known to cause test failures that ought to be caught by something other than the test failing.
Comment 6 tomwheeler 2011-04-07 17:47:29 UTC
Thanks, Jesse. I figured that the tests would have to pass on deadlock in order to not consider the build broken.  I did run the tests without my patch applied and they failed. I was in a hurry at the time and didn't write down any details.  

What I found was that the tests pass when run from the command line yet fail when run from the IDE.  I didn't want to pollute this issue with details about that, so I filed #197548 and documented them all in there.

I will make the three changes Jarda requested in the next few days and attach a patch.  I will verify my test works correctly by running it from the command line.
Comment 7 tomwheeler 2011-04-08 18:31:00 UTC
Created attachment 107619 [details]
Revised patch which satisfies Y01, Y02 and Y03

I have attached a revised patch which satisfies Y01, Y02 and Y03.  I am still having problems running tests, so I'd certainly recommend running the test before committing changes related to this patch.

While working on the test case, I found that command-line arguments meant for the JVM (i.e. -J-Xmx256m or -J-Dname=value) are not being properly interpreted when supplied for a test case using the addStartupArgument method.  Options like --branding or --userdir work properly, as evidenced by the test case.  A startup argument like "-J-Dname=value" yields a message "Ignored unknown option: -J-Dname=value" in the output, followed by "Post-initialization command-line options could not be run" and the system property is never set.

Although I don't currently need to specify JVM parameters at startup, it would be nice if that also worked.  If you can give me a hint as to what I should change to support that, I will try and create a new patch to handle it.  Otherwise, I'd be happy to have the attached patch integrated in the NB JUnit module if it meets with your approval.
Comment 8 Jaroslav Tulach 2011-04-12 08:47:04 UTC
Re. "-J parameters". The NbModuleSuite cannot change certain parameters of JVM as JVM is already running. To do that you need to influence the junit task. Luckily there is a way to incluence that: 

$ grep test.*arg */nbproject/project.properties 
o.n.core/nbproject/project.properties:test.run.args=-ea -XX:PermSize=32m -XX:MaxPermSize=400m -Xmx520m
performance/nbproject/project.properties:test.run.args=-da -Xmx384M

Otherwise the patch is OK (except missing change of version in manifest). I'll integrate it as soon as API changes are allowed.
Comment 9 tomwheeler 2011-04-12 17:26:46 UTC
Created attachment 107691 [details]
Revised patch which increments version number and updates documentation

Thanks, Jarda.  I have attached a new patch which has the same changes as before, plus an incremented version number.  I also updated the documentation slightly to mention the test.run.args property and to remove obsolete references to XTest.
Comment 10 Jaroslav Tulach 2011-05-14 15:24:17 UTC
Integrated as ergonomics#879d18f15152
Comment 11 Jaroslav Tulach 2011-05-14 15:25:29 UTC
*** Bug 197066 has been marked as a duplicate of this bug. ***


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo