This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
It would be wonderful if Synergy could generate a statistic page showing overall contribution of individuals in some way. I mean something like: Test run: NetBeans IDE 8.0 Beta readiness check Number of testers: 18 Contributed hours: 15.25 Executed test cases: 500 out of 2000 (25 %) Pass rate: 89 % Found bugs: 57 - link to Bugzilla Activity of tribes: # Tribe Passed Tested Time Testers 1. JavaFX 94 % 256 tc 7.1 hr 5 2. DB 95 % 112 tc 6.9 hr 6 3. etc. Activity of testers: # Name Tested Time 1. Lou Dasaro 75 tc 2.1 hr 2. Glenn Holmer 73 tc 2.0 hr 3. etc.
fyi data (time and issues) are now being saved while testing. Time is being saved only if Synergy is configured to track test cases duration (it is configured that way) Changes will be live in 5 minutes. What's left is to create statistics page
I've found case where I'm not sure what can be done. Imagine following scenario: - tester starts testing, reports issue and adds issue number in Synergy - tester pauses or finishes testing, issue is added to test case (so everyone can see it) and added to assignments stats => Synergy now knows that he added a new issue (because when submitting data, it wasn't in test case). It will be correctly showed in statistics. - but when user Restarts testing, again marks the same case as failed, provides the same issue number and submits results => Synergy now compares test case and submitted data and finds that issue has been already added to test case => assignments statistics no longer shows this issue as added by user I really don't know what to do #1 keep current state but it could resolve in lost information in statistics #2 keep track of all issues user added when marked case as failed, but it could show falsely positive data (issues not submitted by this user) #3 keep track of all issues user added and when computing statistics filter these issues with Bugzilla DB to compare reporter and bug creation time I would go with option #3, it would require some extra work but would give the best results. Right now, #1 is used.
#3 is of course ideal but if we continue with #1 it's not a big issue since I assume that restarts will happen very rarely.
I agree, #3 is best, others would cause confusion. I'll finish it by asap
first (incomplete) draft is available at http://services.netbeans.org/synergy/client/app/#/statistics/4 (link is also on test run page). TODO: bugs and tribes
Jirko please check [1], what do you think? Issues and times are not available in this run (will be in next one). There is still some work left to do: - should I count only testers who completed at least 1 test case? - do you want to see zeros in cells or rather empty cells? - tribes statistics are a bit off. The reason is tester can be in multiple tribes => his/her statistics are used in multiple tribes. It does make sense if you focus on people in tribes, but sum of numbers in tribes table won't match sum numbers of test runs. - if tester is moved/removed from tribe, tribes table will be different. One solution would be to save tribe information to assignment. This _would have to_ be done before 8.0 testing. Should I do it? - I need to create a test run to fully test issues. Do you want to also show e.g. priorities of reported issues (available information is bugId, priority and reporter)? Synergy searches for issues reported from [testRun start date - 1 day] to [testRun end date +1 day], hopefully I'll be able to match timezone used in Synergy and bugzilla DB (might be the same, I don't know right now) [1] http://services.netbeans.org/synergy/client/app/#/statistics/4
I just noticed "testers" column in tribes table is actually number of assignments, will be fixed asap.
Lado, I very much like it! My few comments: 1. Once we have times for both testers and tribes tables, can the entried be sorted by the time not number of completed test cases? 2. I do not think it's necessary to show testers who didn't complete any test case. 3. It would be great, if you could provide the bug information you suggested and visualize by color severity of the failures. E.g.: total number of reported bugs (the link to Bugzilla) would have green background if all reported bugs are P3 or P4 issues, yellow if there is one P2 issue, orange if there are more P2 issues and red if there is P1 issue. 4. Yes, please save the tribe information to the assignment entry to ignore later drafts of testers between tribes, which will occur rarely anyway.
(In reply to Jiri Kovalsky from comment #8) > Lado, I very much like it! My few comments: > > 1. Once we have times for both testers and tribes tables, can the entried be > sorted by the time not number of completed test cases? > 2. I do not think it's necessary to show testers who didn't complete any > test case. > 3. It would be great, if you could provide the bug information you suggested > and visualize by color severity of the failures. E.g.: total number of > reported bugs (the link to Bugzilla) would have green background if all > reported bugs are P3 or P4 issues, yellow if there is one P2 issue, orange > if there are more P2 issues and red if there is P1 issue. Fixed, please check http://services.netbeans.org/synergy/client/app/#/statistics/4/ (to see colored issue box, you might have to reload page using Ctrl+R) > 4. Yes, please save the tribe information to the assignment entry to ignore > later drafts of testers between tribes, which will occur rarely anyway. I implemented the JSON solution. Every time a test run is closed (freezen), Synergy dumps all information required for the statistics page to JSON file called statistics.json. If test run is closed, link to statistics changes to http://services.netbeans.org/synergy/client/app/#/statistics/4/archive (was http://services.netbeans.org/synergy/client/app/#/statistics/4) and the page loads static JSON file instead of calling Synergy API (both links works, but the one without "archive" shows live data). If you "unfreeze" test run and freeze it again, old JSON is renamed to statistics.jsonX (where X is number) to be preserved and a new statistics.json is created. Unfortunately Synergy cannot write (permission issue) to directory used for this. I'll ask Honza about it and about setting up cron job to push everything from this data directory to repository.
Great Lado! I think we are done with this one provided that times and bugs will be recorded for the next test run. Thanks for implementing!
I'm glad to help. I would let it open until issues with permissions and cron job (see comment #9) are resolved.
(In reply to Vladimir Riha from comment #11) > I'm glad to help. I would let it open until issues with permissions and cron > job (see comment #9) are resolved. In the meantime, I've created fallback if the JSON file (created when someones freezes test run) is not found to look somewhere else (for file committed to repository by someone - so this file is not created automatically). Closing as fix for now.
Happily verifying. Thanks again.