This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 227109 - Local Synchronization takes a very long time
Summary: Local Synchronization takes a very long time
Status: VERIFIED FIXED
Alias: None
Product: php
Classification: Unclassified
Component: Code (show other bugs)
Version: 7.3
Hardware: All All
: P1 normal with 4 votes (vote)
Assignee: Tomas Mysik
URL:
Keywords:
: 227134 (view as bug list)
Depends on:
Blocks:
 
Reported: 2013-03-06 15:02 UTC by prembo
Modified: 2013-05-21 08:44 UTC (History)
8 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter:


Attachments
No progress indicated on Local Synchronization activity bar. (26.73 KB, image/jpeg)
2013-03-06 15:02 UTC, prembo
Details
IDE Log (63.69 KB, application/octet-stream)
2013-03-06 20:30 UTC, prembo
Details
IDE Profiler snapshot - when a large project is added (95.17 KB, application/octet-stream)
2013-03-06 23:44 UTC, prembo
Details
IDE Profiler snapshot - when a large project is added (1.33 MB, application/octet-stream)
2013-03-06 23:46 UTC, prembo
Details
profiler snapshot (312.08 KB, application/octet-stream)
2013-03-28 09:38 UTC, bazi
Details
snapshot from 7.2.1 (216.85 KB, application/octet-stream)
2013-05-13 12:47 UTC, Vladimir Riha
Details
snapshot from 7.3 after delete (5.83 MB, application/octet-stream)
2013-05-13 12:51 UTC, Vladimir Riha
Details
IDE log from 7.3 where copied sources remained after delete (79.06 KB, application/octet-stream)
2013-05-13 12:53 UTC, Vladimir Riha
Details
7.2.1 with -J-Dorg.netbeans.modules.php.project.copysupport.level=FINE (35.47 KB, application/octet-stream)
2013-05-14 12:38 UTC, Vladimir Riha
Details
7.3 with -J-Dorg.netbeans.modules.php.project.copysupport.level=FINE (101.67 KB, application/octet-stream)
2013-05-14 12:39 UTC, Vladimir Riha
Details

Note You need to log in before you can comment on or make changes to this bug.
Description prembo 2013-03-06 15:02:07 UTC
Created attachment 132272 [details]
No progress indicated on Local Synchronization activity bar.

Product Version: NetBeans IDE 7.3 (Build 201302132200)
Java: 1.6.0_41; Java HotSpot(TM) 64-Bit Server VM 20.14-b01-445
Runtime: Java(TM) SE Runtime Environment 1.6.0_41-b02-445-11M4107
System: Mac OS X version 10.7.5 running on x86_64; MacRoman; en_US (nb)
User directory: /Users/theuser/Library/Application Support/NetBeans/7.3
Cache directory: /Users/theuser/Library/Caches/NetBeans/7.3

PROBLEM:
After Netbeans 7.3, it completes "Background scanning of projects" (~90seconds), and then gets 'stuck' on "Local Synchronization" for a very long time ~ at least 15-20 minutes.  No progress is indicated on the progress bar.  CPU usage is very high (50-120%).  It seems to do it each time I start up NB7.3.

BACKGROUND:
I just installed NB7.3 and it imported approximately 20 projects across from NB7.2 (which is still installed).  The projects are all PHP.  I have MAMP running with PHP 5.4.4 and 5.2.17.

Most projects Mercurial version controlled with subrepositories.  There are some Git repositories also.  All projects reside on the local hard disk.  Most projects are fairly large CMS projects (> 2000 files).  Each project copies across to a web server on the local machine.

In previous NB upgrades, all imports/initialisation completed quickly (<5 mins).  Also, NB7.2 starts to a completely usable state (all initialisation completed) within 2-3 mins.
Comment 1 prembo 2013-03-06 20:30:40 UTC
Created attachment 132294 [details]
IDE Log
Comment 2 prembo 2013-03-06 23:25:09 UTC
I removed all the projects and started to add them gradually.  I noticed that each project took about 70-120 seconds in the Local Synchronization stage.  

Could it be that the the IDE isn't caching the file status and is having to reindex all files of each project each time it's opened?

As expected, unchecking the "Copy files from Sources Folder to another location" made the problem disappear. 

I can confirm that it isn't related to Mercurial subrepositories.  I measured the time taken on another large 
 PHP project (~4000-5000 files) without (nested) Mercurial subrepositories and it took just as long.

Small projects (<100 files) take ~ 5 seconds to do Local Synchronization.  

Understandably, the larger the project the longer it takes, but it just doesn't happen in 7.2.1.  The IDE loads up quickly - we're in business within 2 minutes.  On 7.3, it brings the IDE to a standstill as it does the Local Synchronization.

Hope this info helps.
Comment 3 prembo 2013-03-06 23:44:37 UTC
Created attachment 132300 [details]
IDE Profiler snapshot - when a large project is added

Adding a large projects demonstrates the issue described in this bug report. Ie, same as opening the IDE with the large project already in the workspace. 

Interesting to note that the profiler indicates that the IDE is spending most of its time waiting! 

Hope this helps.
Comment 4 prembo 2013-03-06 23:46:22 UTC
Created attachment 132302 [details]
IDE Profiler snapshot - when a large project is added
Comment 5 LucitheR 2013-03-22 11:01:31 UTC
I can confirm this. takes forever and works only after clicking the "do not use it" Dialog as OK. 


Mac OS X, Netbeans 7.3 with
java version "1.7.0_10-ea"
Java(TM) SE Runtime Environment (build 1.7.0_10-ea-b16)
Java HotSpot(TM) 64-Bit Server VM (build 23.6-b04, mixed mode)
Comment 6 Tomas Mysik 2013-03-22 11:13:22 UTC
@vriha, @kanakmar: Please, anyone of you, try to reproduce this issue (not only on Mac OS X).

Thanks.
Comment 7 Martin Kanak 2013-03-26 10:58:39 UTC
I try to reproduce this issue but it works for me on Mac OS X 10.7.5, Windows 8, Ubuntu 12.04 with the latest dev build. I used mediawiki as a project, because I don't have any other large project => "Local synchronization" doesn't get stuck after opening IDE.

Product Version: NetBeans IDE Dev (Build 201303252300)
Java: 1.7.0_17; Java HotSpot(TM) Client VM 23.7-b01
Runtime: Java(TM) SE Runtime Environment 1.7.0_17-b02
Comment 8 Tomas Mysik 2013-03-26 11:15:31 UTC
@kanakmar: Martine, just a hint for the next time - to have a "big" project, simply copy mediawiki files several times to the project Source Files ;)

I will look at the profiler snapshots. It would be great if anyone who can reproduce this issue could create 2 profiler snapshots - one in NB 7.2 and one in NB 7.3, preferably with a clean userdir [1].

Thanks a lot for your cooperation.
[1] http://wiki.netbeans.org/FaqAlternateUserdir
Comment 9 bazi 2013-03-28 09:37:41 UTC
I had this issue many times but didn't know the cause of it. As it seems here, it is related to PHP projects. I uploaded profiler snapshot in attachments.

When issue began, there were ~10 projects one of which was php. After restarting IDE, Local synchronization started again. Closing php  project and restarting IDE didn't start Local Sync.

Product Version: NetBeans IDE 7.3 (Build 201302132200)
Java: 1.7.0_10; Java HotSpot(TM) 64-Bit Server VM 23.6-b04
Runtime: Java(TM) SE Runtime Environment 1.7.0_10-b18
System: Windows 7 version 6.1 running on amd64; Cp1251; en_RU (nb)
Comment 10 bazi 2013-03-28 09:38:46 UTC
Created attachment 133161 [details]
profiler snapshot
Comment 11 feeela 2013-04-02 07:45:07 UTC
The local didn't really get stuck but always transfers all files on startup (using NB 7.3 on Ubuntu 12.04), even if there are no changes to the files that are already exist in the target directory. When developing for large projects (e.g. Magento) it usually takes more than 20 minutes (whereas a manual rsync only lasts a few minutes).

This is extremely annoying if the target directory is a mounted network device. The repeating local synchronization then fully blocks the local network.
Comment 12 funkygibbon 2013-04-28 01:15:51 UTC
Since the upgrade to 7.3 from 7.1, I'm experiencing network synchronisation times in excess of an hour for a single Magento project, when it would take less than a minute in 7.1

As this is a critical bug which renders Netbeans entirely unusable, is there any timeline for resolution?
Comment 13 funkygibbon 2013-04-28 01:35:47 UTC
Network connection repeatedly times out attempting to install the profile plugin (no other network problems indicated).

Product Version: NetBeans IDE 7.3 (Build 201302132200)
Java: 1.7.0_21; Java HotSpot(TM) Client VM 23.21-b01
Runtime: Java(TM) SE Runtime Environment 1.7.0_21-b11
System: Windows 7 version 6.1 running on x86; Cp1252; en_AU (nb)
Comment 14 Tomas Mysik 2013-05-13 07:05:38 UTC
(1) Could anyone please attach a profiler snapshot from NetBeans 7.2? So we have exact comparison.

(2) Do you use any SCM (hg, git, ...) for your projects? Could you please try to open "versioned" project and then "non-versioned" project? Do you see any difference?

Thanks.
Comment 15 Tomas Mysik 2013-05-13 07:10:49 UTC
From the snapshots, it seems to me that the problem could be in FS itself (copying and deleting) - the whole time is spent there. Jardo, please could you have a look at the snapshots? PHP support did not change in this area and maybe you will see something suspicious there...

Thanks a lot.
Comment 16 Tomas Mysik 2013-05-13 08:28:24 UTC
*** Bug 227134 has been marked as a duplicate of this bug. ***
Comment 17 Tomas Mysik 2013-05-13 08:41:25 UTC
Snapshot for the same project can be found in issue #227134. For NB 7.2.1, there is no copying and deleting visible in the snapshot; for NB 7.3, it takes 136 seconds (61 seconds for delete, 58 seconds for copy).

(BTW as a fix for issue #226605, I just added a checkbox for copying files on project open so it could help as a workaround with this issue.)

Thanks.
Comment 18 Vladimir Riha 2013-05-13 08:51:25 UTC
I tried project of size of 2 mediawikis (~5600 files of total 147MB) on Windows 7. I always created new project from sources, set up local sync and wait for it to finish, then restarted IDE:
- 7.3: on IDE restart, local sync started, after about 14minutes it was still in progress
- 7.2: on IDE restart, there is no local sync at all

I haven't used any SCM
Comment 19 Vladimir Riha 2013-05-13 12:01:06 UTC
I tried to delete sources for the projects mentioned in my previous post.

NB 7.2 - action finished in ~6min
NB 7.3 - action finished in ~6min 24sec

but in case of 7.3, I got a lot of times dialog "Copy failed" (or something like that), in IDE log, there are messages like

Parent does not exist C:\Users\tester\Downloads\mediawiki-1.20.5.tar - Copy (2)\mediawiki-1.20.5\tests\phpunit\languages
Parent exists: C:\Users\tester\Downloads\mediawiki-1.20.5.tar - Copy (2)\mediawiki-1.20.5\tests\phpunit
Has children [bootstrap.php, install-phpunit.sh, maintenance, Makefile, MediaWikiLangTestCase.php, MediaWikiPHPUnitCommand.php, MediaWikiTestCase.php, phpunit.php, README, run-tests.bat, skins, StructureTest.php, suite.xml, suites, TODO]

and as a result, most of the files folder where it was supposed to be copied, remained there although sources were removed.

I'll attach profiler snapshots asap
Comment 20 Vladimir Riha 2013-05-13 12:47:46 UTC
Created attachment 134362 [details]
snapshot from 7.2.1
Comment 21 Vladimir Riha 2013-05-13 12:51:35 UTC
Created attachment 134363 [details]
snapshot from 7.3 after delete
Comment 22 Vladimir Riha 2013-05-13 12:53:04 UTC
Created attachment 134364 [details]
IDE log from 7.3 where copied sources remained after delete
Comment 23 spye 2013-05-14 11:27:06 UTC
I can confirm that this happens for me on a Windows 8 installation as well.

Product Version: NetBeans IDE 7.3 (Build 201302132200)
Java: 1.7.0_17; Java HotSpot(TM) 64-Bit Server VM 23.7-b01
Runtime: Java(TM) SE Runtime Environment 1.7.0_17-b02
System: Windows 8 version 6.2 running on amd64; Cp1252; en_CA (nb)
User directory: C:\Users\spye\AppData\Roaming\NetBeans\7.3
Cache directory: C:\Users\spye\AppData\Local\NetBeans\Cache\7.3
Comment 24 Vladimir Riha 2013-05-14 12:38:52 UTC
Created attachment 134416 [details]
7.2.1 with -J-Dorg.netbeans.modules.php.project.copysupport.level=FINE

I did:
- started IDE, created some project and set up local sync
- waited for synchronization to finish
- restarted IDE
- saved IDE log
Comment 25 Vladimir Riha 2013-05-14 12:39:26 UTC
Created attachment 134417 [details]
7.3 with -J-Dorg.netbeans.modules.php.project.copysupport.level=FINE

Same steps as for 7.2.1 above
Comment 26 Tomas Mysik 2013-05-15 07:46:27 UTC
The cause of copying files on project open is the fix of issue #214888. So, we have added a checkbox "Copy files on project open" which is unchecked by default. But please notice that the copied files can be out of date (if one e.g. invokes "git pull" in the project and NetBeans is not running) and unfortunately, there is no way to find this situation out.

@vriha: Láďo, please verify the checkbox functionality and mark this issue as a patch candidate.

Thanks.

http://hg.netbeans.org/web-main/rev/048df7f8373c [1]
http://hg.netbeans.org/web-main/rev/36251d76c9c2

[1] This is fix of issue #226605 which is related with this one.
Comment 27 Vladimir Riha 2013-05-15 08:43:09 UTC
verified in web-main

- checkbox is by default unchecked, once one checks it and confirm project properties dialog, sources are synced
- if checkbox is checked on startup, sources are synced as expected



Product Version: NetBeans IDE Dev (Build web-main-10523-on-20130515)
Java: 1.7.0_21; Java HotSpot(TM) Client VM 23.21-b01
Runtime: Java(TM) SE Runtime Environment 1.7.0_21-b11
System: Linux version 3.2.0-41-generic-pae running on i386; UTF-8; en_US (nb)
Comment 28 Ondrej Brejla 2013-05-15 08:45:07 UTC
Patch seems to be ok for me.
Comment 29 Quality Engineering 2013-05-16 02:28:34 UTC
Integrated into 'main-golden', will be available in build *201305152300* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/36251d76c9c2
User: Tomas Mysik <tmysik@netbeans.org>
Log: Related to #227109 - Local Synchronization takes a very long time

Copying files on project open is off by default.
Comment 30 Tomas Mysik 2013-05-16 06:30:45 UTC
Just found out that the new project wizard was not properly saved - fixed now.

Thanks.

http://hg.netbeans.org/web-main/rev/3ccbb549d08e
Comment 31 Tomas Mysik 2013-05-16 06:44:56 UTC
Changesets transplanted to release73 branch of releases repo. Spec version of php.project module increased as well.

@vriha: Láďo, please verify:
- changing project properties (local copying)
- creating new project (local copying)
- sanity check of upload-on-save (it shares some code with local copying)

Thanks.

http://hg.netbeans.org/releases/rev/d1882e764669
http://hg.netbeans.org/releases/rev/cacf688f4f81
http://hg.netbeans.org/releases/rev/cd409d3eb305
http://hg.netbeans.org/releases/rev/d9783110a9fa
Comment 32 Quality Engineering 2013-05-17 00:04:51 UTC
Integrated into 'releases', will be available in build *201305162200* or newer. Wait for official and publicly available build.
Changeset: http://hg.netbeans.org/releases/rev/cacf688f4f81
User: Tomas Mysik <tmysik@netbeans.org>
Log: Related to #227109 - Local Synchronization takes a very long time

Copying files on project open is off by default.
(transplanted from 36251d76c9c263bfd62952f30ae59507d60882f1)
Comment 33 Quality Engineering 2013-05-17 09:51:16 UTC
Integrated into 'main-golden', will be available in build *201305170640* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/3ccbb549d08e
User: Tomas Mysik <tmysik@netbeans.org>
Log: #227109 cont'd - Local Synchronization takes a very long time

Save new project wizard properly.
Comment 34 Vladimir Riha 2013-05-21 08:44:37 UTC
verified in 7.3.1

Product Version: NetBeans IDE 7.3.1 RC (Build 201305202200)
Java: 1.7.0_21; Java HotSpot(TM) Client VM 23.21-b01
Runtime: Java(TM) SE Runtime Environment 1.7.0_21-b11
System: Linux version 3.2.0-43-generic-pae running on i386; UTF-8; en_US (nb)