This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 207737 - SFTP Keep alives seems to slow down upload if the server not support keep-alives
Summary: SFTP Keep alives seems to slow down upload if the server not support keep-alives
Status: RESOLVED DUPLICATE of bug 205437
Alias: None
Product: php
Classification: Unclassified
Component: FTP Support (show other bugs)
Version: 7.1
Hardware: All All
: P2 normal (vote)
Assignee: Tomas Mysik
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-01-25 14:36 UTC by michaelwidthfield
Modified: 2012-01-27 08:37 UTC (History)
3 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter:


Attachments
message.log (37.63 KB, text/plain)
2012-01-25 15:16 UTC, michaelwidthfield
Details
sftp 0s keep-alive, snapshot and log file (2.38 MB, application/gzip)
2012-01-26 12:31 UTC, Vladimir Riha
Details
sftp 30s (87.10 KB, application/gzip)
2012-01-26 12:33 UTC, Vladimir Riha
Details
ftp 30s (2.07 MB, application/gzip)
2012-01-26 12:34 UTC, Vladimir Riha
Details

Note You need to log in before you can comment on or make changes to this bug.
Description michaelwidthfield 2012-01-25 14:36:41 UTC
Since I installed the version 7.1.0 uploading files (On save) seems to be very slow and let's netbeans crash eventually. I tried with a Keep-alive interval set of 0 but this does not work.
When I switch back to 7.0.1 then it works (fast).
Comment 1 Tomas Mysik 2012-01-25 14:58:48 UTC
(In reply to comment #0)
> let's netbeans crash eventually.

Can you provide messages.log file? Also, paste here your NB and JDK version (see Help > About).

BTW what exactly is slow? NetBeans or the file transfer itself?

Just reopen this issue and provide requested information.

Thanks.
Comment 2 Tomas Mysik 2012-01-25 15:13:19 UTC
Láďo, please try to reproduce this issue if possible. Thanks.
Comment 3 michaelwidthfield 2012-01-25 15:16:57 UTC
Created attachment 115244 [details]
message.log

message.log
Comment 4 michaelwidthfield 2012-01-25 15:19:14 UTC
Hi thanks for the quick reply.

About info:

Product Version: NetBeans IDE 7.1 (Build 201112071828)
Java: 1.6.0_29; Java HotSpot(TM) Client VM 20.4-b02
System: Windows 7 version 6.1 running on x86; Cp1252; nl_NL (nb)
User directory: C:\Users\xxx\.netbeans\7.1
Cache directory: C:\Users\xxx\.netbeans\7.1\var\cache


> BTW what exactly is slow? NetBeans or the file transfer itself?

The transfer is slow and the after a while netbeans freezes.

The message.log is in the attachment.
Comment 5 Tomas Mysik 2012-01-25 15:35:25 UTC
I see out-of-memory error... What are the exact steps to reproduce this issue, please? How big is your project (how many files)? Is your project on a local disk or a network drive? What is your IDE memory usage [1]?

Thanks.
[1] Can be found in the main toolbar of NetBeans; if not visible, use View > Toolbars > Memory.
Comment 6 michaelwidthfield 2012-01-25 15:43:38 UTC
> I see out-of-memory error... What are the exact steps to reproduce this issue,
please? 

Right-click on source files in the project tree and then upload.

> How big is your project (how many files)? 

About 5000 files.

> Is your project on a local disk or a network drive? 

Local disk SSD drive so it should be fast.

> What is your IDE memory usage [1]?

About 320 MB
Comment 7 Tomas Mysik 2012-01-25 15:59:53 UTC
(In reply to comment #6)
> Right-click on source files in the project tree and then upload.

If you disable upload-on-save, is it better or not?

> About 320 MB

Seems to be quite low; what are the 2 numbers (x / y MB) please?

Please, try to:
- use the development version of NetBeans [1] and let us know if you can reproduce;
- using NB 7.1, attach profiler snapshot [2] (start profiling just before uploading and cancel it after about one minute or so (or before the out-of-memory error occurs, if possible)).

Also, it would be really great if you could have a look at [3] and try to find out what could be the cause of your issue.

The last thing: from the messages log file you have attached, I see that the file [4] should exist (it is a memory dump if out-of-memory error occurs) - can you please publish it somewhere and send me a private email with its URL?

Thanks a lot for your cooperation.
[1] http://bits.netbeans.org/download/trunk/nightly/latest/
[2] http://wiki.netbeans.org/GenerateThreadDump#Profile_Me_Now
[3] http://wiki.netbeans.org/FaqNetBeansAndOOME
[4] C:\Users\xxx\.netbeans\7.1\var\log\heapdump.hprof
Comment 8 Vladimir Riha 2012-01-26 12:31:45 UTC
Created attachment 115271 [details]
sftp 0s keep-alive, snapshot and log file

I can reproduce it with NB 7.1 on Win7 but I think it may not be limited to SFTP. I reproduced it with simple FTP as well and I don't think that setting keep-alive to 0 or some value changes anything. I'll attach snapshots and log files for SFTP 0s, SFTP 30s and FTP 30s tests. Going to test trunk build...
Comment 9 Vladimir Riha 2012-01-26 12:33:02 UTC
Created attachment 115272 [details]
sftp 30s
Comment 10 Vladimir Riha 2012-01-26 12:34:34 UTC
Created attachment 115273 [details]
ftp 30s
Comment 11 Tomas Mysik 2012-01-26 12:35:48 UTC
(In reply to comment #8)
> I can reproduce it with NB 7.1 on Win7

You mean the slowdown or the out-of-memory error?
Comment 12 Vladimir Riha 2012-01-26 12:43:06 UTC
Actually both, after some time it is getting slower and the out of memory occurs (I think in this order).

I've just noticed the reporter mentioned upload and I've tried only download during project creation. Should I repeat it with upload?
Comment 13 Tomas Mysik 2012-01-26 12:52:31 UTC
(In reply to comment #12)
> Actually both, after some time it is getting slower and the out of memory
> occurs (I think in this order).

In such case, we need a thread dump, see comment #7 for instructions.

> I've just noticed the reporter mentioned upload and I've tried only download
> during project creation. Should I repeat it with upload?

Perhaps it is not needed to try it with upload since the problem is already reproducible.

Thanks.
Comment 14 Tomas Mysik 2012-01-26 13:02:01 UTC
sftp 0s keep-alive:

In the log file, I see a huge number of IAE from progress. I have reported it as issue #207771.

I cannot see anything suspicious in the profiler log.

Láďo, please, reproduce the problem and create _heap dump_ (see comment #7).

Thanks.
Comment 15 Tomas Mysik 2012-01-26 13:05:13 UTC
Waiting for heap dump...
Comment 16 mariusvw 2012-01-26 14:18:51 UTC
I'm having the same issue with SFTP.
Netbeans gives a black screen when uploading large projects.

Tho it does finish the upload on some way.

I tried raising the memory limit which did not help. After changing the keep-alive value to 8 seconds Netbeans seems to upload the project without crashing but when you try to upload another project after that the log gives an error "... line too long".
Comment 17 Vladimir Riha 2012-01-26 15:17:51 UTC
Not reproducible in trunk with keep-alive set to 0 and 30s for both upload/download with SFTP. Both operation are way faster.

I've tried upload in NB 7.1 and the slowdown seems to be a lot smaller with upload then with download.
Comment 18 Tomas Mysik 2012-01-27 05:16:44 UTC
(In reply to comment #16)
> I'm having the same issue with SFTP.
> Netbeans gives a black screen when uploading large projects.

Can you provide a screenshot, please?

> the log gives an error "... line too long".

Which line? We need messages.log [1] or the output you have in the Output window otherwise we _really_ cannot help you, sorry.

One note: keep-alive for SFTP is provided by the SFTP library; keep-alive for FTP is provided by NetBeans itself (there is a bug in the FTP library we use).

IMPORTANT: If anyone can help to solve this issue, see comment #7 for instructions.

Thanks.
[1] from your <nb-userdir>/var/log/ directory
Comment 19 Tomas Mysik 2012-01-27 05:31:23 UTC
(In reply to comment #17)
> Not reproducible in trunk with keep-alive set to 0 and 30s for both
> upload/download with SFTP. Both operation are way faster.

That is really weird but I don't remember any changes since NB 7.1 (I can be wrong, of course).

> I've tried upload in NB 7.1 and the slowdown seems to be a lot smaller with
> upload then with download.

This could be caused by issue #207771 since it really takes some resources to write e.g. 30 MB to the log file...

However, yesterday I tried to reproduce this issue with trunk and creating a project (mediawiki, about 1900 files) took 555 seconds (the download); then, with upload-on-save, I made some changes (no problem occured) and the download of all source files took 549 seconds (so no difference, basically).

I will look at the thread dump Láďa has provided to me via email.

@michaelwidthfield: If you want to help us, please, follow the instructions in the comment #7. Otherwise we are not able to help you, I'm afraid.

Thanks.
Comment 20 Tomas Mysik 2012-01-27 08:12:01 UTC
From the heap dump I see that the memory error occurs in the Output window. The issue is already fixed and ready to be part of NB 7.1 patch1 I guess.

Thanks for reporting.

*** This bug has been marked as a duplicate of bug 205437 ***
Comment 21 Jaroslav Havlin 2012-01-27 08:37:33 UTC
There is a workaround for NetBeans 7.1.
Please, see http://wiki.netbeans.org/Bug205437inNB71
Sorry for the inconvenience.