This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 235114 - Suggestion about automatic copying for remote development
Summary: Suggestion about automatic copying for remote development
Status: STARTED
Alias: None
Product: cnd
Classification: Unclassified
Component: Remote (show other bugs)
Version: 7.3.1
Hardware: PC Windows 7
: P2 normal (vote)
Assignee: Maria Tishkova
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-08-27 16:42 UTC by john_ousterhout
Modified: 2014-07-21 09:20 UTC (History)
1 user (show)

See Also:
Issue Type: ENHANCEMENT
Exception Reporter:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description john_ousterhout 2013-08-27 16:42:41 UTC
I use NetBeans for remote C++ development (IDE on Windows, apps run on remote Linux). Overall, I'm a huge fan; NetBeans is easily the best C++ development environment I've ever seen.

However, I have had issues with automatic copying mode, and have ended up developing a quirky development pattern around the problems. I'm wondering if an alternative approach to automatic copying would end up with a faster and more useful system. Right now, I have two problems. The biggest problem is performance: builds are significantly slower when run under NetBeans than if I run them directly on the remote machine. In particular, the link step for my application (involving 50-100 object files) takes 5-10 minutes under NetBeans, vs. 5-10 seconds when run directly on the build/test machine. This is such a big penalty that I have to hack around it: I start the build under NetBeans, then once it reaches the link step I go to a separate ssh window on the build machine and run a script that kills the link step and reruns the make from that window. Even with this hack, builds still take longer than if I ran them purely native (there's a 10-20 second delay at the start of each build). 10-20 seconds may not seem like much, but when you do hundreds of builds a day, those little bits add up.

The second problem I have is that automatic copying doesn't always happen perfectly, and I occasionally end up in situations where some of the files needed on the target machine are not present (actually, they are present, but their contents are all zeroes). Unfortunately, I don't yet have a reproducible test case that makes this happen. Once this happens, it's hard to get the files properly sync-ed again; after trying several approaches, I've decided that the easiest thing is just to manually rsync the entire project tree from the IDE machine to the build/test machine (I've tried the "Upload to ..." menu entries from the IDE, but haven't found that these always do what I need).

My understanding is that the current approach hooks all of the build tools running on the build machine to catch file accesses, and then does the copies on-demand. This is elegant, but I'm wondering if a much simpler (and way less elegant) approach might actually work out better overall. Suppose that the hooking and on-demand copying were eliminated, and instead you just did the following:

* Before running any command on the remote machine for anything, copy over any files that have been changed in the IDE. Really copy the files: no placeholders with zero content, so it wouldn't be necessary to hook the build tools.

* Offer a "sync all" option in the IDE to force a copy of the entire workspace (I've seen something like this in the current IDE, but it doesn't appear to be totally predictable in what it does). People would use this, for example, if they modify files outside the IDE. Perhaps NetBeans could check for modifications to any files in the project workspace and copy those as well, which would make "sync all" almost never needed.

I think this approach would make builds way faster: it would eliminate the slow links, and I suspect it would also reduce the lags at the start of builds. In most cases, very few files are modified, so copying entire files should be fast.

In addition, this would simplify the mechanism in NetBeans, which would probably make it much easier to maintain.

I realize this is a non-trivial suggestion, but I thought I would throw it out there anyway; I'd love to see NetBeans become even more awesome than it already is!
Comment 1 Maria Tishkova 2013-08-29 10:34:40 UTC
Hello,

We are glad you like NetBeans IDE and use it for your development. 


Can you clarify the way you do use the IDE now:

Where do you have your source files? What is the scenario of "remote" build. Do you have Windows with source files and 1,2,.. remote machines you build your sources?

If there is a chance you can put your files on remote machine I would suggest you to use "Full Remote" + Sharing mode  feature instead of automatic copying.
Full remote mean s you have all your source  files on remote machine and they are copied on Windows to provide code model.

P.S. I will log it as an enhancement not a defect and we will consider your suggestion when we will plan next NetBeans release:)

==Maria
Comment 2 john_ousterhout 2013-08-29 15:28:29 UTC
I keep the official copy of my source files on a Windows laptop running Windows 7. I do my builds and executions on a single Linux server, using "automatic copying" mode.

I have tried using "Full Remote" + Sharing mode, but this doesn't work very well for me. The file sharing mechanisms between Windows and Linux are flakey at best, and even when that works, I found that the response is very slow, especially when I'm working from home with a VPN between my Windows laptop and the Linux NFS server. Thus, at least in my experience so far, automatic copying mode has worked much better than full remote mode, in spite of its issues.
Comment 3 jdavidbakr 2014-06-27 15:22:47 UTC
I wanted to resurrect this as while the new copying in 8.0 is significantly better, I still get an occasional file that doesn't get uploaded and I have to manually upload it (after several minutes of not understanding why my change didn't compile) - I would love to see the ability to use rsync to make your local copy the master and have the remote copy always be a mirror, i.e. I never care about downloading anything from the remote server.
Comment 4 Vladimir Kvashin 2014-07-21 09:20:32 UTC
First of all, I'd like to note that starting from 8.0 a new SFTP way of synchronization is available. It is more reliable than Auto Copy; I would recommend to try it.

Supporting rsync is definitely a good idea; however I'm not sure I'll be in time to add rsync support in 8.01

Probably the concept should be a bit wider: adding an ability to use an external tool for synchronization, with some simple way of configuring this tool.