Bug 176121 - Remote Sources Improvement: Download changed files
Remote Sources Improvement: Download changed files
Status: VERIFIED FIXED
Product: php
Classification: Unclassified
Component: FTP Support
6.x
All All
: P2 with 3 votes (vote)
: 6.x
Assigned To: Tomas Mysik
issues@php
:
: 174518 (view as bug list)
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2009-11-08 14:15 UTC by dominicgamble
Modified: 2010-01-26 08:56 UTC (History)
4 users (show)

See Also:
Issue Type: ENHANCEMENT
:


Attachments
Screenshot of how a project is set up to use source files on a remote machine. (20.27 KB, image/png)
2009-11-12 15:47 UTC, dominicgamble
Details
screenshot showing the context menu that allows you to upload files to the remote server, or download them to your local machine. If you have 'upload on save' checked in the config, its automatic (7.79 KB, image/png)
2009-11-12 15:52 UTC, dominicgamble
Details

Note You need to log in before you can comment on or make changes to this bug.
Description dominicgamble 2009-11-08 14:15:23 UTC
The feature which allows you to use an SFTP connection to download sources and upload on save is brilliant and makes working from multiple remote locations possible.

One issue with it that can be quite painful is that if you work directly with the source code some of the time, to copy those changes to your remote netbeans project, you have to download the whole lot again. My project has a large number of small files and can take hours to download the whole lot from my work PC to my home PC.

Instead of this, netbeans could store the modification time and size of each remote file, and then the user could run a 'synchronize' command through the IDE. Initially it could list the changed files and allow you to download only those.

This could be expanded in the future to handle conflicts (where files have changed both locally and remotely), but as a first step being able to download only changed files would be such a great improvement.

Not only is this important to save on waiting hours for things to download, conflict resolution could also save you from overwriting files which you failed to upload, and losing work.
Comment 1 Jesse Glick 2009-11-12 15:09:53 UTC
I have no idea what feature you are referring to but it definitely does not exist in the basic project system. Do you use some third-party module? Sounds like what you want is a version control system anyway.
Comment 2 dominicgamble 2009-11-12 15:47:55 UTC
Created attachment 90954 [details]
Screenshot of how a project is set up to use source files on a remote machine.
Comment 3 dominicgamble 2009-11-12 15:52:12 UTC
Created attachment 90955 [details]
screenshot showing the context menu that allows you to upload files to the remote server, or download them to your local machine. If you have 'upload on save' checked in the config, its automatic
Comment 4 dominicgamble 2009-11-12 16:23:48 UTC
Hi Jesse,

Thanks for your response on this. I may not have explained myself well enough so I'm sorry about that. I've attached two screenshots which show the functionality that is actually part of the base netbeans install - I haven't installed any plugins. The version of netbeans I downloaded was the 'PHP bundle' but this functionality is not PHP specific, and would apply to any language so I would imagine its a core feature.

You can see in the configuration it's set to upload files on save.

You said 'Sounds like what you want is a version control module anyway'.

We do use subversion for version control, but this is done on my development machine, not through netbeans. The reason version control is not suitable at this level, is that during development I need to make sure that each time I save the file, it's uploaded to the development machine instantly and then I can test it in a web browser. I don't want to check a file in before testing, as it would mean that I would have to commit code to the repository which would be broken and untested, and it would be very slow to do so anyway. Imagine having to right click to commit and then write a comment every time you make a small change. Right now I just hit ctrl-s on your keyboard and then reload in my web browser - the way it should be.

The remote sources solution is the perfect solution, and yes it does have some similarities to version control (copying files to a remote location, synchronising/merging differences...hopefully!!) but its for an entirely different purpose. Version history and rollbacks are not required. All this does is allow a developer to work from remote locations still using the one development server to test and run his application, without having to set up a development environment at each location to do testing. In my current setup I actually run a windows machine which runs netbeans, and I have a linux virtual machine without X installed or any gui, which runs apache, php, postgresql and so forth. I don't want to set up my windows machine with all of that stuff, especially because php on windows has some differences with linux which is where the production environment runs.

I have used the 'remote system explorer' plugin in eclipse which also allows you to work remotely, but because it doesn't download and upload sources into a local project, it can't scan project files and do things like code hinting, because it would be too slow. Netbeans is a much better product and as soon as this 'download remote sources' and 'upload on save' via SFTP feature came out I jumped on it. Now I want it to synchronize as well. It would polish this feature off and make netbeans the ultimate choice.

I also have another feature request which I'll put through once this one gets implemented: Download remote sources as a single compressed file. Because my project has 20,000+ small files, it can take hours to download over a 2mbit connection because of the overhead of the sftp protocol. If the initial download could connect via ssh and create a tar'ed and gzipped stream to download the data it would make this initial download take a fraction of the time (source code compresses extremely well too). 

For now though, it's more important that we get this synchronize feature in here, because of the data loss issue. I have lost work a few times by overwriting which is pretty frustrating.

Thanks for your attention to this Jesse, and if there's any way to get a few people behind this particular feature request and make it happen then I would appreciate your input.
Comment 5 Petr Dvorak 2009-11-13 02:53:16 UTC
This is a PHP feature - no need for VCS there, usually you just need to synchronize with FTP (VCS's are sometimes too heavyweight for average size PHP projects).
Comment 6 Tomas Mysik 2009-11-13 03:11:45 UTC
(In reply to comment #5)
> This is a PHP feature - no need for VCS there, usually you just need to
> synchronize with FTP (VCS's are sometimes too heavyweight for average size PHP
> projects).

IMHO VCS should always be _strongly_ preferred (if possible).
Comment 7 dominicgamble 2009-11-16 04:50:06 UTC
A quick and easy way to get this functionality could be to use RSYNC.
Comment 8 dominicgamble 2009-11-18 18:30:47 UTC
After a little more research, I found that RSYNC is no good, because it does one way synchronization and doesn't handle conflicts.

I found that unison does two way synchronization and has conflict resolution. It's not written in java though - I think it's in c.

http://www.cis.upenn.edu/~bcpierce/unison/

It's pretty impressive and does everything I need, so I might use it to manually sync the local and remote source trees and just use netbeans for "upload on save". 

It would have been nice to find something written in java so that you guys could integrate it into netbeans. All I could find so far is JFileSynchronizer, which doesn't seem to be able to synchronize over a LAN, but instead runs slow and exhaustive byte-by-byte comparisons of entire directories.

http://jfilesync.sourceforge.net/

I didn't read much more than that. It looks like what would be ideal here is for someone to port unison to java and build a gui dialog for the conflict resolution.

Anyway thanks for exploring this with me guys, and I hope one day it happens.
Comment 9 ohlavacek 2009-12-21 08:43:29 UTC
Other solution that would be sufficient: after opening a file netbeans could check the file on the remote server and if it got changed (timestamp probably), user would be noticed and the new file downloaded. If there were a conflict, the file could be downloaded with a different filename and the user would merge it manually. 

Either this or Donwload changed files would be really great. This is the only thing we're missing in NB. Setting up these things in eclipse is a real struggle, but it has the Refresh (F5) function on files from (remote) filesystems.

Upload on save is a great function, but you need a bit more if you share development among more people.
Comment 10 Tomas Mysik 2009-12-21 15:56:44 UTC
(In reply to comment #9)
> Other solution that would be sufficient: after opening a file netbeans could
> check the file on the remote server and if it got changed (timestamp probably),
> user would be noticed and the new file downloaded. If there were a conflict,
> the file could be downloaded with a different filename and the user would merge
> it manually. 

Checking timestamp is tricky because it's not reliable (server can change the time, or it is common to have one hour time difference). The same applies for size, it can differ as well (because of new lines).

> Upload on save is a great function, but you need a bit more if you share
> development among more people.

I agree, you need a version control system :)
Comment 11 ohlavacek 2009-12-22 04:07:49 UTC
(In reply to comment #10)
> (In reply to comment #9)
> > Other solution that would be sufficient: after opening a file netbeans could
> > check the file on the remote server and if it got changed (timestamp probably),
> > user would be noticed and the new file downloaded. If there were a conflict,
> > the file could be downloaded with a different filename and the user would merge
> > it manually. 
> 
> Checking timestamp is tricky because it's not reliable (server can change the
> time, or it is common to have one hour time difference). The same applies for
> size, it can differ as well (because of new lines).

I agree, but from my point of view having redundant file downloads is better than having missing file downloads. I guess Eclipse uses timestamps (or something similar) as I got to refresh files more often than they are actually changed. This is OK, much better than rewriting with an older version during save without noticing. 

> > Upload on save is a great function, but you need a bit more if you share
> > development among more people.
> 
> I agree, you need a version control system :)

We use VCSs, but the situation is more complicated. Our dev server is pretty far away (and without chance to move it anywhere nearer) and using Samba to access the files is just a no-go. Saving the file takes seconds, git over Samba takes minutes, SVN over Samba takes hours (speaking about hundreds and thousands of files). Indexing the source files over Samba takes almost eternity (actually it is faster than SVN ;). git and SVN could be used through ssh, but not the indexing. So I keep the source files on local drive and upload them on save to the server, where it is versioned. And there, some projects do not have a branch for every user (SVN repos), so users work in a shared directory and repo. It would be nice, if there is a way to check automatically changed files on the remote side.
Comment 12 berniesaurus 2010-01-02 04:20:41 UTC
This feature would be great.

Zend Studio already have this feature in the latest release (7.1 - http://www.zend.com/en/products/studio/studio-whats-new#remote ), and much more concerning this task (diff, ...), I've tried this, and it almost work correctly but the "upload on save" feature is buggy (like so much things in ZS :) ) and the changes are not transferred to the remote server.

I think this feature will help many of the PHP developers out there.

I've already tried to deal this tasks with a VCS (git) and a local copy but for only 2 developers this is a huge amount of wasted time in administrative tasks (merge, ...).

Currently I have to download always the whole state from remote, which takes about 1,5 hours (~5900 files), this is very annoying. 

I think it isn't necessary to have a solution like in Zend Studio (which not works :) ), but the following 2 features would be great, and will save a lot of time: 

- Download only modified files
- Notify at save when the remote file is newer than the last "checked out" state of the file (a simple diff would be great :) )
Comment 13 Tomas Mysik 2010-01-25 09:05:27 UTC
*** Bug 174518 has been marked as a duplicate of this bug. ***
Comment 14 Tomas Mysik 2010-01-26 07:57:38 UTC
Should be fixed, file download is based on the remote file timestamp (newer files are preselected). Filipe, please verify, thanks.
http://hg.netbeans.org/web-main/rev/6d66b02b9acc
Comment 15 Filip Zamboj 2010-01-26 08:56:15 UTC
verified 
Product Version: NetBeans IDE Dev (Build 100126-f3c43c81a4d8)
Java: 1.6.0_16; Java HotSpot(TM) 64-Bit Server VM 14.2-b01


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo