This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 22101 - Slow response from CVS server [solaris migration]
Summary: Slow response from CVS server [solaris migration]
Status: RESOLVED INVALID
Alias: None
Product: obsolete
Classification: Unclassified
Component: collabnet (show other bugs)
Version: 3.x
Hardware: PC Windows ME/2000
: P1 blocker (vote)
Assignee: support
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2002-04-04 13:07 UTC by mvinar
Modified: 2009-11-08 02:29 UTC (History)
2 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter:


Attachments
Log (67.57 KB, text/plain)
2002-04-04 13:09 UTC, mvinar
Details
log2 (27.40 KB, text/plain)
2002-04-04 13:10 UTC, mvinar
Details

Note You need to log in before you can comment on or make changes to this bug.
Description mvinar 2002-04-04 13:07:28 UTC
We have a problem witch response from cvs server, for ex. 
if i do cvs update -d -P. I'll getting correct update bat 
this operation takes lot of minutes, this problem has 
started after solaris migration. Into attachment I'm 
sending some log files. Please let us know what's the 
problem. 
Thanks Marek
Comment 1 mvinar 2002-04-04 13:09:12 UTC
Created attachment 5288 [details]
Log
Comment 2 mvinar 2002-04-04 13:10:47 UTC
Created attachment 5289 [details]
log2
Comment 3 Taska 2002-04-04 15:55:06 UTC
Okay, we'll take a look.
Comment 4 _ ttran 2002-04-04 16:01:53 UTC
I am raising the priority to P2.  This issue is becoming a significant
 problem for developers here.

(I was tempted to give it P1 given the fact that the upgrade was
planned to solve the performance problems with the site)
Comment 5 Taska 2002-04-04 16:46:49 UTC
Hmm.  Marek, the log files look normal, but of course they don't
include time-stamps.  I'm doing a checkout right now:
cvs -d :pserver:anoncvs@cvs.netbeans.org:/cvs co standard
... and I'll do an update as soon as the checkout is done.  The
checkout seems to be clipping along at normal speed.

Trung: this upgrade was not planned to solve any performance problems.
 First, it was not an upgrade; it was a migration from a Linux server
to a Solaris server.  Second, the migration was planned because Sun
wanted to run NetBeans.org on a Solaris server, not because it would
necessarily have any performance improvements.
Comment 6 Taska 2002-04-04 19:23:30 UTC
For lack of a better test, I did a cvs up of openide on both the live
site and our old linux staging site, right after having checked each
out.  On the linux staging server, it flies right through the update;
on the live site, it seems to qualitatively take about as long to do
the up as the checkout.

I'm having our instantiations engineer take a look (PCN9010).
Comment 7 _ ttran 2002-04-04 19:55:57 UTC
> Trung: this upgrade was not planned to solve any performance problems

sorry, I was to fast with my speculation, taking my comment re reason
of upgrade back.
Comment 8 jcatchpoole 2002-04-05 10:39:40 UTC
FWIW while this migration was not done specifically to improve
performance, that was supposed to be a happy by-product.  From issue
16827, regarding performance issues :

> 2) We believe that the Solaris migration will remedy the problem
> further, via:
> a) more bug fixes;
> b) performance-tuned hardware and software;
> c) significant differences in the JVM.

Does it look like the current CVS speed problems are machine-load
related ?
Comment 9 Taska 2002-04-05 10:46:11 UTC
Actually, it doens't look like it's load-related.  We've replicated
the problem on the staging server.  Our cvs group is looking at the
issue now.
Comment 10 Patrick Keegan 2002-04-05 13:52:48 UTC
Since the migration, cvs has been very slow, but now I'm getting these 
messages every 30 seconds:

cvs server: [04:54:36] waiting for tigrisc's lock in 
/cvs/usersguide/javahelp/or
g/netbeans/modules/usersguide/vcs-nb/cvs-nb

I got this message after most of my update was completed (4:54 a.m. 
PST, I suppose). The messages have continued since then (about an 
hour). I tried interrupting the update and then running it again, but 
the same message keeps appearing on my console.
Comment 11 Martin Entlicher 2002-04-05 14:45:32 UTC
The same happens to me and other developers.
cvs server: [06:38:23] waiting for tigrisc's lock in
/cvs/vcscore/src/org/netbeans/modules/vcscore/commands

Comment 12 Martin Entlicher 2002-04-05 14:52:42 UTC
Upgrading to P1, the CVS server is unusable.
Comment 13 jcatchpoole 2002-04-05 15:23:44 UTC
Svata Dedic reports :

--
I've attempted to perform a cvs update -dP on all nb 
modules and got this message:

cvs [server aborted]: cannot open /tmp/cvs-
serv19419/lexer/CVS: No space left 
on device
--
Comment 14 Jesse Glick 2002-04-05 15:28:17 UTC
I cannot currently update from core/src/org/netbeans/core/resources/ dir:

[jglick@localhost jglick]$ cd
/space/src/nb_all/core/src/org/netbeans/core/resources/
[jglick@localhost resources]$ cvs up
cvs server: Updating .
cvs server: [07:02:58] waiting for tigrisc's lock in
/cvs/core/src/org/netbeans/core/resources
cvs server: [07:03:28] waiting for tigrisc's lock in
/cvs/core/src/org/netbeans/core/resources
cvs server: [07:03:58] waiting for tigrisc's lock in
/cvs/core/src/org/netbeans/core/resources
cvs server: [07:04:28] waiting for tigrisc's lock in
/cvs/core/src/org/netbeans/core/resources
cvs server: [07:04:58] waiting for tigrisc's lock in
/cvs/core/src/org/netbeans/core/resources
cvs server: [07:05:28] waiting for tigrisc's lock in
/cvs/core/src/org/netbeans/core/resources
cvs server: [07:05:58] waiting for tigrisc's lock in
/cvs/core/src/org/netbeans/core/resources
cvs server: [07:06:28] waiting for tigrisc's lock in
/cvs/core/src/org/netbeans/core/resources
cvs server: [07:06:58] waiting for tigrisc's lock in
/cvs/core/src/org/netbeans/core/resources
cvs server: [07:07:28] waiting for tigrisc's lock in
/cvs/core/src/org/netbeans/core/resources
cvs server: [07:07:58] waiting for tigrisc's lock in
/cvs/core/src/org/netbeans/core/resources
cvs server: [07:08:28] waiting for tigrisc's lock in
/cvs/core/src/org/netbeans/core/resources
cvs server: [07:08:58] waiting for tigrisc's lock in
/cvs/core/src/org/netbeans/core/resources
cvs server: [07:09:28] waiting for tigrisc's lock in
/cvs/core/src/org/netbeans/core/resources
cvs server: [07:09:58] waiting for tigrisc's lock in
/cvs/core/src/org/netbeans/core/resources
cvs server: [07:10:28] waiting for tigrisc's lock in
/cvs/core/src/org/netbeans/core/resources
cvs server: [07:10:58] waiting for tigrisc's lock in
/cvs/core/src/org/netbeans/core/resources
cvs server: [07:11:28] waiting for tigrisc's lock in
/cvs/core/src/org/netbeans/core/resources
cvs server: [07:11:58] waiting for tigrisc's lock in
/cvs/core/src/org/netbeans/core/resources
cvs server: [07:12:28] waiting for tigrisc's lock in
/cvs/core/src/org/netbeans/core/resources
cvs server: [07:12:58] waiting for tigrisc's lock in
/cvs/core/src/org/netbeans/core/resources
cvs server: [07:13:28] waiting for tigrisc's lock in
/cvs/core/src/org/netbeans/core/resources
cvs server: [07:13:58] waiting for tigrisc's lock in
/cvs/core/src/org/netbeans/core/resources
cvs [update aborted]: received interrupt signal
[jglick@localhost resources]$ 

I have not tried other dirs.

I recall that on the Linux machine there was some kind of daemon in
place (?) that would look for, and maybe automatically kill, stale CVS
locks (i.e. a lock in the repo that stayed around for more than ten
minutes or so, indicating a problem). If so, did this daemon not get
migrated?
Comment 15 Taska 2002-04-05 18:47:24 UTC
I've escalated the internal issue to P1 and moved it back to
operations to triage.
Comment 16 Taska 2002-04-06 00:24:20 UTC
We're able to replicate the slowness, but not the lock or space
errors- the /tmp is only apparently 15% full.  Are you still getting
the latter, or only the slowness again now?
Comment 17 Jesse Glick 2002-04-07 01:28:19 UTC
I just successfully updated the core and openide modules. Have not
tried anything else yet.
Comment 18 Taska 2002-04-10 04:35:37 UTC
Jesse, Marek, et al.:
Our operations engineer has applied a temporary test fix.  Could you
check to see whether the performance has improved?  Thanks.
Comment 19 Patrick Keegan 2002-04-10 15:17:22 UTC
The response seems to be reasonable again (about what it 
was before the migration).
Comment 20 Taska 2002-04-10 15:46:03 UTC
Thanks for the update Patrick.  I'll let our engineers know.
Comment 21 jcatchpoole 2002-04-10 17:24:57 UTC
Does seem to be back to normal for me also, thanks.
Comment 22 jcatchpoole 2002-04-11 11:47:47 UTC
Not sure if this is relevant but I saw this on nbusers, 
posted at 6:30pm CET by barter1@llnl.gov :

--
While following the instructions at:
http://www.netbeans.org/ide/docs/nb-cvs-33.html

I received an error.  A portion of the log is shown below.

-----------------------------------
[snip]
U debuggercore/www/javadoc/index-files/index-24.html

U debuggercore/www/javadoc/index-files/index-25.html

U debuggercore/www/javadoc/index-files/index-3.html

Terminated with fatal signal 11

Core dumped; preserving /tmp/cvs-serv12320 on server.

CVS locks may need cleaning up.

  ------------------------------------------
Comment 23 rbalada 2002-04-12 03:46:10 UTC
Taska,
I see locks again:
cvs server: [19:10:19] waiting for tigrisc's lock in /cvs/ant
...
cvs server: [19:41:20] waiting for tigrisc's lock in /cvs/ant
cvs server: [19:41:50] waiting for tigrisc's lock in /cvs/ant
Comment 24 Jesse Glick 2002-04-12 10:19:51 UTC
I saw similar messages (from core module, not ant) at one point
yesterday during a full update of the repository, and the update hung
for a while because of it. But then it resumed.
Comment 25 jcatchpoole 2002-04-12 15:45:35 UTC
Some of my scripts are returning strange output

The following is generated from the command

cvs commit -m "bla" platform/index.html devhome/index.html 
U ide/index.html

--
2002-04-12 16:10: Queued file(s) for publishing 
ide/index.html platform/index.html devhome/index.html 
U ide/index.html
U platform/index.html
U devhome/index.html
Checking in ide/index.html;
/cvs/testwww/www/ide/index.html,v  <--  index.html
new revision: 1.108; previous revision: 1.107
done
Processing log script arguments...
More commits to come...
Checking in platform/index.html;
/cvs/testwww/www/platform/index.html,v  <--  index.html
new revision: 1.107; previous revision: 1.106
done
Processing log script arguments...
More commits to come...
Checking in devhome/index.html;
/cvs/testwww/www/devhome/index.html,v  <--  index.html
new revision: 1.106; previous revision: 1.105
done
Processing log script arguments...
Mailing the commit message to cvs@testwww.netbeans.org 
(from mvinar@netbeans.org)
cvs [update aborted]: cannot rename file index.html to 
CVS/,,index.html: No such file or directory
U devhome/index.html
U ide/index.html
cvs update: [06:56:53] waiting for tigrisc's lock 
in /usr/local/tigris/data/helm/cvs/repository/testwww/www/i
mages/screenshots/platform
U platform/index.html
cvs update: [06:57:23] obtained lock 
in /usr/local/tigris/data/helm/cvs/repository/testwww/www/i
mages/screenshots/platform
--

Note the strange "aborted" msg.  All 3 commits do appear to
have completed successfully.

I am seeing a similar msg in another script;

--
Mailing the commit message to cvs@testwww.netbeans.org 
(from jcatchpoole@netbeans.org)
cvs [update aborted]: cannot rename file coding.html to 
CVS/,,coding.html: No such file or directory
U devhome/faqs/coding.html
--

This also comes from a single cvs commit from a parent dir
to multiple subdirs which I recall has caused weird 
problems in the past, ie

cvs commit dir1/file.html dir2/another.html
Comment 26 Taska 2002-04-12 16:06:26 UTC
Jack, Rudolf, Jesse- if you're seeing problems when checking in code
to www directories, that's probably a remote branding problem, not a
cvs problem.  Could you create a new issue for that one, and let me
know what directories you're seeing it in?  Our operations folks can
then fix any un-synced files on the server.

With respect to the "lock" message, it's pretty normal to briefly see
a lock message- someone may be checking code into the very files
you're trying to check out, while you're trying to check them out.
Comment 27 Jesse Glick 2002-04-12 16:47:51 UTC
Re. double-comma problem: yes, I have seen the same from time to time
while committing to www/ dirs. Don't remember the exact circumstances,
but it appeared to be harmless.

Re. lock messages appearing while committing: yes, this is normal in
CVS. It is *not* normal for the message to repeat ten times or so at
half-minute intervals. That means that a single lock file - which only
controls one (non-recursive) directory - was left stale on the server
for five minutes or more (see Ruda's reported 30+ minutes). It is
implausible that any CVS operation would require five minutes in a
single directory of sources unless the files involved were many
megabytes in size or had tens of thousands of tags etc.; processing
one directory should typically be a matter of a few seconds at most,
depending on the operation. Much more likely is that some CVS process
died while accessing that directory and left a stale lock, and it did
not get cleaned up for a while. In the meantime, any other CVS
processes traversing that directory hang, and will only unhang when
the stale lock is cleared - manually or via a daemon cleaner.
Comment 28 Taska 2002-04-15 19:04:12 UTC
I've spoken to our CVS specialists on this matter, and yes, lock files
may occassionally get left stale when CVS crashes during an action. 
We do have a lock-cleanup script that runs regularly to get rid of
stale locks.  We are designing our next-generation version control
system, Subversion, to deal with this situation more elegantly.
Comment 29 jcatchpoole 2002-04-16 09:41:43 UTC
Taska : FWIW I guess something got fixed as I havent seen 
any over the last 2 days (as opposed to hundreds last 
week).
Comment 30 Unknown 2004-10-13 09:20:27 UTC
closing...
Comment 31 Marian Mirilovic 2009-11-08 02:29:56 UTC
We recently moved out from Collabnet's infrastructure