This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 17012 - NNTP data corrupted
Summary: NNTP data corrupted
Status: RESOLVED INVALID
Alias: None
Product: obsolete
Classification: Unclassified
Component: collabnet (show other bugs)
Version: 3.x
Hardware: PC Windows 3.1/NT
: P1 blocker (vote)
Assignee: support
URL:
Keywords:
Depends on:
Blocks: 17321
  Show dependency tree
 
Reported: 2001-10-26 10:33 UTC by Martin Ryzl
Modified: 2009-11-08 02:28 UTC (History)
2 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Ryzl 2001-10-26 10:33:47 UTC
Today, I found out that there are no new postings since yesterday 9pm (CET) (I'm also subscribed to
nbdiscuss so it was easy to compare) and some older postings were expired. I re-subscribed all groups and
new postings appeared but all postings between Aug/31 and today are missing.
Comment 1 Jesse Glick 2001-10-26 11:36:35 UTC
I can confirm. I use a direct link to news.netbeans.org from Netscape
Comm. 4.76. All was working fine in NNTP land last night (Europe
time). This morning, there appear to be no new messages in any of the
groups, and clicking on any group gives me e.g.:

---%<---
Error!

newsgroup server responded: Bad article number

Perhaps the article has expired

<netbeans.nbdiscuss/3BD85D77.D78F4945@Sun.COM> (899)

Click here to remove all expired articles 
---%<---

(This from Rochelle Raccah's post yesterday "why do newsletters come
on Tuesday?" on netbeans.nbdiscuss.)

Please leave as P1, it is impossible to see any new messages over NNTP
while this is broken.
Comment 2 mboyer 2001-10-26 15:02:25 UTC
Are you using an external news server?
Comment 3 Martin Ryzl 2001-10-26 15:42:36 UTC
I use directly news.netbeans.org. Netscape 4.78.
Comment 4 Taska 2001-10-26 16:23:35 UTC
Turned this over to our operations group (PCN6217).
Comment 5 Taska 2001-10-27 20:40:22 UTC
Our operations group is looking into this.
Comment 6 Taska 2001-10-30 03:28:28 UTC
Our operations group has replicated the problem and has escalated to
the developer who set up the NNTP server.
Comment 7 Taska 2001-10-30 15:57:44 UTC
*** Issue 16780 has been marked as a duplicate of this issue. ***
Comment 8 Taska 2001-10-30 16:11:04 UTC
Updates from issue 16780:
nbui newsgroup hasn't been updated since Sunday, October 14.
Comment 9 Jesse Glick 2001-11-01 21:55:34 UTC
Is there any progress on this? I have not been able to access the NB
lists for a week now (nbui for longer). A daily update would be very
much appreciated.
Comment 10 Taska 2001-11-01 22:27:58 UTC
Our development group and our operations group are working on this. 
There is no estimated date for fix delivery at this time.
Comment 11 Martin Ryzl 2001-11-02 10:06:30 UTC
OK, it seems older postings are back but there are no new postings:
nbusers last: news Thu 11:27, mail Fri 10:30
Comment 12 Taska 2001-11-02 16:01:33 UTC
Our NNTP server developer has rebuilt the history and the overview for
the NNTP server.  Please give us updates as to whether you can now see
new NNTP messages, and if there is anything that looks out of the
ordinary anymore.  Thanks.
Comment 13 Jesse Glick 2001-11-02 17:37:48 UTC
I no longer get errors about expired messages, but I only see messages
up through Oct 25 in the lists (checked netbeans.modules.openide.dev
and netbeans.nbnews). I am using the same Netscape (4.76)
configuration I was using before the problem started, as I don't wish
to lose information about read/unread messages. nbui still stops after
Oct 12.
Comment 14 Jesse Glick 2001-11-02 17:45:05 UTC
Note: I also started up Mozilla (0.9.5) after having deleted any
previous news account settings, and connected to news.netbeans.org
from scratch and subscribed to the groups. I still saw only messages
up to Oct 25.
Comment 15 Unknown 2001-11-02 17:58:06 UTC
When I used netscape, at first I only saw messages through Oct 25th. 
Then I told netscape to sort by date, rather than thread -- and I
started to see recent messages.  I would hazzard a guess that the
messages between Oct 25th and Nov 1st wound up with lower IDs than
those from August - OCt 25th which were temporarily unavailable.  

When I tried this just now, the most recent date which I found in
nbusers was from :

Thu, 1 Nov 2001 22:26:57 +0000 (UTC)

Unfortunately, that's yesterday, so that's not exactly evidence that
it's working currently.

nbui is definitely not in a good state.
Comment 16 Jesse Glick 2001-11-03 14:24:47 UTC
I tried sorting by date on netbeans.modules.openide.dev and
netbeans.nbusers and still saw only messages thru Oct 25 or
thereabouts.
Comment 17 Martin Ryzl 2001-11-05 16:13:28 UTC
I did a short investigation on nbusers:

  telnet news.netbeans.org

  group netbeans.nbusers
  211 23283 1 12162 netbeans.nbusers
              ^^^^^ it means last message should have number 12162

But last message I have received so far has number 10879.
  NNTP-Posting-Date: Mon, 5 Nov 2001 07:50:20 +0000 (UTC)
  Xref: news.collab.net netbeans.nbusers:10879

So what's wrong? I did one more investigation:

  telnet news.netbeans.org

  group netbeans.nbusers
  211 23285 1 12162 netbeans.nbusers
  head 12162
  ...
  NNTP-Posting-Date: Fri, 26 Oct 2001 00:17:55 +0000 (UTC)

OK, here is the problem:
message received on Oct 26, 00:17:55 with subject
"[nbusers] Can Forte debugger display in HEXadecimal" has number 12162
and the next one has number 10619!

As a side effect, some threads have bad order :(.

Solution is not so hard, just turn off the news server, backup it, run 
perl script that renumbers all messages with wrong number and update 
control structures accordingly.

Please, do it as soon as possible, I dont't want to wait several 
months to see new postings again :(.

Comment 18 Taska 2001-11-06 00:14:10 UTC
We're going to be taking down the NNTP server for testing and bugfixes
starting at noon Pacific time on Tuesday, Nov 5.  The downtime may
last up to 8 hours.  I'll post updates here.
Comment 19 Unknown 2001-11-06 02:03:06 UTC
I used INNd's renumber function, which seems to have put things into a
slightly better state -- going forward.  However, I believe that at
this point, some number of the messages from around the end of August
to the begining of September are being hidden by messages from late
October and the begining of November.  For example, on netbeans.nbdev,
messages # 10387-10599 are hidden by more recent messages.  The
message IDs for the messages which are causing this hiding are
incorrect -- they're much earlier than they should be -- and this
messes up most clients' threading.

I was able to verify that the hidden messages appear to be present on
the filesystem, just inaccessible by NNTP.  I was also able to verify
-- using telnet -- that new messages are getting new IDs.  Before I
did the renumber, the highest numbered message was 12001, and the most
recent message was numbered 10599.  After the renumber, subsequent
messages recieved numbers above 12002, and now the display in netscape
appears correct.

This does not resolve in incorrect estimates as to the total number of
messages -- netbeans.nbdev still estimates more than 22000, when the
actual number is more than 12000.

At this point, I have some additional ideas, but I'd like to do some
more research before I touch anything further -- so I won't be
touching the live machine tomorrow.  However, it's possible that there
will be downtime on Nov 7th similar to what Taska described.

The observations about OE's behavior were helpful -- thanks.
Comment 20 Unknown 2001-11-06 02:23:36 UTC
As a side note --  Martin, if you want to point me to a perl script
which does what you described, or provide one yourself, I'd love to
take a look at it.

Without reading the internals of INNd (certainly possible since the
source is available -- just not fast ;-), it's not obvious to me how
I'd go about updating the control structures (which are not flat
files) without using the tools provided or risking further
inconsistencies.  I'm sure it is possible, but none of the perl
scripts provided in the distribution appeared to handle this kind of
problem.

I'll be testing a solution using the tools provided by INNd -- ctlindd
and makehistory.

I've made a change to nbui which may resolve its problems -- AFAICT
the issue was that the newsgroup didn't have the appropriate gateway
address subscribed (it's possible to remove this through the admin UI;
examining Log for that list might help explain how the address got
unsubscribed).  Future messages should appear in the newsgroup.
Comment 21 Unknown 2001-11-07 04:10:45 UTC
I've rebuilt the overview and renumbered again in such a way that the
estimated message count is not far from the actual message count --
eg.:

211 12238 1 12026 netbeans.nbdev

The difference is the number of messages which have duplicate message
IDs (the newer ones hide the older ones) -- 212 in this case.  It is
possible, as Martin indicated, to modify the files (changing the Xref
header in each one) so that the numbering will be correct after
another rebuild of the overview -- but writing and testing such a
script will take some time.

The operation described above took about 30 minutes -- NNTP was only
down for about that long.  New messages were not appearing in NNTP
during that time, plus an additional 2 & 1/2 hours (time for backup
and some testing afterwards).

The current status is that new messages should work, but some older (2
months old) messages are not visible, and some more recent messages
are sorted incorrectly by common NNTP clients.  Separately, I've
verified that netbeans.nbui is actually recieving messages again.
Comment 22 Jesse Glick 2001-11-07 10:52:25 UTC
I can now see new messages. All messages from 10/26 up until around
Tuesday are missing, at least in Netscape 4.76.
Comment 23 Taska 2001-11-07 23:37:53 UTC
Okay, so we have fixed the problem going forward.  There are some
historical messages which have been hidden by some other messages
because their numbers are the same.  Sometime in the next month, we
will be moving to a new NNTP server, and this historical problem will
be fixed in the process.
Comment 24 Unknown 2004-10-13 09:14:00 UTC
closing..
Comment 25 Marian Mirilovic 2009-11-08 02:28:58 UTC
We recently moved out from Collabnet's infrastructure