This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 185850 - not receiving NB mail
Summary: not receiving NB mail
Status: RESOLVED FIXED
Alias: None
Product: www
Classification: Unclassified
Component: Admin (show other bugs)
Version: 6.x
Hardware: All All
: P1 normal (vote)
Assignee: leawang
URL:
Keywords:
: 185943 (view as bug list)
Depends on:
Blocks:
 
Reported: 2010-05-10 15:24 UTC by ivan
Modified: 2010-06-01 16:22 UTC (History)
16 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter:


Attachments
E-mail which needed 1.5 day to be delivered to DT mailing list. (14.16 KB, text/plain)
2010-05-12 08:04 UTC, Jiri Kovalsky
Details
Another slow e-mail to aumasters list which needed 3 days (!) to be delivered. (7.38 KB, text/plain)
2010-05-13 12:14 UTC, Jiri Kovalsky
Details
Chart showing daily traffic of messages on the NetCAT mailing list. (16.65 KB, image/png)
2010-05-19 11:48 UTC, Jiri Kovalsky
Details
IGNORE - user probably left Sun (FYI: source of an email returned to me as undeliverable; I'm not sure if it is of any help but here it is anyway) (8.20 KB, text/plain)
2010-05-20 00:32 UTC, David Konecny
Details

Note You need to log in before you can comment on or make changes to this bug.
Description ivan 2010-05-10 15:24:29 UTC
Ever since the relocation on the weekend of May 1st I haven't been
receiving mail sent to ivan@netbeans.org. (e.g. if I mail to
myself I wont' receive it).
This in turn impacts my ability to receive bugzilla notifications
as well as any newsgroup posts. Strangely enough I get email
through "virtual" lists like apichanges.
I know other engineers in menlo park who also have stopped receiving 
most netbeans communication.

Please answer me directly with 
ivan
.
soleimanipour
@ 
oracle
.
com

as I won't get any notifications on status change of this IZ.
Comment 1 Jesse Glick 2010-05-10 17:30:03 UTC
I have not received mail sent to jglick for a week or two now. I filed an iSupport request since it looks like the problem is the Oracle mail servers (they are rejecting the messages for unclear reasons), but of course there is no response from the iSupport people yet.

In the meantime I have redirected my @netbeans.org mail (via my account settings) to my personal GMail account, which works but is not pleasant since my mail filters are not set up for it.
Comment 2 Jiri Kovalsky 2010-05-10 18:56:41 UTC
For example Ulf Zibis (NetCAT 6.9 participant) is facing the same issue. I only hope this is treated as the highest priority issue because it impacts our huge NetBeans community as well.
Comment 3 Martin Fousek 2010-05-11 08:44:00 UTC
pblaha perhaps found cause of this issue and pkucera already fixed it. Could anybody of affected users take a chance it again if it was root case of this issue, please?
Comment 4 ulfzibis 2010-05-11 13:53:56 UTC
Seems to work again.
- received emails via netcat list from other users
- received test email to ulfzibis at netbeans.
Comment 5 Martin Fousek 2010-05-11 14:19:05 UTC
Thanks to two NetCAT users, we got first feedbacks to fix of this issue. It helped just one of them (ulfzibis) but it was also possible that the second one hadn't propagated new DNS entries yet.

Please could anybody else who was affected give us feedback? Thanks...
Comment 6 Petr Blaha 2010-05-11 17:50:49 UTC
*** Bug 185943 has been marked as a duplicate of this bug. ***
Comment 7 Jesse Glick 2010-05-11 18:17:54 UTC
I have not yet had any success getting my @netbeans.org to forward to my @oracle.com; has been broken since about the beginning of the month. (Today I tried again and now I can no longer get it sent to my GMail either, so this is serious; and https://netbeans.org/people/19-Jesse-Glick/edit just gives me a 502 Proxy Error.)
Comment 8 alied 2010-05-11 19:52:52 UTC
I receive issues notifications, but haven't received mails in a week.
Today I received Ulf's test mail:
See CC: list

Cheers,

-Ulf 

and Jiri's NetCAT Status Report - 8.
other than that, Zero.

(In reply to comment #7)
> I have not yet had any success getting my @netbeans.org to forward to my
> @oracle.com; has been broken since about the beginning of the month. (Today I
> tried again and now I can no longer get it sent to my GMail either, so this is
> serious; and https://netbeans.org/people/19-Jesse-Glick/edit just gives me a
> 502 Proxy Error.)
Comment 9 David Konecny 2010-05-11 20:35:04 UTC
For last 12 hours I configured dkonecny@netbeans.org to point to a gmail account and I have received bunch of emails. Today morning I changed it to point to my oracle address and I never received "Time to Verify..." email. Then I changed it to my old sun.com address and this time I got "Time to Verify..." email and was able to confirm email change. Since then I've got some emails (eg. last two change notifications from this bug; one hg commit notification). But sending anything to "dkonecny@netbeans.org" directly by me (or from a gmail account) has never been delivered to me (> 30 minutes ago).
Comment 10 Marco Walther 2010-05-11 21:12:36 UTC
Oracle incoming MTA's do not seem to like us:-( I'll try to file a ticket with them, but I don't know how quickly they will fix it:-(

May 11 21:06:59 mailout01 postfix-out/smtp[5861]: [ID 197553 mail.info] C770611C90: to=<v*.k*@oracle.com>, relay=none, delay=435096, delays=434975/0.06/120/0, dsn=4.4.1, status=deferred (connect to acsinet11.oracle.com[141.146.126.233]:25: Connection timed out)
May 11 21:06:59 mailout01 postfix-out/error[6623]: [ID 197553 mail.info] 947A5122D0: to=<v*.k*@oracle.com>, relay=none, delay=42455, delays=42334/122/0/0.07, dsn=4.4.1, status=deferred (delivery temporarily suspended: connect to acsinet12.oracle.com[141.146.126.234]:25: Connection timed out)
May 11 21:06:59 mailout01 postfix-out/qmgr[266]: [ID 197553 mail.info] C770611C90: from=<nbusers-owner@netbeans.org>, status=expired, returned to sender
May 11 21:06:59 mailout01 postfix-out/cleanup[6612]: [ID 197553 mail.info] 119AD12C53: message-id=<20100511210659.119AD12C53@mailout01.kenai2.com>
May 11 21:06:59 mailout01 postfix-out/error[6639]: [ID 197553 mail.info] 90A7614D10: to=<v*.k*@oracle.com>, relay=none, delay=386923, delays=386802/122/0/0.05, dsn=4.4.1, status=deferred (delivery temporarily suspended: connect to acsinet12.oracle.com[141.146.126.234]:25: Connection timed out)
May 11 21:06:59 mailout01 postfix-out/bounce[6642]: [ID 197553 mail.info] C770611C90: sender non-delivery notification: 119AD12C53
May 11 21:06:59 mailout01 postfix-out/qmgr[266]: [ID 197553 mail.info] 119AD12C53: from=<>, size=7776, nrcpt=1 (queue active)
May 11 21:06:59 mailout01 postfix-out/qmgr[266]: [ID 197553 mail.info] C770611C90: removed
May 11 21:06:59 mailout01 postfix-out/smtp[5867]: [ID 197553 mail.info] 119AD12C53: to=<nbusers-owner@netbeans.org>, relay=mailin02.kenai.com[192.9.171.174]:25, delay=0.11, delays=0.01/0/0.01/0.09, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 1795D657CB)

Thanks,
-- Marco
Comment 11 Marco Walther 2010-05-11 21:55:09 UTC
OK, while trying to file an Oracle-internal support request:

-----------------------------------------------------------
The following error occurred:
	 
  	  java.lang.NullPointerException 	 
  	No further information.
-----------------------------------------------------------
Comment 12 David Konecny 2010-05-11 22:28:35 UTC
After two hours I just received my test email from gmail to dkonecny@nb. I also another email (notice it is from 6th of May):

The mail system
<david.konecny@oracle.com>: delivery temporarily suspended: connect to
    rcsinet11.oracle.com[148.87.113.123]:25: Connection timed out

Reporting-MTA: dns; mailout01.kenai2.com
X-Postfix-Queue-ID: 06DF5144FE
X-Postfix-Sender: rfc822; dkonecny@netbeans.org
Arrival-Date: Thu,  6 May 2010 21:37:49 +0000 (GMT)

Final-Recipient: rfc822; david.konecny@oracle.com
Original-Recipient: rfc822;dkonecny@netbeans.org
Action: failed
Status: 4.4.1
Diagnostic-Code: X-Postfix; delivery temporarily suspended: connect to
    rcsinet11.oracle.com[148.87.113.123]:25: Connection timed out
Comment 13 leawang 2010-05-12 00:53:49 UTC
We are working w/ Oracle IT now to try to get proper person engaged to have Oracle MTAs accept emails from netbeans.org.
Comment 14 arittner 2010-05-12 05:54:15 UTC

I've problems with the nbdt-Mailing list - some emails goes throw in minutes - some mails needs two days



very confusing, br, josh.
Comment 15 leawang 2010-05-12 07:34:55 UTC
Do you have an example of emails that take days?  It might help us to track down issues.  Thanks.
Comment 16 Jiri Kovalsky 2010-05-12 08:04:20 UTC
Created attachment 98815 [details]
E-mail which needed 1.5 day to be delivered to DT mailing list.
Comment 17 Jiri Kovalsky 2010-05-12 08:08:11 UTC
I have attached an example of such slow e-mail. My e-mail client shows that it was sent on May 10 at 12:26 PM but I received it last night from May 11th to 12th.
Comment 18 leawang 2010-05-12 16:57:19 UTC
Thanks for the attachment.  Will look into it.
Comment 19 leawang 2010-05-12 17:01:12 UTC
We had some issues w/ firewall config wrt emails out to Oracle MTAs.  It has been corrected.  Please confirm that you are receiving emails to oracle email address now.
Comment 20 leawang 2010-05-12 17:49:04 UTC
Resolved on emails to oracle.com email addresses.  Josh, we had emails problems last week for various reasons.  Can you see whether you still have issues on the nb mailing list delay?  If yes, please file a separate bug.  Thanks.
Comment 21 David Konecny 2010-05-12 20:28:05 UTC
My dkonecny@n.o points to oracle address now and it looks as I'm getting all emails now. However emails I sent directly to dkonecny@n.o myself (from oracle or gmail address) have not arrived yet (>45 minutes). Can somebody else try it? Yesterday it took about 2 hours to deliver one from gmail account. Any explanation of that?
Comment 22 leawang 2010-05-12 20:30:25 UTC
Yes, we ran into the same thing ourselves.  We are working on that as well.  Just realized that we have identified our outgoing MTAs host name wrong.  We were still using a name that was used for intrim testing.  Duh.  Correcting them now..
Comment 23 David Konecny 2010-05-12 20:36:21 UTC
Thanks.
Comment 24 Jiri Kovalsky 2010-05-13 12:14:56 UTC
Created attachment 98937 [details]
Another slow e-mail to aumasters list which needed 3 days (!) to be delivered.
Comment 25 Jesse Glick 2010-05-13 13:56:07 UTC
I switched my account back to @oracle.com and I seem to be getting mails sent by the system, including directly to jglick@nb.org. I do not get anything I mail myself to jglick@nb.org for whatever reason.
Comment 26 leawang 2010-05-13 14:31:28 UTC
Jesse, David, Oracle.com MTAs can receive emails from us now, but still not sending emails to our incoming MTAs. We are still working w/ IT on that.

Wrt small mailing list delivery, we still need to look into more why.
Comment 27 leawang 2010-05-13 20:23:36 UTC
Emails from oracle.com can be sent to netbeans.org email addresses now.  Please verify that it works for you all.

Wrt the mailing list slowness, we have resolved a problem.  If still experiences issues, please let us know.
Comment 28 leawang 2010-05-13 20:24:37 UTC
Also, Yahoo! is rejecting emails from kenai.com and netbeans.org.  We are currently working w/ them to resolve them.  All rejected emails are queuing right now.
Comment 29 Jesse Glick 2010-05-14 16:53:49 UTC
(In reply to comment #27)
> Emails from oracle.com can be sent to netbeans.org email addresses now.  Please
> verify that it works for you all.

Mails sent from my Oracle SMTP to my @netbeans.org address do not seem to arrive, or maybe they are being held up indefinitely. Not sure where the problem lies. Much lower priority, since mails coming from e.g. Bugzilla are being delivered correctly via jglick@ to my Oracle inbox.
Comment 30 leawang 2010-05-14 17:04:33 UTC
A lot emails sent from oracle.com account was being processing slowly. At least all my test emails finally arrived after 1.5 days.  I sent you an email just now from my oracle account to your nb account.
Comment 31 Jesse Glick 2010-05-14 18:09:33 UTC
(In reply to comment #30)
> I sent you an email just now from my oracle account to your nb account.

Which it seems I got.
Comment 32 David Konecny 2010-05-17 01:51:59 UTC
I got some emails over the weekend but none today (direct email nor issuezilla notifications).
Comment 33 Jan Pirek 2010-05-17 12:45:52 UTC
Looks like Bugzilla notifications and Hg commit messages are not going through now at all. Not sure if this is related to the @oracle issue identified in this case
Comment 34 leawang 2010-05-17 18:21:40 UTC
There were some issues w/ amavisd got stuck.  It should be resolved now.
Comment 35 Jesse Glick 2010-05-18 01:23:59 UTC
No mails appear to be going through to the lists.
Comment 36 David Konecny 2010-05-18 01:34:30 UTC
Today morning I've found most of email notifications from bugzilla which I was expecting yesterday and over weekend. My direct email to dkonecny@nb.org still has not arrived (>24h). I seem to be getting some notifications from bugzilla today.
Comment 37 Vladimir Voskresensky 2010-05-18 07:27:51 UTC
reopening: Hg commit messages are not delivered
Comment 38 Martin Fousek 2010-05-18 12:22:28 UTC
HG commit emails should be related to HG system on hg.nb.org, it shouldn't be troubles of kenai infrastructure. Let's wait for resolution of Robert Novak.
Comment 39 leawang 2010-05-18 15:04:30 UTC
Jesse, which mailing list email that you are not receiving? 

hg notification emails don't use Kenai infrastructure..
Comment 40 Jesse Glick 2010-05-18 15:33:43 UTC
(In reply to comment #39)
> which mailing list email that you are not receiving?

Several, but for example nothing since May 14th in <http://netbeans.org/projects/www/lists/nbdev/archive> (and I definitely posted myself yesterday). At least I do not see anything through either the GMane NNTP mirror or the archives. <http://netbeans.org/projects/platform/lists/dev/archive> shows nothing after May 14th except for two today (usually this gets many messages daily).

> hg notification emails don't use Kenai infrastructure

They do in that most (not all) of them are sent to @netbeans.org addresses, generally lists. For example, nothing at <http://netbeans.org/projects/platform/lists/core-commits/archive> since May 14th, though there have certainly been changes in those modules since then.
Comment 41 Jesse Glick 2010-05-18 15:44:14 UTC
Update: I just got a couple of commit messages from ten hours ago, and then one from May 14th. Many more are missing, so I guess there is a long backlog somewhere.
Comment 42 leawang 2010-05-18 15:48:13 UTC
The long backlog is being processed again...  Mark it resolved for now.  If any urgent issues, please do let us know.
Comment 43 ulfzibis 2010-05-18 16:26:14 UTC
I do not receive any emails from NetCat list since 14.05.2010 21:43 CET.
Not on Ulf.Zibis ... gmx dot de and
not on Ulf.Zibis ... CoSoCo dot de.
Comment 44 leawang 2010-05-18 19:35:53 UTC
Ulf, do you see the latest email sent to you about 1 hour or so ago?

May 18 18:19:08 mailout01 postfix-out/smtp[7669]: [ID 197553 mail.info] EA312111B
A: to=<ulf dot zibis at gmx dot de>, relay=mx1.gmx.net[213.165.64.102]:25, delay=2.1, delays
=0.02/0.01/0.63/1.4, dsn=2.6.0, status=sent (250 2.6.0 Message accepted {mx118})
Comment 45 ulfzibis 2010-05-19 10:13:30 UTC
I only received one from 8:55 CET from Ingo Reinhart yesterday evening, and today one from 17.05.2000-19:51 from Matthies Nicklas and Jiri's Status Report from 22:17,
... and again I don't receive my own NetCAT response from 11:48 CET.
Comment 46 Jiri Kovalsky 2010-05-19 11:48:04 UTC
Created attachment 99194 [details]
Chart showing daily traffic of messages on the NetCAT mailing list.

Please note that there were only 3 (!) messages delivered during the last 5 days which is abnormal situation. This issue heavily affects our community which can't voice its opinion in the crucial phase of NetBeans IDE 6.9 stabilization. Can this issue be finally fully fixed?
Comment 47 schkovich 2010-05-19 12:38:31 UTC
(In reply to comment #46)
> Created an attachment (id=99194) [details]
> Chart showing daily traffic of messages on the NetCAT mailing list.
> 
> Please note that there were only 3 (!) messages delivered during the last 5
> days which is abnormal situation. This issue heavily affects our community
> which can't voice its opinion in the crucial phase of NetBeans IDE 6.9
> stabilization. Can this issue be finally fully fixed?
Could not agree more. Planing such huge infrastructure change while the development of the NetBeans IDE 6.9 release is in the last phase is arrogant,  irresponsible and poor decision.
Comment 48 Jiri Kovalsky 2010-05-19 12:51:39 UTC
In addition to that, please don't close this issue immediately when you _think_ it was fixed. It's better to observe the traffic at least for few days before closing this bug. Thanks!
Comment 49 leawang 2010-05-19 15:25:39 UTC
Unfortunately, we don't control when the infrastructure move needs to be done. 
It was a decision that we had to follow.

Sorry that we still have email issues.  Please list mailing list aliases that
are experiencing issues in addition to Ulf's email issue, we will look into it
right away.  Thanks.
Comment 50 Jan Pirek 2010-05-19 15:59:54 UTC
another list having issues: webmaster@netbeans.org
Comment 51 Troy Giunipero 2010-05-19 16:08:13 UTC
Since May 14th (last Friday) I've received very little communication on
following mailing lists:

nbdocs_feedback@usersguide.netbeans.org
nbj2ee@netbeans.org
nbusers@netbeans.org

When emails do manage to get through, they are often hours, sometimes days,
late.  This continues to be the case for me today, despite the efforts to
resolve the matter.
Comment 52 David Konecny 2010-05-19 18:50:54 UTC
(In reply to comment #36)
> My direct email to dkonecny@nb.org still has not arrived (>24h).

It's been couple of days now and direct email from my @oracle address to my @netbeans address still has to arrive.
Comment 53 schkovich 2010-05-19 19:04:10 UTC
(In reply to comment #49)
> Unfortunately, we don't control when the infrastructure move needs to be done. 
> It was a decision that we had to follow.
> 
> Sorry that we still have email issues.  Please list mailing list aliases that
> are experiencing issues in addition to Ulf's email issue, we will look into it
> right away.  Thanks.
Well, I did not say that it was you. 

Wise guy(s) who did make this decision should come out of a dark and apologize to the NetBeans developers and community for the mess they created. Otherwise bitter  taste will remain and ones spreading word that Oracle plans to drop the NetBeans IDE will have one more argument.
Comment 54 David Konecny 2010-05-20 00:32:43 UTC
Created attachment 99227 [details]
IGNORE - user probably left Sun (FYI: source of an email returned to me as undeliverable; I'm not sure if it is of any help but here it is anyway)
Comment 55 leawang 2010-05-22 00:21:48 UTC
We have increased additional spam filter workers.  The backlog of emails should be finished catching up.

Please let us know whether still seeing any issues w/ emails...  Thanks.
Comment 56 Jiri Kovalsky 2010-05-24 12:18:54 UTC
Thanks for your update Lea! I am still observant due to these e-mail counts for the past days:

nbusers@netbeans.org
====================
5/18: 28
5/19: 31
5/20: 51
5/21: 38
5/22: 7
5/23: 2
5/24: 1

netcat@netbeans.org
===================
5/18: 5
5/19: 12
5/20: 11
5/21: 8
5/22: 1
5/23: 0
5/24: 0
Comment 57 Martin Fousek 2010-05-24 12:47:22 UTC
I think that it worked at least somehow till about 2pm PRG time today (24/05). But then it seems that emails stopped flowing again, I already reported that to kenai-staff team.
Comment 58 Martin Fousek 2010-05-24 12:50:01 UTC
I think that it worked at least somehow till about 2pm PRG time today (24/05). But then it seems that emails stopped flowing again, I already reported that to kenai-staff team.
Comment 59 leawang 2010-05-24 19:32:34 UTC
Our spam filter got jammed again.. Mails should be flowing again.  We will keep any close eye on it from this point on...  Thanks.
Comment 60 John Jullion-ceccarelli 2010-05-25 07:48:49 UTC
are there any automated tests in place that check for jammed spam filters?
Comment 61 Martin Fousek 2010-05-25 08:41:18 UTC
What I know kenai-staff team has a watchdog if the service is up which perhaps not detect this issue.

I find out that yesterday so I'm going to setup something for that soon with email to kenai-staff team.
Comment 62 leawang 2010-05-26 15:17:41 UTC
Yes, we have MTA monitoring including queue info and trying to figure out why one of the MTA seems receiving large % of emails v.s. another one.  The emails are flowing, but one of the active queue is having ard 6600 emails and being catching up.
Comment 63 John Jullion-ceccarelli 2010-05-26 15:50:25 UTC
Yes some emails are getting through but for most part takes about a day. Which makes having efficient email threads with NetCAT participants next to impossible. Seems all nb.org email aliases are affected.
Comment 64 leawang 2010-05-26 15:55:23 UTC
Yes, we had 12k backlog to go through midnight last night PMT, now it is down to 6300.  It is trying to catch up the old backlog in addition to new emails.

Sorry for the bad timing.  We had some IO bottlenecks w/ amavisd and now we use DB to process.
Comment 65 leawang 2010-05-26 17:35:49 UTC
The backlog is down to 4755 now... Catching up continues.
Comment 66 Marco Walther 2010-05-27 06:35:48 UTC
After some DB cleanup, both mailin0? hosts are down to zero backlog.
Comment 67 leawang 2010-05-27 19:24:38 UTC
Since midnight PM last night, email should be flowing.  Both incoming MTAs have no backlog for last 12 hours.  If anyone is still aware of an email issue, please provide specific info.  Thanks.
Comment 68 leawang 2010-06-01 16:22:08 UTC
After 5 days, no reports of additional email issues reported.