Bug 191321 - push-to-* jobs broken by pretxnchangegroup.binary hook
push-to-* jobs broken by pretxnchangegroup.binary hook
Status: RESOLVED FIXED
Product: www
Classification: Unclassified
Component: Builds & Repositories
7.0
All All
: P3 (vote)
: 7.4
Assigned To: Jiri Rechtacek
nbre
http://hg.netbeans.org/main-silver/re...
: NO701, NO71, NO72
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2010-10-25 18:18 UTC by Jesse Glick
Modified: 2013-08-01 12:36 UTC (History)
3 users (show)

See Also:
Issue Type: DEFECT
:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jesse Glick 2010-10-25 18:18:30 UTC
51bbcfa1565c changed lib.profiler/release/**.dll rather than using binaries-list, which violates forbid_external.py. For some reason this changeset was accepted by both profiler-main and main-silver, but is not being accepted by other team repos, so push-to-* jobs are now broken. (And we cannot manually merge with silver, or we would not be able to push the results back to team repos.)

1. Please make sure that all public repos run exactly the same set of pretransaction hooks.

2. lib.profiler should be fixed to store its DLLs in a single ZIP file using http://wiki.netbeans.org/ExternalBinaries rather than dumping them in Hg.

3. In the meantime, forbid_external.py needs to be hacked to accept at least this changeset, so that normal propagation can be resumed.

4. It seems that this problem was not noticed earlier because mail delivery from bertram seems broken: http://bertram.netbeans.org/hudson/job/push-to-core-main/1683/console
Comment 1 Tomas Hurka 2010-10-25 18:46:25 UTC
Those files are the for years. I don't know what was changed recently, which caused this problem. They are not external binaries, they are build from sources in netbeans repository. This is the same case as with nbexex.dll, nbexec.exe, app.exe, pre7_app.exe and other binary files in netbeans repository.
Comment 2 Jesse Glick 2010-10-25 19:12:29 UTC
(In reply to comment #1)
> I don't know what was changed recently, which caused this problem.

You modified the files recently. The changesets adding the files initially were already in all repos so did not trigger the hook.

> They are not external binaries, they are build from
> sources in netbeans repository.

Nonetheless compiled executables would better not be stored in the repo.

> This is the same case as with nbexec.dll [...]
> and other binary files in netbeans repository.

These should probably be fixed too.
Comment 3 Tomas Hurka 2010-10-26 13:01:47 UTC
This problem was introduced by the following commit:

http://hg.netbeans.org/nb-hooks/rev/9bc05eed1e59

Looking at the forbid_external.py, the easiest way is to remove dll from the list of forbidden extensions. Since .exe, .so, .sl, .jnilib are not there, I don't understand why we should block files with dll extensions.
Comment 4 Jesse Glick 2010-10-26 14:15:01 UTC
Those other extensions should likely be added. It doesn't matter much what is done in the short term to get this one changeset through.
Comment 5 Tomas Hurka 2010-10-26 14:31:52 UTC
(In reply to comment #4)
> Those other extensions should likely be added. It doesn't matter much what is
> done in the short term to get this one changeset through.

I am not talking about this one changeset. In fact I do not see a reason, why these files should be treated as external binaries. They are not external, their size is small and they are changed very sporadically. I understand that having binary files in repository is not very nice, but the real problem here is that we don't have the infrastructure to build them as part of the ordinary build process.
Comment 6 Jesse Glick 2010-10-26 16:58:32 UTC
(In reply to comment #5)
> They are not external

Doesn't really matter.

> their size is small and they are changed very sporadically

So not a high priority.

> we don't have the infrastructure to build them as part of the ordinary
> build process.

Of course, and we probably never will, unless someone ports gcc to Java and provides open-source copies of every relevant header file. (Fortunately we use JNA for most native integration in the IDE, which avoids the issue.)
Comment 7 pgebauer 2011-09-07 08:45:38 UTC
The profiler DLLs are handled manually on demand. It means, pretxnchangegroup.binary hook is switched off for a commit of DLLs into a specific single repo and it is switched on back after a push-to-* job successfully distribute DLLs to that team repo. Repositories are processed one by one.
Comment 8 Jesse Glick 2011-09-08 12:11:35 UTC
push-to-core-main (IIRC) was just broken by this recently. Anyway any solution which requires manually disabling hooks temporarily is not acceptable. Either the binary files should be uploaded using ExternalBinaries probably in a single ZIP, or *.dll & *.so should be removed from the blacklist (probably bad), or the hook should permit files under, say, 64Kb.

Using ExternalBinaries is probably best since lib.profiler/release/lib/deployed has 2.3Mb of binary content overall. I could make this change if agreed.
Comment 9 Tomas Hurka 2011-09-12 07:43:46 UTC
(In reply to comment #8)
> push-to-core-main (IIRC) was just broken by this recently. Anyway any solution
> which requires manually disabling hooks temporarily is not acceptable. Either
> the binary files should be uploaded using ExternalBinaries probably in a single
> ZIP, or *.dll & *.so should be removed from the blacklist (probably bad), or
> the hook should permit files under, say, 64Kb.
> 
> Using ExternalBinaries is probably best since lib.profiler/release/lib/deployed
> has 2.3Mb of binary content overall. I could make this change if agreed.

As I wrote previously, the easiest way is to remove dll from the list of the forbidden extensions. I understand the need to block zips and jars, but how many problem were prevented by blocking dlls? I would like to see the fix for platform and netbeans launcher binaries first. Even if this is fixed in netbeans, we still have to verify that moving profiler binaries to external binaries does not break VisualVM, integration of Java VisualVM into JDK and integration of VisualVM into Linux distributions.  They cannot use external binaries and need to be completely build-able from sources.
Comment 10 Jesse Glick 2011-10-14 16:52:16 UTC
(In reply to comment #9)
> Even if this is fixed in
> netbeans, we still have to verify that moving profiler binaries to external
> binaries does not break VisualVM, integration of Java VisualVM into JDK and
> integration of VisualVM into Linux distributions.  They cannot use external
> binaries and need to be completely build-able from sources.

http://hg.netbeans.org/binaries/ is effectively a part of sources, merely kept outside the Hg repo for technical SCM reasons (would not be true if using Subversion, for example). When we create a source ZIP the necessary binaries are included by the build process, so they do not need to be downloaded.
Comment 11 Jiri Rechtacek 2013-08-01 12:36:29 UTC
http://hg.netbeans.org/nb-hooks/rev/7ba514fff57c


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo