I have found further consequences of this issue. This issue actually blocks
build publishing system from working. I cannot workaround this issue w/o
rewriting core part of build publishing script. After discussion with Jack I
promote this issue to P1.

Looking into this issue , will be updating the issue with the status as soon as
possible .

Jobin.

I had filed a issue to operations the issue no is 35501 , will be updating this
issue with the status .

Jobin.

The operations have worked on the issue , and was able to upload the file "ls
-ld 4_1/daily/200503131900/netbeans*-link.txt", please verify if this is still a
issue .

Jobin.

Created attachment 20834 [details]
SSH verbose session log

The issue is still there.

It looks like the custom "shell" does escaping of wildcards. I mean it changes
"*" to "\*" before passing the command to regular shell.

How did the build publishing system work on the staged version of the software 
then?

Unfortunatelly the ssh/scp to /download was not working properly before upgrade.
I haven't had even role-request access to sun-netbeans project on extranet and
my ssh/scp issue probably got lost amongst other issues Jack was solving. We
didn't get the staging box's ssh/scp access to state where it could be worth to
test it with production publishing tools.

BTW: can we get back the version of "custom shell" which was there before upgrade?

IMHO the new version of "custom shell" comes with more unrequested restrictions
and w/o any useful additional features (I don't count allowing 'mkdir' command
to additional features as we used to use recursive uploads instead).

It is bad that we did not test this thoroughly on the staging box.  In any case,
we still need it fixed ASAP.  This is blocking publishing of any builds.

I have an issue open internally to get some investigation here but I could use 
an explanation as to how not being able to use wildcards in ssh commands 
prevents the build publishing system from working.

I don't know how this could help you fix the issue, but here we go.
The build publishing system uses 'ls' command to list available files. To list
those files the script uses set of wildcards. The result lists are then
registered in downloads database (not @netbeans.org). Updating the script to
avoid using wildcards requires 2-3 days of shell scripting work. This is good
reason to call this issue major regression.

Please do not assign the issue to me before you want me to test the fix. From my
point of view you have got enough information for fixing and obviously to keep
it assigned to 'support' until the fix is in place.

Was there any good reason to update the custom shell assigned to the 'upload'
account?

I forgot to mention that even the publishing script update is 2-3 days of shell
scripting working it will then require update with every new deliverable file.
In  this case wildcards are used to save copy-pasting of the almost-the-same
code for each and every possible deliverable. In year total it could become
15-20 days of shell scripting work if we go w/o wildcards. 

Does this sound like pretty good business justification?

We have disabled globbing in the upload shell for security reasons and that is 
not going to change, which is why I asked about the process you are using.

Why could the script (and command lines for that matter) not be easily amended 
to list the files, recursively if necessary, and egrep for the files needed?

Brian,

there is simply no security risk in doing 'ls *'. This is P1 REGRESSION and
please fix it ASAP by allowing 'ls' command to be used with wildcards. Actualy I
don't need to do 'rm *', what is probably the mentioned security reason.

Just in case there is even minimal security risk related with current version of
'ls' command installed on the box, I would like to see i.e. SERT response or
document on how-to issue a security attack with 'ls' command in connection with
wildcards (where wildcards is a must for successful security breach).

I see no reason to accomodate work-labour workaround into publishing scripts,
because there's no reason to keep this regression in place as it is now.

Collab, any update ?

Regardless of how "*" is used or not, this is changed behaviour in a critical SC
component.  We need this fixed ASAP.  Please respond.

I have updated the internal issue on the reason why this is a critical issue and
the importance why this issue has to be fixed as soon as possible , awaiting the
engineers response , will be updating as soon a response is received .

The workaround to your scripting would be to use:

ssh upload@cvs.netbeans.org "ls -ld 4_1/daily/200503131900/" | \
   egrep -e 'netbeans.*-link.txt'

instead of:

ssh upload@cvs.netbeans.org "ls -ld 4_1/daily/200503131900/netbeans*-link.txt"

A find-and-replace in the scripts should suffice to make this change.

It would be nice to have a _technical_ reason why our workaround would be effort 
intensive to implement or why it would not work at all.  Please call the support 
hotline at 650.228.2561 if you would like to discuss in person.

Wildcard use has been disabled following the fundamental security principle of 
least privilege.  (Searching www.cert.org for the words "least privilege" 
provides hundreds of hits, if you are looking for information on this principle.
)  The concept is such that capabilities are provided only if they are 
absolutely necessary.  We do not need to provide exploit information to justify 
our decision to disable globbing.  

We dispute this is a regression given the wildcard functionality was an 
undocumented feature in the product.  Also, this situation is only critical now 
because this was not tested while the stage system was available.

I've spoken to Rudolf, he's home and offline and not able to chekc the details
here.  However he is very adamant that wildcards are the only real soloution
here.  The workaround will require some days work now, and updates at least once
a month.  This is not a satisfactory option from his POV.

"Least privilege" could also imply removal of the entire ssh shell we have, so I
don't really see that as a valid justification here.  Of course it is good
practice to always allow least privilege, until that affects the required
functionality.

Yes, it is critical because it wasn't tested properly.  Yes, it is not a
regression since Collab didn't know we were using this functionality.  Moving
on, we need it back.

Rudolf is adamant wildcard utilization is the only solution and we have provided 
a solution that makes use of the wildcard within the egrep command.  It would be 
of great help if we could be edified why, specifically, the grep solution would 
not work.  To date we have been only been told, "it is necessary", but there has 
been no justification.  Given wildcards are not going to be available, the 
scripts need to be modified and if portions of the scripts could be included 
here, maybe we could begin to understand the adjustments that need to be made 
such that they can be adapted with as little effort as possible.

Given Rudolf's explanation of how the system works at present ("The build 
publishing system uses 'ls' command to list available files. To list those files 
the script uses set of wildcards."), the grep solution will work.

We have mobilized some of the best people at CollabNet to help with this and 
want to find a solution as urgently as you do.  Please provide us with the 
details of your scripts such that we can assist in adapting them.

I was pushed to fix it on my side as there's much better way how to utilize my
work time than handling this ridiculous restriction. Also time constraint was
taken into account. It is probable my fix could be ready much earlier than yours.

We recently moved out from Collabnet's infrastructure