This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 101067

Summary: I18N - Change scope of l10n.list file
Product: www Reporter: rbalada <rbalada>
Component: Builds & RepositoriesAssignee: rbalada <rbalada>
Status: RESOLVED FIXED    
Severity: blocker CC: jchalupa, jkovalsky, kfrank, mzlamal, rnovak
Priority: P1 Keywords: I18N
Version: 6.x   
Hardware: All   
OS: All   
Issue Type: ENHANCEMENT Exception Reporter:
Bug Depends on:    
Bug Blocks: 71609, 93080    
Attachments: Patch for nbbuild/antsrc/org/netbeans/nbbuild/L10nTask.java
Commit log

Description rbalada 2007-04-15 11:42:09 UTC
We got a request for enhancement to change current process of creating L10N
kits. Current process defines one l10n.list file per whole CVS module. With
current increased complexity of NetBeans IDE and Update Center modules it came
to our attention that it's almost impossible to care about one l10n.list per CVS
module. New solution could be to define l10n.list files at level of respective
NetBeans module (module in terms of NB IDE), that is there would be l10n.list
file at the same level as well known "nbproject" subdirectory. This solution
allows for per NB IDE module granularity.

The only need for change is to update our Ant's custom task
nbbuild/antsrc/org/netbeans/nbbuild/L10nTask.java , which takes list of NB IDE
modules, reduces it to list of toplevel CVS modules and finds l10n.list file in
each toplevel CVS modules. Necessary change is only to avoid the reduction to
CVS modules, that is the process would newly look for l10n.list files in
directories where respective NB IDE modules have their "nbproject" directory.

Advantage I can see at this moment:
1) one list for all processes around one configuration - one list of modules for
building modules, NBMs as well as L10N kit
2) you can have more lists for each configuration (i.e. IDE vs. Update Center)
3) easy change (actually commenting-out a few lines)
4) whi could be a solution for issue 71609

What needs to be done before applying the patch:

This would be a change in process affecting *all* NB IDE modules hosted in
cvs.netbeans.org, so it should be communicated widely. Each and every IDE module
owner/developer must be aware of this change and take an action. Maintenance of
l10n.list files should remain in developer's ownership. This involves at least
creation of l10n.list files at module level.


I'll attach a patch for 
nbbuild/antsrc/org/netbeans/nbbuild/L10nTask.java
Comment 1 rbalada 2007-04-15 11:49:53 UTC
Created attachment 40926 [details]
Patch for nbbuild/antsrc/org/netbeans/nbbuild/L10nTask.java
Comment 2 Ken Frank 2007-05-30 04:02:36 UTC
is this still viewed as enhancement only and at p3 ?
I am guessing that means it does not need to be done for nb6 ?

Its getting close to time in nb6 when l10n kits will be needed which means that
if there are
changes to l10n.lists layout and approach,  it needs to happen soon as to
getting feedback
from developers, being implemented and being tested out, and how it might imact
building
of ml builds or langpacks.

that is, l10n team is requesting existence of l10n kits even earlier than the normal
date for when the kits need to be accurate and complete, which is normally at HR
date -
which could also impact when this might need to be done.

ken.frank@sun.com
Comment 3 rbalada 2007-06-04 12:35:47 UTC
Changing priority to P1, issue type to FEATURE and milestone to 6.0M10.

I have got majority of the work done. Preliminary estimates are that first kits
could be available on Wednesday or Thursday this week (6/6-6/7).

Because this change is major and nontrivial, I'll let the build run in dual mode
for some time to verify the kits are same.
Comment 4 rbalada 2007-06-04 12:37:02 UTC
Marking STARTED
Comment 5 Ken Frank 2007-06-04 16:24:59 UTC
Thanks for starting this.

How many l10n kits will there be and what will be granularity of them ?
I know its not a direct topic of this issue but it will be helpful
to still have some separate kits, even though its all one product.

Perhaps at granularity of the downloadable components seen on installer page ?

ken.frank@sun.com
Comment 6 Ken Frank 2007-06-04 16:30:23 UTC
Could you note some rules for developers about this new approach to l10n.list
so they can start to rewrite/reorganize their current l10n.list files per module ?

Then this could be placed on wiki and shared with dev teams, nb-tech, etc ?


One question - since uc modules will be localized separately from product modules
and on a different schedule, can/should instructions state that files of stable
uc modules
not be put into any l10n.list ?

There is separate discussion about need for separate kit for stable uc modules,
and a separate l10n.list.uc is not the solution that is acceptable, so a subteam
is discussing
how dev can indicate about stable uc files to translate.

But having the uc files in the product kits can result in some problems.

ken.frank@sun.com
Comment 7 rbalada 2007-06-04 18:05:12 UTC
Ken,

basic introduction of the change. Now we have one l10n.list file per each CVS
module. After this change we would have one global l10n.list and bunch of small
l10n.list files for local exceptions

The change of scope is actually scope from several points of view.

Current solution: We have a list of toplevel CVS modules. Each CVS module has
got one (and only one) l10n.list file in the root. Number of CVS modules exactly
matches number of l10n.list files. Each l10n.list file holds wildcard masks for
all necessary submodules within CVS module's directory structure.


New solution: List of submodules matching exact build configuration of NB IDE
(or component/pack/UC list). There's new global nb_all/nbbuild/l10n-global.list
file which holds generic masks covering 80-90% localizable files within these
(sub)modules. In case that some submodule needs exception, you put *new*
l10n.list file under NB IDE module's structure, at exactly the same level where
it has it's 'nbproject' directory. For example for autoupdate/services module
the nbproject directory is 'autoupdate/services/nbproject' and the local
override l10n.list would be 'autoupdate/services/l10n.list' file. Such a file is
used only at time when we need L10Nkit for the 'autoupdate/services' module.
This particular module is specific by having 'libsrc' subdirectory, which
contains some localizable stuff. In this case this local overriding l10n.list
file may contain something like:

<!-- sample autoupdate/services/l10n.list file -->
read global
${l10n-module}/libsrc/**/Bundle.properties
${l10n-module}/libsrc/**/resources/*.gif
${l10n-module}/src/**/*.png
<!-- end of sample -->

Let me read it for you. The line "read global" is a command to load generic
masks from 'nb_all/nbbuild/l10n-global.list'. Loading global generic masks is
optional, mainly when you have very little amount of exceptions an generic masks
are safe to use.
Lines starting '${l10n-module}'. This property reference would be evaluated at
runtime and would be replaced with 'autoupdate/services' string. As a result we
get masks which are valid from the nb_all root.

Why we do all of these?

Let me summarize pluses first:
- unlimited number of L10N kit configurations based on build content 
  configuration system allowing us to have extra kits for IDE, VWP, SOA, UML,
  Update Center etc.
- developer cares only about modules in his charter, his change almost cannot
  affect others
- developer cares only when global generic masks don't fit module's
  localization needs
- if the global l10n.list would be written properly, then majority of modules
  won't need specific local overriding l10n.list file

Minuses:
- in case you put local overriding l10n.list file somewhere in upper level
  module and the file would have been written too liberally, then the kit may
  include data also for deeper level submodules (i.e. autoupdate/l10n.list vs.
  autoupdate/services/l10n.list
- any general minuses when using global mask and local overrides
Comment 8 Ken Frank 2007-06-04 18:35:21 UTC
Rudolf,

thanks for the info/spec;  some followup questions/comments:

- seems dev would need info to know if/when they would
need to create/maintain an override file
I guess just some explanation of what typical kinds of file
names/types is used by l10n-global.list file could be provided to them.

- what about the excludes syntax - this is used by various modules
usually on specific files, sometimes with wildcards -
would this be a case for the override kind of l10n.list file ?  does it allow
exclude syntax
as do the current l10n.lists ?

- is this correct - the override file goes at the nbproject dir level,
but there might be >1 such file for a given top level cvs dir,
since there might be >1 nbproject dirs under that module ?

- from example about sample override l10n.list file

${l10n-module}/libsrc/**/Bundle.properties
${l10n-module}/libsrc/**/resources/*.gif
${l10n-module}/src/**/*.png

--> but shouldn't these patterns for Bundle.properties
and png and gif be part of the global file so that dev
would not need to use these ?

- as to pluses about having extra kits be easier to do -
it sounds good, but for UC kit; how does the 
build system/RE know what parts of what top level
cvs modules have files only meant for uc kit ?
(and uc files should not be in the other kits)

seems that only dev knows about that ?


Thanks - Ken
Comment 9 rbalada 2007-06-05 15:53:44 UTC
Hi Ken,

here go my comments

> - seems dev would need info to know if/when they would
> need to create/maintain an override file
> I guess just some explanation of what typical kinds of file
> names/types is used by l10n-global.list file could be provided to them.

I'll document it at wiki.netbeans.org.

> - what about the excludes syntax ...

I made no changes to syntax. However I did some bugfixing affecting syntax and
hopefully leading to intended behavior. Now it is possible to use '#' character
for commenting out lines at any position in the l10n.list file, or the reference
"read global" is now correctly interpreted each time it is referred.
It's also possible to run the l10nkit build on Windows box and it will produce
same result as on unix box.

> - is this correct - the override file goes at the nbproject dir level,
> but there might be >1 such file for a given top level cvs dir,
> since there might be >1 nbproject dirs under that module ?

First of all there's no recursive search for another l10n.list file. It's always
interpreted locally in current context/directory. The l10n-global.list has been
written in such a way that it does not allow easily descent into submodule (with
one exception where there's a clash between modules core and core/javahelp,
causing a need for local override in core)

> - from example about sample override l10n.list file
>
> ${l10n-module}/libsrc/**/Bundle.properties
> ${l10n-module}/libsrc/**/resources/*.gif
> ${l10n-module}/src/**/*.png
> 
> --> but shouldn't these patterns for Bundle.properties
> and png and gif be part of the global file so that dev
> would not need to use these ?

This is actually tricky question :-). The global file must represent just
reasonable majority of cases, not all possible cases. We have to find a balance
between less than accurate and more than accurate kit (in terms of included data
compared to localization need). In this case the 'libsrc' subdirectory exists in
about 6-8 modules (out of ~280 modules/=codenamebases/ constructing NB IDE).
Regarding the pattern "src/**/*.png" the global file contains less liberal
pattern "src/**/resources/**/*.png". Adding all *.png files is not always good
option (not all of them are need for localization process. 

I decided to not put 'libsrc' into global pattern file. It must not be the best
decision, but helps keep the global file short and not too much liberal.

> - as to pluses about having extra kits be easier to do -
> it sounds good, but for UC kit; how does the 
> build system/RE know what parts of what top level
> cvs modules have files only meant for uc kit ?
> (and uc files should not be in the other kits)
> 
> seems that only dev knows about that ?

Build Engineering always gets list of modules for Update Center enough in
advance. This allows us to build those modules and stage these modules on
Staging Update Center as well as start preparing L10N kit for the list of modules.

When we get the list of modules, we process it down to submodules so as a result
the list reads something like:

<!-- sample list -->
ant/docs,ant/freeform/samples,apisupport/apidocs,ide/projectimport
<!-- end of sample list -->

This list is exactly build configuration in a form which is used for building
module binaries as well for L10Bkit build. This configuration is then put into
nbbuild/build.properties file. This property file can hold number of
configurations. The number is limited just by our ability to maintain those
configurations.

For example there's configuration called 'daily-alpha-nbms'. This configuration
holds list of modules which should be daily refreshed on so called "Development
Update Center". Building l10nkit for this configuration is simple. 
Just run "ant -Dmoduleconfig=daily-alpha-nbms l10n-kit" in the nbbuild module. 

If you're not sure you've got all necessary sources in your working directory
just add "checkout " target before the "l10n-kit" string and you should get all
sources first.
Comment 10 rbalada 2007-06-05 16:09:57 UTC
Status update:

I've committed all necessary changes to run L10Nkit build in dual mode. Starting
tomorrow morning we are going to have two 'same' L10N kits on our internal
storage. Please note that those kits are not 100% same, because current
l10n.list files do not match product content. At this moment I reached a level
where all the differences look reasonable to me and I would need help with
shooting offending differences.

More info tomorrow.
Comment 11 Ken Frank 2007-06-05 16:57:22 UTC
comments to Rudolf's comments in next to last section below start here
and start with lines that start with &&&

Hi Ken,

here go my comments

> - seems dev would need info to know if/when they would
> need to create/maintain an override file
> I guess just some explanation of what typical kinds of file
> names/types is used by l10n-global.list file could be provided to them.

I'll document it at wiki.netbeans.org.

> - what about the excludes syntax ...

I made no changes to syntax. However I did some bugfixing affecting syntax and
hopefully leading to intended behavior. Now it is possible to use '#' character
for commenting out lines at any position in the l10n.list file, or the reference
"read global" is now correctly interpreted each time it is referred.
It's also possible to run the l10nkit build on Windows box and it will produce
same result as on unix box.

> - is this correct - the override file goes at the nbproject dir level,
> but there might be >1 such file for a given top level cvs dir,
> since there might be >1 nbproject dirs under that module ?

First of all there's no recursive search for another l10n.list file. It's always
interpreted locally in current context/directory. The l10n-global.list has been
written in such a way that it does not allow easily descent into submodule (with
one exception where there's a clash between modules core and core/javahelp,
causing a need for local override in core)

&&& suggest documenting this part also for developers


> - from example about sample override l10n.list file
>
> ${l10n-module}/libsrc/**/Bundle.properties
> ${l10n-module}/libsrc/**/resources/*.gif
> ${l10n-module}/src/**/*.png
> 
> --> but shouldn't these patterns for Bundle.properties
> and png and gif be part of the global file so that dev
> would not need to use these ?

This is actually tricky question :-). The global file must represent just
reasonable majority of cases, not all possible cases. We have to find a balance
between less than accurate and more than accurate kit (in terms of included data
compared to localization need). In this case the 'libsrc' subdirectory exists in
about 6-8 modules (out of ~280 modules/=codenamebases/ constructing NB IDE).
Regarding the pattern "src/**/*.png" the global file contains less liberal
pattern "src/**/resources/**/*.png". Adding all *.png files is not always good
option (not all of them are need for localization process. 

I decided to not put 'libsrc' into global pattern file. It must not be the best
decision, but helps keep the global file short and not too much liberal.


&&& I agree, it is a tricky situation.
As to png or filenames in general, I thought that the global file would just be
a list
of filenames/suffixes so that all of them in the cvs src would be obtained.

But I guess its not so easy since cvs src has test files, and other non product
files,
which might have files with that suffix.

As to image files, usually they are in javahelp and all are usually in l10n.list
files from
before; as to others, if they are not included, I guess it can be part of dev review
of global l10n.list and rules about them needing to do override file.

And I think its impt for them to do really the same review of src as they did before
when they created their own l10n.list -- to compare the global list and see
which dirs they still need to either add files to or exclude them.

I think it depends on complexity of global list - can you add here or send me a copy
of it ?


> - as to pluses about having extra kits be easier to do -
> it sounds good, but for UC kit; how does the 
> build system/RE know what parts of what top level
> cvs modules have files only meant for uc kit ?
> (and uc files should not be in the other kits)
> 
> seems that only dev knows about that ?

Build Engineering always gets list of modules for Update Center enough in
advance. This allows us to build those modules and stage these modules on
Staging Update Center as well as start preparing L10N kit for the list of modules.

When we get the list of modules, we process it down to submodules so as a result
the list reads something like:

<!-- sample list -->
ant/docs,ant/freeform/samples,apisupport/apidocs,ide/projectimport
<!-- end of sample list -->

This list is exactly build configuration in a form which is used for building
module binaries as well for L10Bkit build. This configuration is then put into
nbbuild/build.properties file. This property file can hold number of
configurations. The number is limited just by our ability to maintain those
configurations.

For example there's configuration called 'daily-alpha-nbms'. This configuration
holds list of modules which should be daily refreshed on so called "Development
Update Center". Building l10nkit for this configuration is simple. 
Just run "ant -Dmoduleconfig=daily-alpha-nbms l10n-kit" in the nbbuild module. 

If you're not sure you've got all necessary sources in your working directory
just add "checkout " target before the "l10n-kit" string and you should get all
sources first.

&&& that's good news as it seems it wil solve the need for separate uc kits
without needing separate uc l10n.list

but actually the kit that is needed is for stable uc modules only; not all uc
modules.

and what is on stable can change from time to time.
will this solution be able to provide a kit for that ?


But am assuming even for this case dev will need to review the global list and
see if any
exeption files are needed ?
Comment 12 Jesse Glick 2007-06-06 21:07:33 UTC
I would recommend

**/*src/**/Bundle.properties

to catch libsrc and similar dir names.
Comment 13 rbalada 2007-06-13 16:42:34 UTC
Created attachment 43634 [details]
Commit log
Comment 14 rbalada 2007-06-13 16:43:06 UTC
Major infrastructure pushed into production
Comment 15 Ken Frank 2007-07-20 03:45:07 UTC
whats the plan to communicate to dev about these changes and the parts they need to do ?
(special exclude/include lists).  Even though there is new process and global list, dev still owns
it.

I don't think there is a category that such a task can be filed on this  - what's your suggestion
on how to get this on  dev plate and tracked (the dev part) ?

since there is a new i18n readiness milestone (if you and dev approve) that has early l10n kit by beta 2 -
but this means not just the already going kits you are publishing but that dev has done at least some work
on their lists -- but it does not have to be 100% accurate.

whereas the usual l10n kit complete milestone means that kit is 100% accurate

and that all modules are covered by RE scripts as to being included in kits

and that 

stable uc modules are not in product kit

product kit has no stable uc or other uc only module files
(these last 2 are part of uc kit issue71609

--> should another issue be filed as to having kit include all nb6 tarfiles (they dont need to be correct,
just that all the tars are there -- or is this complete as per this issue ?


ken.frank@sun.com