Bug 98701 - Allow to open very big files?
Allow to open very big files?
Status: NEW
Product: editor
Classification: Unclassified
Component: -- Other --
8.0.2
All All
: P3 with 6 votes (vote)
: 6.x
Assigned To: Milutin Kristofic
issues@editor
: PERFORMANCE
: 28662 154326 (view as bug list)
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2007-03-22 16:52 UTC by belur_1d
Modified: 2017-08-15 14:31 UTC (History)
11 users (show)

See Also:
Issue Type: ENHANCEMENT
:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description belur_1d 2007-03-22 16:52:56 UTC
Hi

I have been using netbeans for almost a year now.  There is one issue
which really gets to be a slight pain sometimes. When i have to open
very big files, nb always popups the warning asking the user
whether he wud like to open it or not. if it opens it will cause
Out of Memory error. 

1> Here i dont like the idea. See, I have to open the file in order to 
work on it. There is no use showing the popup.

2> I think even id there was a reason to put it in, please look at this way.
As we know, it has to do with memory  at the end. What if my machine
has a good RAM .Here u see , there might be no issue at all to handle the file

For example netbeans can detect if possible, the size that wud be avaliable to
it via the Runtime class API in java and do an intelligent guess as to whether
it can handle the file or not ?

What I am trying to say is essentially:

1> I am using netbeans as the IDE of my choice and cant switch to another
just to open the big file.

2> Sometimes, this warning is useless

My Request:

Please remove the popup OR make it configurable somewhere top show the popup or not

-swaroop belur
Comment 1 Marian Mirilovic 2007-03-26 08:42:32 UTC
I think we could add button (checkbox) "Don't show again".

I am not sure about the component, so Petr feel free to reassign ...
Comment 2 Petr Nejedly 2007-03-26 13:46:39 UTC
DataEditorSupport in loaders module.
Anyway, it's just a heuristic.
And no, you don't need to open the file to work with it. There are files that
can really take the IDE to its knees and you won't be able to work with it
anyway. What's the threshold is hard to decide, it depends on the file size,
file type, installed featureset and so on.
Comment 3 belur_1d 2007-03-26 14:02:04 UTC
Hi 

This thought just occured to me. I dont know if you guys find it correct ?
If the file is too big and say its jav file , can nb offer a way to
open the file in say "reduced -feature" mode . For example in text mode. This will
be the simplest. Somewhere a right click against the file and in the pop-up window
also in some manner -i.e provide a means to open the file in alternate manner

Ofcourse you can warn the user, that file will be opened in a reduced feature
mode or something like that ?

Like  may be for java file say

1> skip syntax highlighting check
2> skip code folding check and expand/collapse

Somethings like the above and may be nb team can think of more.

Some option like the above + say keep simple/basic refactoring 
options like rename for example , i think will help nb cope
with v ery big files?


-swaroop belur


Comment 4 Jaroslav Tulach 2007-03-27 08:39:14 UTC
I'd be happy to implement anything guys from editor tell me to handle big 
files. My ideas would include a special EditorKit and mimetype for big files 
that would allow editing of memory mapped, etc. So editor guys, it is up to 
you. When you find a way to open big file, let me know I can modify 
openide/text or openide/loaders.
Comment 5 Vitezslav Stejskal 2007-03-27 09:19:51 UTC
I am not sure why we can't open big files, besides of the commonly known thruth
that big files need a lot of memory of course. But how big are big files and how
much memory they need in the IDE? If we are serious about this we should start
measuring the memory consumption for some common file types - eg text/plain and
text/x-java for the beginning - and look at what features take what amount of
memory.

I agree that poping up a dialog is not very useful and crashing with OOME is
even less useful. I always thought that it would be generally handy if people
were able to open files in alternate editors, such as opening a struts config
file as a plain xml or a java file as a plain text. If we had that option we
could warn a user when opening a big file and suggest opening it in the plain
text editor. We could even try to catch a potential OOME if the user chooses to
open the file in the file's own fully fledged editor and fall back to the plain
text editor automatically. That would of course need the plain text editor to
work reliably with files of pretty much any size, which brings us back to the
beginning. If the plain text editor is only marginally more memory efficient
then say java editor, it will not provide for a good fallback. But there might
be a way how to improve it or perhaps how to improve all editors. So, we really
need some numbers.
Comment 6 Petr Nejedly 2007-03-27 10:10:32 UTC
Let me state first that I'm not trying to dismiss this issue. Here I'm just
collecting the data necessary for finding the balance between potential
implementation complexity(cost) and the benefits from implementing it.

OK, before we get technical, let's look at the use cases. We don't like to
invest into complicated infrastructure to support next-to-nonexistent use cases,
do we?
So what kind of large files is the user likely to encounter?

The biggest java file I have ever seen was X.java from ProjectX (video
processing suite), accounting for 500KB and 17.000 lines. The file is basically
ineditable, not because of the editor but because of the lack of structure.
There's no technical reason for not splitting the file/code among more classes.
In fact, that file was obviously a result of architectural and technical
incompetency on steroids, yet our editor handled it.

For xml, I believe that there are some large WSDL files out there that may need
editing, yet we are likely to hit some other wall (tree view updates) before
editor's OOME.

But generally, I don't believe humans should write or edit such large files
(view them to verify how they were generated, right, but not edit) or at least
not commonly.

Now to the technical issues. Our editor (plain) has nontrivial (>100B) overhead
per line (not per byte) of the text. If you add features (annotations,
syntax/semantic highlighting, support for navigation), the figure multiplies
(sorry, no exact numbers at hand).
Even if we managed to take the document off-memory with smart indexing, the rest
of the IDE would consume the memory much faster than how we spared it.

Regarding catching the OOME - this is nice idea on the first sight but has a lot
of problems. Other thread may be the one that fails and it is even very likely
in this case - editor just opens the file OK with a plenty of headroom
initially, but all kinds of asynchronous/delayed/on demand stuff kick in and
consume much more memory later. Not talking about the usability of the IDE near
its memory limit.

I tried to agree with the limit proportional to available memory, but found
myself writing phrases like "power user", which would, in fact, benefit more
from "don't warn again" check box.

Also, I realized we can quite easily provide simple _viewer_ for files of
virtually any size (output window handle it right with partial mmaps()).
The question is whether it would cover enough use cases.
Comment 7 Miloslav Metelka 2007-03-27 15:48:34 UTC
My post collided with Petr's one, so just briefly:
currently there is ~200 bytes per line (includes positions, line-element,
line-view). Plus the number-of-characters-on-line * 2 bytes-per-char.
Questions affecting design:
1. Should big-file editor be just read-only viewer of the file?
2. Should BFE mmap its content?
  Problem: Swing convention is that Document.getText() returns '\n' line
separators only. For mmap-ed file there would have to be a structure that would
allow conversion.
BTW does anyone know how does the byte->char conversion work for mmap-ed files
and if there's any support for an offset conversion work e.g. in utf8 2 bytes
may become single character etc.?

IMHO there will always be a certain overhead per-line so for a file with too
many lines the IDE still may run out of memory.
In case we agree on read-only viewer the impl. from output window shoul IMHO be
reused.
Comment 8 Vitezslav Stejskal 2007-03-27 23:12:57 UTC
Looks like this hasn't been the first time we talk about it - issue #28662.
Duplicate?
Comment 9 Miloslav Metelka 2007-03-28 12:22:56 UTC
*** Issue 28662 has been marked as a duplicate of this issue. ***
Comment 10 Miloslav Metelka 2007-03-28 12:29:35 UTC
I have duplicated the older issue 28662 to this one since this one imho contains
more useful information.
One thing I did not realized (but present in the notes of the other issue):
In order to support the not-fully-in-memory documents
we will likely be forced to update constraints for certain
functionality in the editor. For example current word-match Ctrl-K/L
scans the contents of the current doc plus all other opened docs. This
needs to be performed in fractions of second.
So certain functionality would IMHO have to be disabled for not-fully-in-memory
docs.
Comment 11 Milutin Kristofic 2012-11-29 18:11:29 UTC
*** Bug 154326 has been marked as a duplicate of this bug. ***
Comment 12 sproger1 2015-02-02 20:31:53 UTC
The decision should be up to the user via a pop-up dialog and have the option to save the decision (Do Not ask again checkbox; along with an entry in Tools-->Options-->Editor... entry where it can be changed).

Option 1) 'The user can choose' to cancel the task and not open the document.

Option 2) 'The user can choose' to ignore the warning and open the document in Full editor mode at their own risk (Thankfully we can already achieve this by setting a system property '-J-Dorg.openide.text.big.file.size=25', Personally I currently use 25mb to reduce the occurrence of this issue).

Option 3) User can choose an external editor (command) and netbeans will launch it with the selected file as an argument. Similar to the QuickOpener or CoolEditor plugin.

Option 4) Open the file in a plain text editor/viewer, use mmap (to avoid loading the entire file into memory), and disable all editor features that require the entire file to be loaded into memory (thus significantly reducing the time to open/load the text viewing window). Any highlighting... features should be done only on the currently visible portion of the document. Allow find/search, but use progress bars... to allow user to cancel those possibly lengthy operations. IMHO read-only mode is fine, but if enabling editing of a mmap based plain text file can be accomplished without too much overhead, then I see no reason not to add it as a nice bonus feature. Not very many free editors out there handle large text files very well, so It would be great if netbeans could support this as an open source java based solution (that others can use and build on using netbeans RCP framework).


It is NOT a RARE USE-CASE to want to open/view a large file, but I agree that it is rare to want to EDIT a large (10mb+) file. At a minimum, netbeans should be able to do this without OOME or crawling to a stop. Netbeans is not just used for editing small java files!!! Netbeans is an RCP client with countless use-cases.

I am currently using netbeans for an RCP and will likely have to end up working around this issue by writing my own memory mapped text viewer for opening large text files (I am likely not the only one that will end up having to reinvent the wheel to do this, and wish there was already some built-in support for this). Many of the editor... packages are private/final... so I will likely have to build my own custom version of netbeans from source....(pain...)... create my own delegate for OpenAction...;


Many other users have come across this issue for almost 10 years now and this issue is still not addressed. For every one person that reports this, there are likely hundreds that come across it, but don't report.
Comment 13 bubergeson 2015-07-08 17:02:52 UTC
I would like to add my voice to this issue as well. This is very annoying.  The various solutions already presented seem reasonable.  In the meantime isn't there anyway around it such as opening it for viewing only?  My file is 6MB big.  I tried getting a ViewCookie for a file with a .log extension and it was not supported.
Comment 14 twolf2919 2015-12-05 19:31:54 UTC
I'd like to add my vote to better handling of large files.  Specifically, I regularly have to handle large XML files - on the order of 10-100mb - and that brings NB to its knees on an i7 machine with 16GB of memory.  Since the CPU is pegged at near 100%, I'm guessing it's not a memory issue, but NB trying to 'grok' the XML structure.

Why do I look at large XML files?  Usually it's to find a single data item and check its surrounding data, in order to figure out why my app is having trouble with ingesting/processing the file.

Today, I had to kill the NB process when I tried to open a 33MB XML file.  I had hoped to use NB to more easily traverse it's "structure" (e.g. see the end tag of a highlighted begin tag) - since I needed to determine what parts of this huge XML I was interested in and write a SAX parser fort those parts.

Someone mentioned a "viewer" - that would be cool by me - as long as that viewer can apply formatting to the file (e.g. indentation for XML files) and let me jump between the beginning of a code block or xml tag and its end.
Comment 15 sproger1 2016-07-14 14:55:36 UTC
Common editor team, please give Jaroslav Tulach(see comment above) the go ahead to implement the special memory mapped EditorKit and mimetype, and to modify openide/text and/or openide/loaders as needed.

The approach of changing(overriding) the mimetype(to say 'text/large') for large(user configurable by system property) text files, and using a special EditorKit to view/edit such files seems like the best approach.

Having a special Editor and mimetype would make opening large files work quick and smooth since the text editing features of original mime type will be disabled (i.e. parsing, syntax highlighting, auto-complete... and general loading into memory will all be disabled). Just provide a simple plain vanilla text editor for large text files.


This would be a really cool/useful feature. And make the IDE much less annoying to use, when you want to view a large text/log file or xml quickly.
Comment 16 mdr 2017-08-15 14:31:23 UTC
Our typical use case:
- Large log files.
- More seldom XML data files.
- Mostly not in the project tree. I.e. opened via the command line.

It would be enough to have the pop-up offering the simplified editor as an alternative.

Simple editing would be nice, though. E.g. to insert markers, comments or to truncate the file and cut out the relevant parts, or to modify small portions of a large input file for testing purposes.

A button, to re-open the truncated file in a fully fledged editor would be a breeze.

Hope, this helps understanding use-cases.


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2014, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo