opensourcemirror
/
neovim
mirror of https://github.com/neovim/neovim.git
1
0
Fork 0
Code Issues Releases Wiki Activity

Merge #4486 'refactor: Remove maxmem, maxmemtot options' 

After this change we never release blocks from memory (in practice it
never happened because the memory limits are never reached).  Let the OS
take care of that.

---

On today's systems the 'maxmem' and 'maxmemtot' values are huge (4+ GB)
so the limits are never reached in practice, but Vim wastes a lot of
time checking if the limit was reached.

If the limit is reached Vim starts saving pieces of the swap file that were in
memory to the disk. Said in a different way: Vim implements its own
memory-paging mechanism. This is unnecessary and inefficient since the
operating system already has virtual memory and will swap to the disk if
programs start using too much memory.

This change does...

1. Reduce the number of config options and need for documentation.
2. Make the code more efficient as we don't have to keep track of memory
   usage nor check if the memory limits were reached to start swapping
   to disk every time we need memory for buffers.
3. Simplify the code. Once memfile.c is simple enough it could be
   replaced by actual operating system memory mapping (mmap,
   MemoryViewOfFile...). This change does not prevent Vim to recover
   changes from swap files since the swapping code is never triggered
   with the huge limits set by default.
This commit is contained in:
Justin M. Keyes 2018-05-02 10:09:51 +02:00
parent c9e340b7be d0b08f32f5
commit e46534b423
16 changed files with 18 additions and 248 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

24
runtime/doc/options.txt
View File

@ -3924,16 +3924,6 @@ A jump table for the options with a short description can be found at |Q_op|.
because the 'w' is used before the next mapping is done. See also
|key-mapping|.
*'maxmem'* *'mm'*
'maxmem' 'mm' number (default between 256 to 5120 (system
dependent) or half the amount of memory
available)
global
Maximum amount of memory (in Kbyte) to use for one buffer. When this
limit is reached allocating extra memory for a buffer will cause
other memory to be freed. The maximum usable value is about 2000000.
Use this to work without a limit. Also see 'maxmemtot'.
*'maxmempattern'* *'mmp'*
'maxmempattern' 'mmp' number (default 1000)
global
@ -3947,20 +3937,6 @@ A jump table for the options with a short description can be found at |Q_op|.
"\(.\)*" on a very long line. ".*" works much better.
Vim may run out of memory before hitting the 'maxmempattern' limit.
*'maxmemtot'* *'mmt'*
'maxmemtot' 'mmt' number (default between 2048 and 10240 (system
dependent) or half the amount of memory
available)
global
Maximum amount of memory in Kbyte to use for all buffers together.
The maximum usable value is about 2000000 (2 Gbyte). Use this to work
without a limit.
On 64 bit machines higher values might work. But hey, do you really
need more than 2 Gbyte for text editing? Keep in mind that text is
stored in the swap file, one can edit files > 2 Gbyte anyway. We do
need the memory to store undo info.
Also see 'maxmem'.
*'menuitems'* *'mis'*
'menuitems' 'mis' number (default 25)
global

2
runtime/doc/quickref.txt
View File

@ -768,9 +768,7 @@ Short explanation of each option: *option-list*
'maxcombine' 'mco' maximum nr of combining characters displayed
'maxfuncdepth' 'mfd' maximum recursive depth for user functions
'maxmapdepth' 'mmd' maximum recursive depth for mapping
'maxmem' 'mm' maximum memory (in Kbyte) used for one buffer
'maxmempattern' 'mmp' maximum memory (in Kbyte) used for pattern search
'maxmemtot' 'mmt' maximum memory (in Kbyte) used for all buffers
'menuitems' 'mis' maximum number of items in a menu
'mkspellmem' 'msm' memory used before |:mkspell| compresses the tree
'modeline' 'ml' recognize modelines at start or end of file

6
runtime/doc/recover.txt
View File

@ -63,10 +63,8 @@ following ones in your vimrc:
This is also very handy when editing files on floppy. Of course you will have
to create that "tmp" directory for this to work!
For read-only files, a swap file is not used. Unless the file is big, causing
the amount of memory used to be higher than given with 'maxmem' or
'maxmemtot'. And when making a change to a read-only file, the swap file is
created anyway.
For read-only files, a swap file is not used right away. The swap file is
created only when making changes.
The 'swapfile' option can be reset to avoid creating a swapfile. And the
|:noswapfile| modifier can be used to not create a swapfile for a new buffer.

2
runtime/doc/usr_11.txt
View File

@ -294,8 +294,6 @@ If you really don't want to see this message, you can add the 'A' flag to the
disk.
'updatetime' Timeout after which the swap file is flushed to disk.
'directory' List of directory names where to store the swap file.
'maxmem' Limit for memory usage before writing text to the swap file.
'maxmemtot' Same, but for all files in total.
==============================================================================

12
runtime/doc/vi_diff.txt
View File

@ -38,18 +38,6 @@ Information for undo and text in registers is kept in memory, thus when making
undo levels and the text that can be kept in registers. Other things are also
kept in memory: Command-line history, error messages for Quickfix mode, etc.
Memory usage limits
-------------------
The option 'maxmem' ('mm') is used to set the maximum memory used for one
buffer (in kilobytes). 'maxmemtot' is used to set the maximum memory used for
all buffers (in kilobytes). The defaults depend on the system used.
These are not hard limits, but tell Vim when to move text into a swap file.
If you don't like Vim to swap to a file, set 'maxmem' and 'maxmemtot' to a
very large value. The swap file will then only be used for recovery. If you
don't want a swap file at all, set 'updatecount' to 0, or use the "-n"
argument when starting Vim.
==============================================================================
2. The most interesting additions *vim-additions*

2
runtime/doc/vim_diff.txt
View File

@ -402,6 +402,8 @@ Other options:
*'imactivatekey'* *'imak'*
*'imstatusfunc'* *'imsf'*
*'macatsui'*
'maxmem' Nvim delegates memory-management to the OS.
'maxmemtot' Nvim delegates memory-management to the OS.
*'restorescreen'* *'rs'* *'norestorescreen'* *'nors'*
'shelltype'
*'shortname'* *'sn'* *'noshortname'* *'nosn'*

4
runtime/optwin.vim
View File

@ -996,10 +996,6 @@ call append("$", "updatecount\tnumber of characters typed to cause a swap file u
call append("$", " \tset uc=" . &uc)
call append("$", "updatetime\ttime in msec after which the swap file will be updated")
call append("$", " \tset ut=" . &ut)
call append("$", "maxmem\tmaximum amount of memory in Kbyte used for one buffer")
call append("$", " \tset mm=" . &mm)
call append("$", "maxmemtot\tmaximum amount of memory in Kbyte used for all buffers")
call append("$", " \tset mmt=" . &mmt)
call <SID>Header("command line editing")

4
src/nvim/globals.h
View File

@ -540,10 +540,6 @@ EXTERN buf_T *curbuf INIT(= NULL); // currently active buffer
#define FOR_ALL_BUFFERS_BACKWARDS(buf) \
for (buf_T *buf = lastbuf; buf != NULL; buf = buf->b_prev)
/* Flag that is set when switching off 'swapfile'. It means that all blocks
* are to be loaded into memory. Shouldn't be global... */
EXTERN int mf_dont_release INIT(= FALSE); /* don't release blocks */
/*
* List of files being edited (global argument list). curwin->w_alist points
* to this when the window is using the global argument list.

152
src/nvim/memfile.c
View File

@ -61,8 +61,6 @@
#define MEMFILE_PAGE_SIZE 4096 /// default page size
static size_t total_mem_used = 0; /// total memory used for memfiles
#ifdef INCLUDE_GENERATED_DECLARATIONS
# include "memfile.c.generated.h"
#endif
@ -99,7 +97,6 @@ memfile_T *mf_open(char_u *fname, int flags)
mfp->mf_used_first = NULL; // used list is empty
mfp->mf_used_last = NULL;
mfp->mf_dirty = false;
mfp->mf_used_count = 0;
mf_hash_init(&mfp->mf_hash);
mf_hash_init(&mfp->mf_trans);
mfp->mf_page_size = MEMFILE_PAGE_SIZE;
@ -136,25 +133,6 @@ memfile_T *mf_open(char_u *fname, int flags)
mfp->mf_neg_count = 0;
mfp->mf_infile_count = mfp->mf_blocknr_max;
// Compute maximum number of pages ('maxmem' is in Kbytes):
// 'mammem' * 1Kbyte / page-size-in-bytes.
// Avoid overflow by first reducing page size as much as possible.
{
int shift = 10;
unsigned page_size = mfp->mf_page_size;
while (shift > 0 && (page_size & 1) == 0) {
page_size /= 2;
--shift;
}
assert(p_mm <= LONG_MAX >> shift); // check we don't overflow
assert((uintmax_t)(p_mm << shift) <= UINT_MAX); // check we can cast safely
mfp->mf_used_count_max = (unsigned)(p_mm << shift) / page_size;
if (mfp->mf_used_count_max < 10)
mfp->mf_used_count_max = 10;
}
return mfp;
}
@ -198,7 +176,6 @@ void mf_close(memfile_T *mfp, bool del_file)
// free entries in used list
for (bhdr_T *hp = mfp->mf_used_first, *nextp; hp != NULL; hp = nextp) {
total_mem_used -= hp->bh_page_count * mfp->mf_page_size;
nextp = hp->bh_next;
mf_free_bhdr(hp);
}
@ -223,12 +200,9 @@ void mf_close_file(buf_T *buf, bool getlines)
if (getlines) {
// get all blocks in memory by accessing all lines (clumsy!)
mf_dont_release = true;
for (linenr_T lnum = 1; lnum <= buf->b_ml.ml_line_count; ++lnum) {
for (linenr_T lnum = 1; lnum <= buf->b_ml.ml_line_count; lnum++) {
(void)ml_get_buf(buf, lnum, false);
}
mf_dont_release = false;
// TODO(elmart): should check if all blocks are really in core
}
if (close(mfp->mf_fd) < 0) { // close the file
@ -246,13 +220,6 @@ void mf_close_file(buf_T *buf, bool getlines)
/// and the size it indicates differs from what was guessed.
void mf_new_page_size(memfile_T *mfp, unsigned new_size)
{
// Correct the memory used for block 0 to the new size, because it will be
// freed with that size later on.
if (new_size >= mfp->mf_page_size) {
total_mem_used += new_size - mfp->mf_page_size;
} else {
total_mem_used -= mfp->mf_page_size - new_size;
}
mfp->mf_page_size = new_size;
}
@ -262,10 +229,7 @@ void mf_new_page_size(memfile_T *mfp, unsigned new_size)
/// @param page_count Desired number of pages.
bhdr_T *mf_new(memfile_T *mfp, bool negative, unsigned page_count)
{
// If we reached the maximum size for the used memory blocks, release one.
// If a bhdr_T is returned, use it and adjust the page_count if necessary.
// If no bhdr_T is returned, a new one will be created.
bhdr_T *hp = mf_release(mfp, page_count); // the block to be returned
bhdr_T *hp = NULL;
// Decide on the number to use:
// If there is a free block, use its number.
@ -273,34 +237,22 @@ bhdr_T *mf_new(memfile_T *mfp, bool negative, unsigned page_count)
// a positive number.
bhdr_T *freep = mfp->mf_free_first; // first free block
if (!negative && freep != NULL && freep->bh_page_count >= page_count) {
// If the block in the free list has more pages, take only the number
// of pages needed and allocate a new bhdr_T with data.
//
// If the number of pages matches and mf_release() did not return a
// bhdr_T, use the bhdr_T from the free list and allocate the data.
//
// If the number of pages matches and mf_release() returned a bhdr_T,
// just use the number and free the bhdr_T from the free list
if (freep->bh_page_count > page_count) {
if (hp == NULL) {
hp = mf_alloc_bhdr(mfp, page_count);
}
// If the block in the free list has more pages, take only the number
// of pages needed and allocate a new bhdr_T with data.
hp = mf_alloc_bhdr(mfp, page_count);
hp->bh_bnum = freep->bh_bnum;
freep->bh_bnum += page_count;
freep->bh_page_count -= page_count;
} else if (hp == NULL) { // need to allocate memory for this block
} else { // need to allocate memory for this block
// If the number of pages matches use the bhdr_T from the free list and
// allocate the data.
void *p = xmalloc(mfp->mf_page_size * page_count);
hp = mf_rem_free(mfp);
hp->bh_data = p;
} else { // use the number, remove entry from free list
freep = mf_rem_free(mfp);
hp->bh_bnum = freep->bh_bnum;
xfree(freep);
}
} else { // get a new number
if (hp == NULL) {
hp = mf_alloc_bhdr(mfp, page_count);
}
hp = mf_alloc_bhdr(mfp, page_count);
if (negative) {
hp->bh_bnum = mfp->mf_blocknr_min--;
mfp->mf_neg_count++;
@ -341,13 +293,7 @@ bhdr_T *mf_get(memfile_T *mfp, blocknr_T nr, unsigned page_count)
// could check here if the block is in the free list
// Check if we need to flush an existing block.
// If so, use that block.
// If not, allocate a new block.
hp = mf_release(mfp, page_count);
if (hp == NULL) {
hp = mf_alloc_bhdr(mfp, page_count);
}
hp = mf_alloc_bhdr(mfp, page_count);
hp->bh_bnum = nr;
hp->bh_flags = 0;
@ -514,8 +460,6 @@ static void mf_ins_used(memfile_T *mfp, bhdr_T *hp)
} else {
hp->bh_next->bh_prev = hp;
}
mfp->mf_used_count += hp->bh_page_count;
total_mem_used += hp->bh_page_count * mfp->mf_page_size;
}
/// Remove block from memfile's used list.
@ -530,82 +474,6 @@ static void mf_rem_used(memfile_T *mfp, bhdr_T *hp)
mfp->mf_used_first = hp->bh_next;
else
hp->bh_prev->bh_next = hp->bh_next;
mfp->mf_used_count -= hp->bh_page_count;
total_mem_used -= hp->bh_page_count * mfp->mf_page_size;
}
/// Try to release the least recently used block from the used list if the
/// number of used memory blocks gets too big.
///
/// @return The block header, when release needed and possible.
/// Resulting block header includes memory block, so it can be
/// reused. Page count is checked to be right.
/// NULL, when release not needed, or not possible.
/// Not needed when number of blocks less than allowed maximum and
/// total memory used below 'maxmemtot'.
/// Not possible when:
/// - Called while closing file.
/// - Tried to create swap file but couldn't.
/// - All blocks are locked.
/// - Unlocked dirty block found, but flush failed.
static bhdr_T *mf_release(memfile_T *mfp, unsigned page_count)
{
// don't release while in mf_close_file()
if (mf_dont_release)
return NULL;
/// Need to release a block if the number of blocks for this memfile is
/// higher than the maximum one or total memory used is over 'maxmemtot'.
bool need_release = (mfp->mf_used_count >= mfp->mf_used_count_max
|| (total_mem_used >> 10) >= (size_t)p_mmt);
/// Try to create swap file if the amount of memory used is getting too high.
if (mfp->mf_fd < 0 && need_release && p_uc) {
// find for which buffer this memfile is
buf_T *buf = NULL;
FOR_ALL_BUFFERS(bp) {
if (bp->b_ml.ml_mfp == mfp) {
buf = bp;
break;
}
}
if (buf != NULL && buf->b_may_swap) {
ml_open_file(buf);
}
}
/// Don't release a block if:
/// there is no file for this memfile
/// or
/// the number of blocks for this memfile is lower than the maximum
/// and
/// total memory used is not up to 'maxmemtot'
if (mfp->mf_fd < 0 || !need_release)
return NULL;
bhdr_T *hp;
for (hp = mfp->mf_used_last; hp != NULL; hp = hp->bh_prev)
if (!(hp->bh_flags & BH_LOCKED))
break;
if (hp == NULL) // not a single one that can be released
return NULL;
// If the block is dirty, write it.
// If the write fails we don't free it.
if ((hp->bh_flags & BH_DIRTY) && mf_write(mfp, hp) == FAIL)
return NULL;
mf_rem_used(mfp, hp);
mf_rem_hash(mfp, hp);
/// Make sure page_count of bh_data is right.
if (hp->bh_page_count != page_count) {
xfree(hp->bh_data);
hp->bh_data = xmalloc(mfp->mf_page_size * page_count);
hp->bh_page_count = page_count;
}
return hp;
}
/// Release as many blocks as possible.

3
src/nvim/memfile_defs.h
View File

@ -56,7 +56,6 @@ typedef struct mf_hashtab {
///
/// The used list is a doubly linked list, most recently used block first.
/// The blocks in the used list have a block of memory allocated.
/// mf_used_count is the number of pages in the used list.
/// The hash lists are used to quickly find a block in the used list.
/// The free list is a single linked list, not sorted.
/// The blocks in the free list have no block of memory allocated and
@ -95,8 +94,6 @@ typedef struct memfile {
bhdr_T *mf_free_first; /// first block header in free list
bhdr_T *mf_used_first; /// mru block header in used list
bhdr_T *mf_used_last; /// lru block header in used list
unsigned mf_used_count; /// number of pages in used list
unsigned mf_used_count_max; /// maximum number of pages in memory
mf_hashtab_T mf_hash; /// hash lists
mf_hashtab_T mf_trans; /// trans lists
blocknr_T mf_blocknr_max; /// highest positive block number + 1

7
src/nvim/memline.c
View File

@ -1766,7 +1766,7 @@ errorret:
* Don't use the last used line when 'swapfile' is reset, need to load all
* blocks.
*/
if (buf->b_ml.ml_line_lnum != lnum || mf_dont_release) {
if (buf->b_ml.ml_line_lnum != lnum) {
ml_flush_line(buf);
/*
@ -2767,9 +2767,8 @@ static bhdr_T *ml_find_line(buf_T *buf, linenr_T lnum, int action)
if (buf->b_ml.ml_locked) {
if (ML_SIMPLE(action)
&& buf->b_ml.ml_locked_low <= lnum
&& buf->b_ml.ml_locked_high >= lnum
&& !mf_dont_release) {
/* remember to update pointer blocks and stack later */
&& buf->b_ml.ml_locked_high >= lnum) {
// remember to update pointer blocks and stack later
if (action == ML_INSERT) {
++(buf->b_ml.ml_locked_lineadd);
++(buf->b_ml.ml_locked_high);

21
src/nvim/option.c
View File

@ -619,27 +619,6 @@ void set_init_1(void)
}
}
/*
* 'maxmemtot' and 'maxmem' may have to be adjusted for available memory
*/
opt_idx = findoption("maxmemtot");
if (opt_idx >= 0) {
{
// Use half of amount of memory available to Vim.
// If too much to fit in uintptr_t, get uintptr_t max.
uint64_t available_kib = os_get_total_mem_kib();
uintptr_t n = (available_kib / 2 > UINTPTR_MAX // -V547
? UINTPTR_MAX
: (uintptr_t)(available_kib /2));
options[opt_idx].def_val[VI_DEFAULT] = (char_u *)n;
opt_idx = findoption("maxmem");
if (opt_idx >= 0) {
options[opt_idx].def_val[VI_DEFAULT] = (char_u *)n;
}
}
}
{
char_u *cdpath;
char_u *buf;

2
src/nvim/option_defs.h
View File

@ -498,9 +498,7 @@ EXTERN long p_mat; // 'matchtime'
EXTERN long p_mco; // 'maxcombine'
EXTERN long p_mfd; // 'maxfuncdepth'
EXTERN long p_mmd; // 'maxmapdepth'
EXTERN long p_mm; // 'maxmem'
EXTERN long p_mmp; // 'maxmempattern'
EXTERN long p_mmt; // 'maxmemtot'
EXTERN long p_mis; // 'menuitems'
EXTERN char_u *p_msm; // 'mkspellmem'
EXTERN long p_mls; // 'modelines'

14
src/nvim/options.lua
View File

@ -1511,13 +1511,6 @@ return {
varname='p_mmd',
defaults={if_true={vi=1000}}
},
{
full_name='maxmem', abbreviation='mm',
type='number', scope={'global'},
vi_def=true,
varname='p_mm',
defaults={if_true={vi=macros('DFLT_MAXMEM')}}
},
{
full_name='maxmempattern', abbreviation='mmp',
type='number', scope={'global'},
@ -1525,13 +1518,6 @@ return {
varname='p_mmp',
defaults={if_true={vi=1000}}
},
{
full_name='maxmemtot', abbreviation='mmt',
type='number', scope={'global'},
vi_def=true,
varname='p_mmt',
defaults={if_true={vi=macros('DFLT_MAXMEMTOT')}}
},
{
full_name='menuitems', abbreviation='mis',
type='number', scope={'global'},

9
src/nvim/os/os_defs.h
View File

@ -25,15 +25,6 @@
// Command-processing buffer. Use large buffers for all platforms.
#define CMDBUFFSIZE 1024
// Use up to 5 Mbyte for a buffer.
#ifndef DFLT_MAXMEM
# define DFLT_MAXMEM (5 * 1024)
#endif
// use up to 10 Mbyte for Vim.
#ifndef DFLT_MAXMEMTOT
# define DFLT_MAXMEMTOT (10 * 1024)
#endif
// Note: Some systems need both string.h and strings.h (Savage). However,
// some systems can't handle both, only use string.h in that case.
#include <string.h>

2
src/nvim/testdir/test_spell.vim
View File

@ -315,7 +315,7 @@ endfunc
" Check using z= in new buffer (crash fixed by patch 7.4a.028).
func Test_zeq_crash()
new
set maxmem=512 spell
set spell
call feedkeys('iasdz=:\"', 'tx')
bwipe!