:browse oldfiles and
:oldfiles read from the viminfo file, so changing the maximum number of previously edited files to be remembered in the
'viminfo' option should work. You can do this by changing the number after the
' character in the viminfo option.
For example, doing
:set viminfo='50 would mean that only 50 old files are remembered.
However, this will overwrite everything currently set in the the
'viminfo' option, so make sure to set the other desired options as well. For example, doing
:set viminfo='50,<50,s10,h would also mean that the maximum number of lines saved for each register would be 50, the maximum size of saved items is 10 KB and the effect of
'hlsearch' is disabled when loading the viminfo file.
For more on this topic, see
NOTE: You may need to wipe your viminfo file before the change will take effect, since all of your old files past your max number will be remembered still.