3

I'm working on large files for wikisource, along with another guy. I generate a big text file (a few Mb) witch I format a bit using scripts, then I upload the text to a temporary page, where the other guy does some more formatting.

My friends computer and internet connection are not the best in the market, and he asked me not to upload more than 400Kb of data, so his computer can handle all the text.

the file is formatted like this:

==header 1==
.
.
.
==header 2==
.
.
.

and has a few hundreds of headers.

What I need to do:

I need to yank at most 400Kb (as close as I can to this number), but I also need to yank all the text between headers - I can't have the text between headers to split over a few files.

I can find the next header using \^={2}[^=]+={2}$, but I don't know how to carry on yanking until I have ~400Kb of data.

Any ideas?

6
  • That regexp doesn't seem to work? Jun 6, 2015 at 19:06
  • @Carpetsmoker what do you mean?
    – elyashiv
    Jun 6, 2015 at 19:07
  • What are you supposed to do if two such headers are more than 400 KB apart?
    – lcd047
    Jun 6, 2015 at 19:17
  • @lcd047 I will upload one. Any way, this is unlikely to happen.
    – elyashiv
    Jun 6, 2015 at 19:29
  • Do you want to round up or down? That is, if the 400 KB boundary falls in the middle of a section, do you want that section included or not?
    – lcd047
    Jun 6, 2015 at 19:32

3 Answers 3

2

Ok, this is somewhat of a quick hack, let's hope it works:

function! <SID>Chunk()
    " find first header
    let l_start = search('\m^==[^=]\+==$', 'cW')
    if !l_start
        " beep
        execute "normal \<Esc>"
        return
    endif

    " translate to byte position
    let b_start = line2byte(l_start)
    if b_start < 0
        " beep
        execute "normal \<Esc>"
        return
    endif

    " start marking
    execute 'normal V'

    " mode 400 KB down
    execute 'goto ' . (b_start + 400 * 1024)
    if line('.') == line('$')
        return
    endif

    " find previous header
    let l_end  = search('\m^==[^=]\+==$', 'bcsW', l_start)

    if l_end <= l_start || l_end <= 0
        " not found
        " go to beginning of line
        execute 'normal 0'

        " search next header
        let l_end  = search('\m^==[^=]\+==$', 'sW')
        if l_end <= 0
            " not found
            " go to end of file
            execute 'normal G$'
            return
        endif
    endif

    " move up
    exec 'normal k$'
    return
endfunction

nnoremap <silent> <leader>H :call <SID>Chunk()<CR>

It marks the next ~400 KB region of full headers. After yanking it you can go to the end of the marked region with '>.

2
  • Common practice is to use s: instead of <SID> where possible. The latter is required in mappings, but it's better to use the former in function definitions and commands.
    – tommcdo
    Jun 7, 2015 at 15:04
  • @tommcdo I believe there's a mapping at the end of the blockquote?
    – lcd047
    Jun 7, 2015 at 15:12
3

You can try something like this: place you cursor on the next line after a header, and hit:

y/\_.\{0,400000}==$/e(Enter)

or (more precise):

y/\_.\{0,400000}\n^==[^=]\+==$/e(Enter)

(translation: yank from cursor at most 400000 characters (not exactly bytes, though), including newlines, ending with two equals at the end of the line (the next header)). The e is needed to move the cursor (here: the yank range) to the end of the match. As the search is greedy, it can contain more than one section, but will end with a new heading and fit into the specified limit.

To repeat, move cursor on next line after the new header, and hit y/(Enter). To view the selection before copying it, use v/(Enter) and, if all is ok, y to yank it.

1

I'd do this in a shell script, which has the advantage that not only can you do this from Vim, with :!./dosplit %, but you can also do it from the command line or as part of a bigger script that also uploads the files.

Here's a dosplit script written in awk. It's usable out of the box on Unix-like systems, and requires an awk program to be installed on Windows (it's available ).

#!/usr/bin/awk -f
BEGIN {max = 409600}
/^==/ {    # It's a header line
    # Decide whether to start a new file
    if (length(chunk) + size > max) {++n; size=0}
    # Write the previous chunk
    size += length(chunk);
    printf "%s", chunk >sprintf("output-%04d.txt", n);
    chunk = "";
}
END {printf "%s", chunk >sprintf("output-%04d.txt", n)} # Write the last chunk
{chunk = chunk $0 "\n"} # Accumulate the current chunk
1
  • Upvoted purely for awk solution! :)
    – VanLaser
    Jun 7, 2015 at 22:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.