3

I'm working on large files for wikisource, along with another guy. I generate a big text file (a few Mb) witch I format a bit using scripts, then I upload the text to a temporary page, where the other guy does some more formatting.

My friends computer and internet connection are not the best in the market, and he asked me not to upload more than 400Kb of data, so his computer can handle all the text.

the file is formatted like this:

==header 1==
.
.
.
==header 2==
.
.
.

and has a few hundreds of headers.

What I need to do:

I need to yank at most 400Kb (as close as I can to this number), but I also need to yank all the text between headers - I can't have the text between headers to split over a few files.

I can find the next header using \^={2}[^=]+={2}$, but I don't know how to carry on yanking until I have ~400Kb of data.

Any ideas?

  • That regexp doesn't seem to work? – Martin Tournoij Jun 6 '15 at 19:06
  • @Carpetsmoker what do you mean? – elyashiv Jun 6 '15 at 19:07
  • What are you supposed to do if two such headers are more than 400 KB apart? – lcd047 Jun 6 '15 at 19:17
  • @lcd047 I will upload one. Any way, this is unlikely to happen. – elyashiv Jun 6 '15 at 19:29
  • Do you want to round up or down? That is, if the 400 KB boundary falls in the middle of a section, do you want that section included or not? – lcd047 Jun 6 '15 at 19:32
2

Ok, this is somewhat of a quick hack, let's hope it works:

function! <SID>Chunk()
    " find first header
    let l_start = search('\m^==[^=]\+==$', 'cW')
    if !l_start
        " beep
        execute "normal \<Esc>"
        return
    endif

    " translate to byte position
    let b_start = line2byte(l_start)
    if b_start < 0
        " beep
        execute "normal \<Esc>"
        return
    endif

    " start marking
    execute 'normal V'

    " mode 400 KB down
    execute 'goto ' . (b_start + 400 * 1024)
    if line('.') == line('$')
        return
    endif

    " find previous header
    let l_end  = search('\m^==[^=]\+==$', 'bcsW', l_start)

    if l_end <= l_start || l_end <= 0
        " not found
        " go to beginning of line
        execute 'normal 0'

        " search next header
        let l_end  = search('\m^==[^=]\+==$', 'sW')
        if l_end <= 0
            " not found
            " go to end of file
            execute 'normal G$'
            return
        endif
    endif

    " move up
    exec 'normal k$'
    return
endfunction

nnoremap <silent> <leader>H :call <SID>Chunk()<CR>

It marks the next ~400 KB region of full headers. After yanking it you can go to the end of the marked region with '>.

| improve this answer | |
  • Common practice is to use s: instead of <SID> where possible. The latter is required in mappings, but it's better to use the former in function definitions and commands. – tommcdo Jun 7 '15 at 15:04
  • @tommcdo I believe there's a mapping at the end of the blockquote? – lcd047 Jun 7 '15 at 15:12
3

You can try something like this: place you cursor on the next line after a header, and hit:

y/\_.\{0,400000}==$/e(Enter)

or (more precise):

y/\_.\{0,400000}\n^==[^=]\+==$/e(Enter)

(translation: yank from cursor at most 400000 characters (not exactly bytes, though), including newlines, ending with two equals at the end of the line (the next header)). The e is needed to move the cursor (here: the yank range) to the end of the match. As the search is greedy, it can contain more than one section, but will end with a new heading and fit into the specified limit.

To repeat, move cursor on next line after the new header, and hit y/(Enter). To view the selection before copying it, use v/(Enter) and, if all is ok, y to yank it.

| improve this answer | |
1

I'd do this in a shell script, which has the advantage that not only can you do this from Vim, with :!./dosplit %, but you can also do it from the command line or as part of a bigger script that also uploads the files.

Here's a dosplit script written in awk. It's usable out of the box on Unix-like systems, and requires an awk program to be installed on Windows (it's available ).

#!/usr/bin/awk -f
BEGIN {max = 409600}
/^==/ {    # It's a header line
    # Decide whether to start a new file
    if (length(chunk) + size > max) {++n; size=0}
    # Write the previous chunk
    size += length(chunk);
    printf "%s", chunk >sprintf("output-%04d.txt", n);
    chunk = "";
}
END {printf "%s", chunk >sprintf("output-%04d.txt", n)} # Write the last chunk
{chunk = chunk $0 "\n"} # Accumulate the current chunk
| improve this answer | |
  • Upvoted purely for awk solution! :) – VanLaser Jun 7 '15 at 22:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.