3

I am trying to change a text file with data in 'document' form into unnormalized csv. The data are a list of hymn authors ('document header') and for each author a list of one or more hymns they have written ('document line'). The specific operation I am struggling with is taking each 'document header' and prepending it to the one or more 'document lines' that follow it.

Example data

For example, I want to turn this

Beskow Natanael
    140     Ack saliga dag
    399     Tränger i dolda djupen ner
    478     Ditt verk är stort
    479     Kärlek av höjden
Bexell Göran
    103     Herren lever våga tro det
Béze Théodore de
    283     Lovsjung nu alla länder Gud
    360     Såsom hjorten ivrigt längtar

into

Beskow Natanael,140,Ack saliga dag
Beskow Natanael,399,Tränger i dolda djupen ner
Beskow Natanael,478,Ditt verk är stort
Beskow Natanael,479,Kärlek av höjden
Bexell Göran,103,Herren lever våga tro det
Béze Théodore de,283,Lovsjung nu alla länder Gud
Béze Théodore de,360,Såsom hjorten ivrigt längtar

The specific change that I am struggling with is taking the 'document header' and prepending it to each 'document line' so I am leaving the other changes (removing the header, comma between hymn number and hymn title, etc.) to one side for the moment.

Question

I could write a function to do this change, but I find that I am collecting quite a number of ad hoc functions for different editing operations, so I would prefer to learn how to do it with a global command, or, if it is not a suitable operation to perform with global, to understand why. How can I do this operation with global; and, if I can't, why?

What I have tried

I have tried using the :global command to execute different combinations of normal commands.

1) Yank, move, put

g/^\S/normal! y$jPa,

I take this to mean

  1. for each line beginning with non-whitespace
  2. yank to end-of-line into unnamed register
  3. move down one line and put before from unnamed register
  4. append a comma

This works, but only for each first 'document line'. The result is

Beskow Natanael
Beskow Natanael,    140     Ack saliga dag
    399     Tränger i dolda djupen ner
    478     Ditt verk är stort
    479     Kärlek av höjden
Bexell Göran
Bexell Göran,    103     Herren lever våga tro det
Béze Théodore de
Béze Théodore de,    283     Lovsjung nu alla länder Gud
    360     Såsom hjorten ivrigt längtar

To repeat the put operation on each 'document line' I think I would need to know how many there are and put in a loop, and I don't see an easy way to do that without writing a custom function.

2) Yank, visually select, substitute

To avoid having to count I thought I could instead visually select the 'document lines' and insert the 'document header' with a substitute over the visual selection.

g/^\S/normal! y$v/^\S^M:s/^\s/\=@" . "," . submatch(0)

I intend this command to mean the following

  1. for each line beginning with non-whitespace
  2. yank to the end of line
  3. visually select until the next line beginning with non-whitespace
  4. substitute over visual selection
    • match beginning-of-line + whitespace
    • substitute contents of unnamed register + a comma + the entire match (i.e., the whitespace)

This command does not work at all.

  • It does not change the buffer at all.
  • It leaves me in visual mode, with the last character on the first line selected.
  • Hitting /<Up> to look at the last search pattern shows /^\S^M:s/^\s/\=@" . "," . submatch(0). This tells me that the ^M does not execute the search as I had hoped, but rather everything following / is interpreted as part of the pattern.
  • Checking the unnamed register after running the command, it contains the last 'document header' in the buffer. This tells me that it goes through the buffer and performs the yank operation.

If, however, I type out the normal mode commands manually they work as expected.

3
  • "If all you have is a hammer everything looks like a nail." - For more complex text manipulation tasks one should better use standard text processors (awk, perl, ...) than an editor. - E.g. with awk (assuming the data fields are TAB separated, otherwise the \t has to be redefined): BEGIN {FS="\t";OFS=","} /^[^\t]/{s=$0;next} {print s,$2,$3} - Since this is "OT" I just provide this comment. I think it's worth to get a feeling when it's appropriate to switch tools.
    – Janis
    Aug 3, 2015 at 12:48
  • @Janis Absolutely! But I don't think Vim is a hammer in this instance :-) I would like to know awk better and I am grateful for the example, but a) it would be a dubious investment to learn an external tool for something that is well within Vim's purview, and b) your command is actually more complex than the simplest working solution within Vim, especially if I add the 'padding' of invoking awk from within Vim as a filter so I can continue editing the text in Vim afterwards.
    – jjaderberg
    Aug 3, 2015 at 20:57
  • Don't get me wrong; vim is great! - Where I have my reservations, though, is when it comes to implementing non-standard vim-script solutions, where even the very powerful vi/vim-base can't solve the issue. - From your point (b) I get the impression that you want to do everything from built-ins within vim - I think this is (in case of complex tasks) not appropriate (YMMV). The solution (IMO) is to use the existing (or new) external tool(s) and apply vi/vim's !G command. - WRT learning new things; it's your choice whether you learn vim-specifics or a (POSIX) standard tool.
    – Janis
    Aug 4, 2015 at 13:30

3 Answers 3

3

This just screams "macro and normal" for me. First, a comma after each name would be helpful as a name terminator, so execute :g/^\S/normal A, first.

Start somewhere on line 2 and type

  • qq to start recording into buffer q
  • 0ma to go to the start of the line and save the position at mark a
  • ?^\S<Enter> to search backward for the next line that does not start with a whitespace (you could also just go 1 line backwards, but add a 0 in that case to go to the start of the line)
  • yf, to yank to the first comma, including it, which should be the name by what we did in the first step
  • 'a0 to return to the line of the mark a and go to the beginning (use `a instead, we saved at the beginning of the line, but I can't type that here with the grey background)
  • p to paste
  • 0x to delete the whitespace at the beginning
  • q to stop recording

That seems to do what you want, adjust accordingly (you could skip the first step in various circumstances, e.g. if all the lines that start with a whitespace have a numeral as the first non-whitespace character... but since you wanted to add commata anyways, I went that way). Now just use :g/^\s/normal! @q to execute the macro we saved in buffer q (@q) on each line that starts with a whitespace (^\s).

This works for your example. It has the additional advantage of being adjustable for the other changes you probably want to do, just by changing the macro when recording it.

See

for more.

I noticed you might want to get right of the author-only lines, and that might be hard later on. Just insert a :g/^\s/normal J after the first step, which produces 1 correct line and gets rid of the author-only line for each author, and then proceed with the macro as given.

1
  • Yes! This is exactly what I was doing, and yet it never occurred to me that I could execute a macro from global//normal. I kept building my commands in a buffer and yank-putting with <C-r> into the :global command. This would make it easier to build the actual command to be executed.
    – jjaderberg
    Aug 3, 2015 at 20:38
2

This is one way to do it (I'm sure it's not perfect, but it gets the job done):

:g/./if getline('.')[0] =~ '\w' | delete | else | s/\s\+\(\w\+\)\s\+\(\w\+\)/\=@" . ',' . submatch(1) . ',' . submatch(2)/ | -join! |endif
:g/^$/delete

BTW, this would look much nicer if you would allow yourself to use functions.

The idea is, you use global to match both headers and non-headers (=> non-empty lines), but act differently on them: if it's a header, delete it (=> it's yanked to the default register), if not, replace first spaces with the current header (by yanking it back from the default register) followed by a comma, and the next spaces with a comma.

If, instead, you only search for headers, it's harder to then act on non-header lines.

0

The second command is missing a final ^M.

In order to understand why the second command works if the normal mode commands are executed manually, but not from within the global command, I recorded the successful sequence of commands as a macro. In the register I find

y$v/^\S^M:s/^\s/\=@".",".submatch(0)^M

Lo and behold, if I use this sequence with global, which gives an identical command to the second one in the question except for the additional ^M at the end

g/^\S/normal! y$v/^\S^M:s/^\s/\=@".",".submatch(0)^M

it works.

It does not, however, finish the last 'document' in the buffer, since there is no start-of-line+non-whitespace to search forward to.

I'm a little disappointed about that being the solution and I'm not sure whether I should edit the question, close it, or leave it like it is. I'll leave it for a while and see if there are more answers, explanations or comments–I'm sure that, even if this global command now works (except for the last line), it could still be improved upon.

5
  • The thing with global is that one should use normal only at the end of the pattern. In my understanding, you can't "come back" to cmd mode once you went to using normal. I can't find the reference, and I hope I'm not mistaken (maybe somebody knows better and can intervene with details/corrections).
    – VanLaser
    Aug 3, 2015 at 11:16
  • Well, the command may be one long normal command, but that should be ok, shouldn't it? F.i. I begin the substitute from within the normal command with :/s rather than using bar and a separate ex command, ` | s/`.
    – jjaderberg
    Aug 3, 2015 at 11:32
  • I know you can't use | to separate commands after normal since it is treated as an argument and not a command separator. I tried using execute "normal ..." as well, but then I had to do lots of escaping in the search and substitute patterns (e.g., \\s).
    – jjaderberg
    Aug 3, 2015 at 11:33
  • 1
    I usually use execute normal to get back to command mode with global. One way to avoid lots of \ is to use single quotes ' instead of double " to surround normal, as in execute 'normal ...'. See :help literal-string vs :help expr-string or, more generally, the bottom of section 41.2 on variables :help 41.2 in the section "STRING VARIABLES AND CONSTANTS" (actually, its quicker to do :help 41.3 and scroll up a couple pages)
    – John O'M.
    Aug 4, 2015 at 1:59
  • Oh, I have misunderstood the paragraph at :h exe|/nice to say that I must give control characters in the \<Esc> form with the execute command. Since that form won't work in '' strings, I thought I had to put control characters in expression strings, so, either do extra escapes ("/^\\S\<CR>") or maybe concatenate literal and expression strings ('/^\S' . "\<CR>"). I see now that it says 'nice', not 'necessary', and '/^\S^M' seems to work just fine. Thanks!
    – jjaderberg
    Aug 4, 2015 at 7:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.