How can I select all non-lowercase multiline blocks which follow a keyword?

Question

I'd like to copy these to a file, but in a speculative future case, I might want to delete them. I might need to find a regex to substitute intervening data with a single blank line.

The highlighting shows it's matching a multiline block but global yank only gets the first line in the block.

These blocks can contain one blank line between lines with all non-lowercase. In some cases, the keyword can follow not(non-lowercase) on a line. Otherwise, I don't want any lines having mixed case or lowercase text. I would rather exclude that leading not(non-lowercase) if possible.

Gvim Version 7.3

In this example case, the keyword is MISHNAH. The highlight seems as if this regex would work:

/MISHNAH\_[^a-z]\+/

but

qAq
:g/MISHNAH\_[^a-z]\+/y A

or

:g/MISHNAH\_[^a-z]\+/d

only get the lines having the keyword.

EXAMPLE DATA:

CHAPTER I

   MISHNAH. FROM WHAT TIME MAY ONE RECITE THE SHEMA' IN THE EVENING?
FROM THE TIME THAT THE PRIESTS ENTER [THEIR HOUSES] IN ORDER TO EAT THEIR
TERUMAH [1] UNTIL THE END OF THE FIRST WATCH. [2] THESE ARE THE WORDS OF R.
ELIEZER. THE SAGES SAY: UNTIL MIDNIGHT. R. GAMALIEL SAYS: UNTIL THE DAWN ...

   EXAMPLE OF MORE BELONGING UPPERCASE TEXT AFTER ONE BLANK LINE

   GEMARA. On what does the Tanna base himself that he commences: FROM WHAT TIME? [6]
Furthermore, why does he deal first with the evening [Shema']? Let him begin with the morning 
...  A whole lot of lines  ...
     HERE IS A LINE I DON'T WANT

but he says two blessings before and one after.
   R. Mani said: He who recites the Shema' in its proper time is greater than he who studies the
Torah. [51] For since it says, HE WHO SAYS LATER LOSES NOTHING, BEING LIKE A MAN
WHO READS IN THE TORAH, we may conclude that one who recites the Shema' at its proper
time is superior. MISHNAH. BETH SHAMMAI SAY: IN THE EVENING EVERY MAN
SHOULD RECLINE AND RECITE [THE SHEMA'], AND IN THE MORNING HE SHOULD ...
ACTED AGAINST THE OPINION OF BETH HILLEL.

(1) This seems to be simply a periphrasis for `if a man is told in a dream'. Two explanations are
then possible of what follows. (i) If he dreams and the dream so far comes true that a sword is
placed on his neck, still he should pray. (ii)
...  A whole lot of lines ...
Even if he only dreams this, he should still pray etc. (R. Bezalel of Regensburg.)

I want to get this:

   MISHNAH. FROM WHAT TIME MAY ONE RECITE THE SHEMA' IN THE EVENING?
FROM THE TIME THAT THE PRIESTS ENTER [THEIR HOUSES] IN ORDER TO EAT THEIR
TERUMAH [1] UNTIL THE END OF THE FIRST WATCH. [2] THESE ARE THE WORDS OF R.
ELIEZER. THE SAGES SAY: UNTIL MIDNIGHT. R. GAMALIEL SAYS: UNTIL THE DAWN ...

   EXAMPLE OF MORE BELONGING UPPERCASE TEXT AFTER ONE BLANK LINE

time is superior. MISHNAH. BETH SHAMMAI SAY: IN THE EVENING EVERY MAN
SHOULD RECLINE AND RECITE [THE SHEMA'], AND IN THE MORNING HE SHOULD ...
ACTED AGAINST THE OPINION OF BETH HILLEL.

I'd like to exclude the leading time is superior. And also exclude lines that begin with non-lowercase and then continue with not(non-lowercase), such as:

   GEMARA. On what does the Tanna base himself that he commences: FROM WHAT TIME? [6]

I have found that perl's '~= m' (match) has an 's' switch to view the data as a string with embedded newlines. My problem seems to be that I don't know how I can get vim to do that. I copy the EXAMPLE DATA to a file called 'foo.txt' and run a perl script [on win cmd line] which can do:

perl -e "$n=chr(10);$d=join('', <>);$r='(MISHNAH(?:[^a-z]|\n)+)';while ($d =~ m/$r/sg) {print $1 . $n}" foo.txt

Answer 1

Problem 1: How :global interacts with multiline matches

The main problem that you're seeing is that you're slightly misunderstanding how :global works. It doesn't run the commands on the matched text. Instead, it runs the commands on the lines where a match is found.

So while your match itself extends over multiple lines, the location of it is the first line, and your :yank command is only run there, as you are seeing.

In the general case, you can fix this by passing a range encompassing the desired lines to the command you want to run instead of including those lines in the :global command's regex.

So instead of:

:g/MISHNAH\_[^a-z]\+/y A

You'd use something like:

:g/MISHNAH/.,/[a-z]/-1y A

In the above, you're yanking all the lines in the range ./[a-z]/-1, which starts on the line where MISHNAH was found, and ends on the line before the first line after that that contains a lowercase letter.

Problem 2: :yank operates on entire lines

However, you also want to exclude parts of the lines from the operations. For the yank case, you could remove the unwanted lowercase characters by deleting them after you paste, but that won't work for the delete case, and no amount of fiddling with the regular expressions can change that.

Solution: use macros

If this is something you're going to be doing frequently, you'd probably be best off writing some Vimscript. However, if it's a one-off, you can quickly solve the problem by using a macro instead of the ex :yank or :delete commands.

Try these:

Yank macro

1. Type qy to start recording a macro into register "y
2. Run the search /MISHNAH to jump to the first MISHNAH
3. Enter visual mode by typing v
4. Run a search to jump to the end of the text you want to operate on: /\l/-1. \l is just a shorter way of specifying all lower-case characters, which also works even when 'ignorecase' is set. -1 tells Vim you want your range to stop on the line above the match
5. Yank the selection: "Ay
6. End the recording by typing: q

You can then run it on your entire file with the commands:

qaq
:g/MISHNAH/norm!@y

Delete macro

This works slightly differently, because you want to keep the text that's left from ending up on the same line:

1. Type qd to start recording a macro into register "d
2. Run the search: /MISHNAH
3. Enter visual mode: v
4. Run the search: /\l
5. Move up to the end of the text you want to delete: --$h
6. Delete the text by typing d
7. End the recording: q

And then, again, to run it globally:

:g/MISHNAH/norm!@d

Answer 2

The problem you have is that g command applies on all the lines even if it has been part of a previous match.

I would propose you to:

1. Put the file upside down

:g/.*/m0

2. Do your filtering in the other direction:

:g!/\_[^a-z]\+MISHNA/d

3. Put the file upside down

:g/.*/m0

If you want to remove the leading time is superior.

You could do:

%s/\C\v^[a-z]+[;,.]?(\s+[a-z]+[;,.]?)*//

Answer 3

Since you have a perl filter that does the job, use it:

:%!perl -e "$n=chr(10);$d=join('', <>);$r='(MISHNAH(?:[^a-z]|\n)+)';while ($d =~ m/$r/sg) {print $1 . $n}"

equivalent to

gg!Gperl -e "$n=chr(10);$d=join('', <>);$r='(MISHNAH(?:[^a-z]|\n)+)';while ($d =~ m/$r/sg) {print $1 . $n}"