How do I remove lines where its content already occured in the previous line?

Question

For example I've the following text:

This is my first line, but longer.
This is my first line
This is my second line, but longer.
This is my second line
This is my third line, but longer.
This is my third line
This

and I expect to remove the shorter lines which share the same (from beginning of the line) content with either previous or next line.

So I'm expect from above example to have only 3 main unique sentences (longer one), as the shorter once are already contained in longer lines. This method has several real usages such as finding the leaf items (such as directories).

Assuming the content is sorted (could be in reverse if that helps), I think we can compare next line with the previous one.

I believe I need to match each line and create some back reference to the previous one, similar as for removing duplicate lines:

:g/^\(.*\)\n\1$/d

but with additional .*

:g/^\(.*\).*\n\1$/d

but it removes the shorter line instead of leaving the longer one.

Any idea how this can be achieved?

Answer 1

Probably not exactly the solution you expect, but I'd rather use something like this:

:g/^/ if search('^' . getline('.'), 'wn') != line('.') | delete | endif

This will also remove duplicates (since you do mention unique sentences), e.g.

This
This is my first line, but longer.
This is my first line, but longer.
This is my first line

becomes:

This is my first line, but longer.

For every line, its content is searched forward in the whole file, starting from current position and wrapping until it finds something. If it doesn't, the search will "land" (find) the same original line used as pattern. In the first case, the line is deleted (its content exists elsewhere).

Answer 2

You are close.

If you can live with an error "E16: invalid range" make the delete command delete the next line:

:g/^\(.*\).*\n\1$/+d

Using :g with patterns matching several lines is tricky and this works for your special case, but it does not always work as one expect it.

Another alternative is to simply use the :s command:

:%s/^\(\(.*\).*\)\n\2$/\1

which replaces the match (the two consecutive lines) with the longer line.

---

Funnily it looks like this pattern matches here only with the old 'regular expression engine' so you might have to set your 're' setting accordingly. But I might be missing some patch here.