I would like to join all lines in a file which are continuous parts of some natural language but not back-to-back lines which are two separate entities. For example:
Join:
There was once a man
called Bob and he lived
in France.
Do not join:
Chapter 1
The arrival
Part 1
I am starting to think this will only be convenient with a modern machine learning technique as otherwise there will not be enough distinguishing features in the text to make a rule-based program. The script basically has to know on a conceptual level what is a self-sufficient unit of text (i.e. a header) vs. what seems like broken natural language, including maybe that it connects to the previous or following text.
If I can train some machine learning function to detect whether or not a line should be joined with the next or previous, either a Boolean true or false or returning “join next”, “join previous” or “do not join”, how could I implement it in Vim? A function which goes through every line, calls the classifier, and calls the Vim join method if it passes? Or is there a better way?
