Highlighting another syntax in a shell heredoc

Question

I edited my original question and title to broaden its scope to any syntax - with SQL used as just an example. You can adapt the examples and the answer to other languages.

By adding suitable highlighting rules, one can make vim highlight multiple languages within the same file. This comes particularly handy for shell scripts, in which each new language block is embedded inside a heredoc. For instance, the above picture shows SQL within four shell heredocs. SQL is displayed mono-color inside EOF delimited heredocs, in which default highlighting rules apply, By contrast, SQL is displayed multi-color inside SQLDOC delimited heredocs, in which new highlighting rules were added specific to SQL syntax elements. We can see an issue, however: the last SQLDOC heredoc is incorrectly highlighted mono-color. How can I fix this and make that block display multi-color SQL highlights?

To reproduce this issue:

1. Close vim and copy the highlighting file to ~/.vim/after/syntax/sh/heredoc-embed.vim (file contents below).

2. Start vim ./vim-heredoc-syntax-issue.sh (file contents below).

We are often instructed to start vim with -u NONE to make sure that plugins don't get in the way of reproducing an issue. In this case I can't tell you so. If I use -u NONE syntax highlighting will not load at all. And I'm unable to find a sequence of vim commands that enables syntax highlighting from -u NONE to my issue. I tried! but failed. So please start your vim as I explained above and hopefully you'll reproduce this issue. To make sure that it occurs in a reasonably pristine vim configuration, I took the above screenshot in a virtual machine, in which I booted a fresh copy of my Linux distribution, installed vim from the package manager, copied the "heredoc-embed.vim" file, and ran vim.

heredoc-embed.vim

" This VIML code sets SQL syntax highlighting for heredoc content blocks
" marked by word <LANGUAGE>DOWN, i.e. SQLDOC.

" source https://subvisual.co/blog/posts/87-smarter-heredoc-syntax-in-vim/ {{{1

let s:bcs = b:current_syntax
" this unlet instruction is needed before we load each new syntax
unlet b:current_syntax
syntax include @SQL syntax/sql.vim

let b:current_syntax = s:bcs

syntax region hereDocDashSQL matchgroup=Statement start=+<<\s*[-'\\]\?\z(SQLDOC\)+  end=+^\s*\z1+ contains=@SQL

if exists("b:current_syntax")
  unlet b:current_syntax
endif

vim-heredoc-syntax-issue.sh

# colorscheme evening

: << EOF
    SELECT * FROM table;
EOF
: << SQLDOC
    SELECT * FROM table;
SQLDOC

foo() {
: << EOF
    SELECT * FROM table;
EOF
: << SQLDOC
    SELECT * FROM table;
SQLDOC
}

Similar questions I found on SE:

1. How do I enable language-specific syntax inside a heredoc?

I'd rather fix mine because the regular expressions are simpler.

2. How can I fix the default bash syntax highlighter?

Maybe Dr Chip can tell me?

3. Highlight heredoc as perl code

My formula but for Perl heredoc.

Answer 1

I would recommend the following approach:

Create the file $HOME/.vim/after/syntax/sh.vim ($HOME/vimfiles/after/syntax/sh.vim on Windows) with the following content:

" safe b:current_syntax to restore it afterwards
" Value could be 'sh', 'posix', 'ksh' or 'bash'
let s:cs_safe = b:current_syntax

" unlet b:current_syntax, so sql.vim will load
unlet b:current_syntax
syntax include @SQL syntax/sql.vim

" restore saved syntax
let b:current_syntax = s:cs_safe

syn region shHereDoc matchgroup=shHereDocSql start="<<\s*\\\=\z(SQLDOC\)" matchgroup=shHereDocSql end="^\z1\s*$"   contains=@SQL
hi def link shHereDocSql        shRedir

This file is sourced after the default sh syntax is sourced. At this point in time the buffer local b:current_syntax is already set, so it has to be cleared before sourcing the sql syntax and restored afterward.

Then another shHereDoc region is defined which contains sql code.

To read about the "after" directory see :help after-directory and :help mysyntaxfile-add.

Answer 2

Answering my own question after some time.

The clue to a solution came from noticing that the same issue happens when the SQLDOC heredoc is contained within other shell constructs, such as if or for. On the other hand, EOF heredocs are always highlighted as such even when they appear inside if, for and functions. Why the difference then? Looking at the shell syntax rules one notices that heredoc regions are defined as contained elements of other regions or clusters, specifically shCaseList, shCommandSubList and shFunctionList. So if I add my hereDocDashSQL region as a contained element of those three clusters, SQL should display multi-colors in all cases. Add a continuation line after the syntax region line in file heredoc-embed.sh:

syntax region hereDocDashSQL matchgroup=Statement start=+<<\s*[-'\\]\?\z(SQLDOC\)+  end=+^\s*\z1+ contains=@SQL
      \ containedin=@shCaseList,shCommandSubList,shFunctionList
" Introduce ^^^^ the new heredoc to $VIMRUNTIME/syntax/sh.vim

The arguments to containedin are the syn clusters that contain shHereDoc in file $VIMRUNTIME/syntax/sh.vim. The continuation line could include argument contained to allow matching the heredoc only when it's contained in the clusters. In this case contained is omitted so the heredoc can be recognized also in the main body of the shell script. Cf. :help syn-containedin, :help syn-contained.

Side-by-side before/after comparison: