2

why isn't the fourth SELECT highlighted like the second one?

I edited my original question and title to broaden its scope to any syntax - with SQL used as just an example. You can adapt the examples and the answer to other languages.

By adding suitable highlighting rules, one can make vim highlight multiple languages within the same file. This comes particularly handy for shell scripts, in which each new language block is embedded inside a heredoc. For instance, the above picture shows SQL within four shell heredocs. SQL is displayed mono-color inside EOF delimited heredocs, in which default highlighting rules apply, By contrast, SQL is displayed multi-color inside SQLDOC delimited heredocs, in which new highlighting rules were added specific to SQL syntax elements. We can see an issue, however: the last SQLDOC heredoc is incorrectly highlighted mono-color. How can I fix this and make that block display multi-color SQL highlights?

To reproduce this issue:

  1. Close vim and copy the highlighting file to ~/.vim/after/syntax/sh/heredoc-embed.vim (file contents below).

  2. Start vim ./vim-heredoc-syntax-issue.sh (file contents below).

We are often instructed to start vim with -u NONE to make sure that plugins don't get in the way of reproducing an issue. In this case I can't tell you so. If I use -u NONE syntax highlighting will not load at all. And I'm unable to find a sequence of vim commands that enables syntax highlighting from -u NONE to my issue. I tried! but failed. So please start your vim as I explained above and hopefully you'll reproduce this issue. To make sure that it occurs in a reasonably pristine vim configuration, I took the above screenshot in a virtual machine, in which I booted a fresh copy of my Linux distribution, installed vim from the package manager, copied the "heredoc-embed.vim" file, and ran vim.

heredoc-embed.vim

" This VIML code sets SQL syntax highlighting for heredoc content blocks
" marked by word <LANGUAGE>DOWN, i.e. SQLDOC.

" source https://subvisual.co/blog/posts/87-smarter-heredoc-syntax-in-vim/ {{{1

let s:bcs = b:current_syntax
" this unlet instruction is needed before we load each new syntax
unlet b:current_syntax
syntax include @SQL syntax/sql.vim

let b:current_syntax = s:bcs

syntax region hereDocDashSQL matchgroup=Statement start=+<<\s*[-'\\]\?\z(SQLDOC\)+  end=+^\s*\z1+ contains=@SQL

if exists("b:current_syntax")
  unlet b:current_syntax
endif

vim-heredoc-syntax-issue.sh

# colorscheme evening

: << EOF
    SELECT * FROM table;
EOF
: << SQLDOC
    SELECT * FROM table;
SQLDOC

foo() {
: << EOF
    SELECT * FROM table;
EOF
: << SQLDOC
    SELECT * FROM table;
SQLDOC
}

Similar questions I found on SE:

  1. How do I enable language-specific syntax inside a heredoc?

I'd rather fix mine because the regular expressions are simpler.

  1. How can I fix the default bash syntax highlighter?

Maybe Dr Chip can tell me?

  1. Highlight heredoc as perl code

My formula but for Perl heredoc.

1

I would recommend the following approach:

Create the file $HOME/.vim/after/syntax/sh.vim ($HOME/vimfiles/after/syntax/sh.vim on Windows) with the following content:

" safe b:current_syntax to restore it afterwards
" Value could be 'sh', 'posix', 'ksh' or 'bash'
let s:cs_safe = b:current_syntax

" unlet b:current_syntax, so sql.vim will load
unlet b:current_syntax
syntax include @SQL syntax/sql.vim

" restore saved syntax
let b:current_syntax = s:cs_safe

syn region shHereDoc matchgroup=shHereDocSql start="<<\s*\\\=\z(SQLDOC\)" matchgroup=shHereDocSql end="^\z1\s*$"   contains=@SQL
hi def link shHereDocSql        shRedir

This file is sourced after the default sh syntax is sourced. At this point in time the buffer local b:current_syntax is already set, so it has to be cleared before sourcing the sql syntax and restored afterward.

Then another shHereDoc region is defined which contains sql code.

To read about the "after" directory see :help after-directory and :help mysyntaxfile-add.

  • Thank you. I like it better than mine, but the highlighting isn't correct, yet. Test by opening file vim-heredoc-syntax-issue.sh and changing the file type from sh to sql. Look at line SELECT * FROM table;. Here I see that highlighting changes as the file type changes. It shouldn't. The line should be highlighted as SQL regardless of the file type. With plugin hilinks it's easy to see that SELECT is recognized as shString instead of sqlStatement... – stepse Apr 3 at 20:36
  • For me: SELECT = sqlStatement, FROM & TABLE = sqlKeyword. But * & ; = shHereDoc, which is expected as in a normal SQL file they have no highlight at all. If I switch from filetype sh to sql, only the color for * and ; change. Did you remove your custom syntax/sh.vim before testing? (BTW: Vim 8.1.1099, but I don't think that makes a difference.) – Ralf Apr 4 at 4:38
  • Sorry, I mistyped. Same for me. SELECT = sqlStatement but * and ; = shHereDoc->ShString. Your explanation of why * and ; are highlighted that way makes sense but I didn't expect to see any shell syntax creep in and colorize parts of SQL. It doesn't happen with the other method. I also noticed that with this method SELECT 'a'||'b' matches 'a' || 'b' as ShStrings, while with the other method the matches are sqlString, sqlFold, sqlString (plus quote tokens). I think that both methods have their merits, and they just aren't exactly equivalent. – stepse Apr 4 at 19:51
  • I need to test my method more because I just discovered some new issues. I'll come back to this. – stepse Apr 4 at 20:16
  • After more testing I prefer your method to mine. So I have marked your answer accepted and upvoted it. Thank you. The only thing I'd change in your answer is the start end pattern pair. I prefer mine because it can match seldom-used but valid heredoc syntax such as <<- \EOF. I'm using start=+<<-\=\s*['"\\]\=\z(SQLDOC\)+ end="^\s*\z1" – stepse Apr 20 at 19:34
1

Answering my own question after some time.

The clue to a solution came from noticing that the same issue happens when the SQLDOC heredoc is contained within other shell constructs, such as if or for. On the other hand, EOF heredocs are always highlighted as such even when they appear inside if, for and functions. Why the difference then? Looking at the shell syntax rules one notices that heredoc regions are defined as contained elements of other regions or clusters, specifically shCaseList, shCommandSubList and shFunctionList. So if I add my hereDocDashSQL region as a contained element of those three clusters, SQL should display multi-colors in all cases. Add a continuation line after the syntax region line in file heredoc-embed.sh:

syntax region hereDocDashSQL matchgroup=Statement start=+<<\s*[-'\\]\?\z(SQLDOC\)+  end=+^\s*\z1+ contains=@SQL
      \ containedin=@shCaseList,shCommandSubList,shFunctionList
" Introduce ^^^^ the new heredoc to $VIMRUNTIME/syntax/sh.vim

The arguments to containedin are the syn clusters that contain shHereDoc in file $VIMRUNTIME/syntax/sh.vim. The continuation line could include argument contained to allow matching the heredoc only when it's contained in the clusters. In this case contained is omitted so the heredoc can be recognized also in the main body of the shell script. Cf. :help syn-containedin, :help syn-contained.

Side-by-side before/after comparison:

SOLVED: now the fourth SELECT is highlighted like the second one

  • Does anyone know if there's a way to define a cluster incrementally? Copying the whole contains list from sh.vim to add ,hereDocDashSQL isn't good for maintainability. Ideally I would like to write something like (vim 8.1 doesn't support this): syn cluster shCaseList contains+=hereDocDashSQL. – stepse Apr 1 at 8:29
  • There is also containedin. Used like syntax region hereDocDashSQL ... containedin=shCaseList contained. See :help :syn-containedin. – Ralf Apr 1 at 9:00
  • Thank you @Ralf, I have updated my answer with your help. – stepse Apr 2 at 12:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.