How do I use syntax-nextgroup in this custom syntax file?

Question

^{Writing vim syntax files never ceases to make me feel like an idiot.}

I have a new custom language call jsond, and I wanted to get syntax highlighting working in vim. Using racket syntax works pretty well (at least with my fork).

Here are some example files:

#lang jsond

#:name albums
["Cycles", "Flatspin", "Long Way to Climb"]

#:name favorites
{
    "Jonathan Gordon": 0,
    "Brad Rubinstein": 2,
    "Paul Willmott": 1,
    "Jason Hall": 0,
    "Adam Rich": 1
}

#lang jsond

#:name abc "abc"

;; comment

#:name num 123

#:name array [1, 2, {"xyz": "abc"}, true, false, null]

As you can see, the language is effectively #:name <id> <json>, with a few other things.

Here's my syntax file:

" Vim syntax file
" Language:     #lang jsond

" Initializing:
if version < 600
  syntax clear
elseif exists("b:current_syntax")
  finish
endif

syntax match fallbackError /./

syntax include @json syntax/json.vim

if version < 600
  set iskeyword=33,35-39,42-58,60-90,94,95,97-122,126,_
else
  " syntax iskeyword 33,35-39,42-58,60-90,94,95,97-122,126,_
  " converted from decimal to char
  " :s/\d\+/\=submatch(0)->str2nr()->nr2char()/g
  " but corrected to remove duplicate _, move ^ to end
  syntax iskeyword @,!,#-',*-:,<-Z,a-z,~,_,^
  " expanded
  " syntax iskeyword !,#,$,%,&,',*,+,,,-,.,/,0-9,:,<,=,>,?,@,A-Z,_,a-z,~,^
endif

syntax match jsondSyntax "#lang " nextgroup=jsondLang
syntax keyword jsondLang jsond

syntax keyword jsondName #:name nextgroup=jsondVarName skipwhite skipempty
syntax match jsondVarName ,\<\k+\>, nextgroup=@json skipwhite skipempty

" Comments
syntax match jsondComment /;.*$/ contains=jsondTodo,jsondNote,@Spell
syntax region jsondMultilineComment start=/#|/ end=/|#/ contains=jsondMultilineComment,jsondTodo,jsondNote,@Spell

syntax keyword jsondTodo FIXME TODO XXX contained
syntax match jsondNote /\CNOTE\ze:\?/ contained

" Define the default highlighting.
" For version 5.7 and earlier: only when not done already
" For version 5.8 and later: only when an item doesn't have highlighting yet
if version >= 508 || !exists("did_jsond_syntax_inits")
  if version < 508
    let did_jsond_syntax_inits = 1
    command -nargs=+ HiLink highlight link <args>
  else
    command -nargs=+ HiLink highlight def link <args>
  endif

  HiLink jsondSyntax Statement
  HiLink jsondName Type

  HiLink jsondComment Comment
  HiLink jsondMultilineComment Comment

  HiLink jsondTodo Todo
  HiLink jsondNote SpecialComment

  HiLink fallbackError Error

  delcommand HiLink
endif

let b:current_syntax = "jsond"

Only the #lang line, comments, and #:name elements are highlighted. The identifiers and the JSON values get red error highlights (and no, removing the fallback group didn't help).

I felt sure that using nextgroup=… skipwhite skipempty would work here, and I'm not sure how to fix it.

Answer 1

1. To debug the syntax one should be able to identify current syntax group under cursor (that question was asked multiple times before).
2. Now we see that jsondVarName never matches. That's because "plus" must be escaped. Also, there's no need in "word borders" as "plus"-quantifier is greedy. So it becomes simply /\k\+/.
3. Now jsondVarName seems to match more than it should. The most obvious way to fix it is to make it contained. It works because nextgroup effectively "overwrites" contained (see :h :syn-nextgroup).
4. JSON still doesn't work except jsonBraces. That's because jsonBraces are transparent and so no included JSON syntax is matched inside. Therefore you have to match braces yourself and to include JSON afterwards.

Answer 2

^{Apologies, dear reader, for the XY problem presented here.}

I can't accept Matt's answer, despite its help pointing out a problem in my syntax, because it didn't answer the underlying question (which I didn't correctly ask). Ultimately I wanted to embed any JSON values in the syntax highlighting. The syntax/json.vim that ships with vim doesn't do this according to the spec! For example, foo.json containing the number 123 or string "abc" has no highlighting. That syntax file assumes everything is an object or array, AFAICT.

So instead, I have implemented the grammar of the spec (sort of) in my syntax file. I include the full code here, but the most up-to-date version is in the repo at syntax/jsond.vim.

" Vim syntax file
" Language:     #lang jsond

if exists("b:current_syntax")
  finish
endif

" Hack: the syntax/json.vim file only permits numbers followed by blanks
" followed by some kind of object or array delimiter
" JSON the spec says a JSON is an element is a whitespace-delimited value, which
" can be any of an object, an array, a string, a number, or the keywords
" true/false/null
" Ref: https://www.json.org/json-en.html
syntax match jsonNumber "-\=\<\%(0\|[1-9]\d*\)\%(\.\d\+\)\=\%([eE][-+]\=\d\+\)\=\>"
syntax keyword jsonBoolean true false
syntax keyword jsonNull null

syntax cluster json contains=jsonObject,jsonArray,jsonNumber,jsonStringMatch,jsonBoolean,jsonNull

syntax region jsonArray matchgroup=jsonBraces start=/\[/ end=/]/ contains=@json fold
syntax region jsonObject matchgroup=jsonBraces start=/{/ end=/}/ contains=jsonKeyMatch fold

syntax match jsonKeyMatch /"\([^"]\|\\\"\)\+"[[:blank:]\r\n]*:/ contains=jsonKey nextgroup=@json skipwhite
syntax region jsonKey start=/"/ end=/"\ze[[:blank:]\r\n]*:/ contained

syntax match jsonStringMatch /"\([^"]\|\\\"\)\+"/ contains=jsonString
syntax region jsonString oneline start=/"/ skip=/\\\\\|\\"/ end=/"/ contains=jsonEscape contained

syntax match jsonEscape ,\\["\\/bfnrt], contained
syntax match jsonEscape /\\u\x\{4\}/ contained

" syntax iskeyword 33,35-39,42-58,60-90,94,95,97-122,126,_
" converted from decimal to char
" :s/\d\+/\=submatch(0)->str2nr()->nr2char()/g
" but corrected to remove duplicate _, move ^ to end
" also exclude comma, for JSON
syntax iskeyword @,!,#-',*-:,<-Z,a-z,~,_,^,,^
" expanded
" syntax iskeyword !,#,$,%,&,',*,+,,,-,.,/,0-9,:,<,=,>,?,@,A-Z,_,a-z,~,^,,^

syntax match jsondSyntax "#lang " nextgroup=jsondLang
syntax keyword jsondLang jsond

syntax keyword jsondName #:name nextgroup=jsondVarName skipwhite skipempty
syntax match jsondVarName ,\<\k\+\>, contained nextgroup=@json skipwhite skipempty

" Comments
syntax match jsondComment /;.*$/ contains=jsondTodo,jsondNote,@Spell
syntax region jsondMultilineComment start=/#|/ end=/|#/ contains=jsondMultilineComment,jsondTodo,jsondNote,@Spell

syntax keyword jsondTodo FIXME TODO XXX contained
syntax match jsondNote /\CNOTE\ze:\?/ contained

highlight default link jsondSyntax Statement
highlight default link jsondName Type

highlight default link jsondComment Comment
highlight default link jsondMultilineComment Comment

highlight default link jsondTodo Todo
highlight default link jsondNote SpecialComment

highlight default link jsonNumber Number
highlight default link jsonBoolean Boolean
highlight default link jsonNull Constant
highlight default link jsonString String
highlight default link jsonEscape Special
highlight default link jsonKey Label

let b:current_syntax = "jsond"