1

Writing vim syntax files never ceases to make me feel like an idiot.

I have a new custom language call jsond, and I wanted to get syntax highlighting working in vim. Using racket syntax works pretty well (at least with my fork).

Here are some example files:

#lang jsond

#:name albums
["Cycles", "Flatspin", "Long Way to Climb"]

#:name favorites
{
    "Jonathan Gordon": 0,
    "Brad Rubinstein": 2,
    "Paul Willmott": 1,
    "Jason Hall": 0,
    "Adam Rich": 1
}
#lang jsond

#:name abc "abc"

;; comment

#:name num 123

#:name array [1, 2, {"xyz": "abc"}, true, false, null]

As you can see, the language is effectively #:name <id> <json>, with a few other things.

Here's my syntax file:

" Vim syntax file
" Language:     #lang jsond

" Initializing:
if version < 600
  syntax clear
elseif exists("b:current_syntax")
  finish
endif

syntax match fallbackError /./

syntax include @json syntax/json.vim

if version < 600
  set iskeyword=33,35-39,42-58,60-90,94,95,97-122,126,_
else
  " syntax iskeyword 33,35-39,42-58,60-90,94,95,97-122,126,_
  " converted from decimal to char
  " :s/\d\+/\=submatch(0)->str2nr()->nr2char()/g
  " but corrected to remove duplicate _, move ^ to end
  syntax iskeyword @,!,#-',*-:,<-Z,a-z,~,_,^
  " expanded
  " syntax iskeyword !,#,$,%,&,',*,+,,,-,.,/,0-9,:,<,=,>,?,@,A-Z,_,a-z,~,^
endif

syntax match jsondSyntax "#lang " nextgroup=jsondLang
syntax keyword jsondLang jsond

syntax keyword jsondName #:name nextgroup=jsondVarName skipwhite skipempty
syntax match jsondVarName ,\<\k+\>, nextgroup=@json skipwhite skipempty

" Comments
syntax match jsondComment /;.*$/ contains=jsondTodo,jsondNote,@Spell
syntax region jsondMultilineComment start=/#|/ end=/|#/ contains=jsondMultilineComment,jsondTodo,jsondNote,@Spell

syntax keyword jsondTodo FIXME TODO XXX contained
syntax match jsondNote /\CNOTE\ze:\?/ contained

" Define the default highlighting.
" For version 5.7 and earlier: only when not done already
" For version 5.8 and later: only when an item doesn't have highlighting yet
if version >= 508 || !exists("did_jsond_syntax_inits")
  if version < 508
    let did_jsond_syntax_inits = 1
    command -nargs=+ HiLink highlight link <args>
  else
    command -nargs=+ HiLink highlight def link <args>
  endif

  HiLink jsondSyntax Statement
  HiLink jsondName Type

  HiLink jsondComment Comment
  HiLink jsondMultilineComment Comment

  HiLink jsondTodo Todo
  HiLink jsondNote SpecialComment

  HiLink fallbackError Error

  delcommand HiLink
endif

let b:current_syntax = "jsond"

Only the #lang line, comments, and #:name elements are highlighted. The identifiers and the JSON values get red error highlights (and no, removing the fallback group didn't help).

I felt sure that using nextgroup=… skipwhite skipempty would work here, and I'm not sure how to fix it.

3
  • LOL. I was about to write my usual comment recommending Learn Vimscript the Hard Way...then I saw who wrote the question. :)
    – B Layer
    Aug 6, 2021 at 19:46
  • 1
    @BLayer xD yeah, the title is perhaps a bit mis-leading about the experience of the poster… OTOH, syntax files are hard for me, no matter how many times I read the manual.
    – D. Ben Knoble
    Aug 6, 2021 at 19:48
  • 1
    For a long time, I've been meaning to spend some time really learning SH but there's always something else that comes up!
    – B Layer
    Aug 6, 2021 at 19:50

2 Answers 2

2
  1. To debug the syntax one should be able to identify current syntax group under cursor (that question was asked multiple times before).
  2. Now we see that jsondVarName never matches. That's because "plus" must be escaped. Also, there's no need in "word borders" as "plus"-quantifier is greedy. So it becomes simply /\k\+/.
  3. Now jsondVarName seems to match more than it should. The most obvious way to fix it is to make it contained. It works because nextgroup effectively "overwrites" contained (see :h :syn-nextgroup).
  4. JSON still doesn't work except jsonBraces. That's because jsonBraces are transparent and so no included JSON syntax is matched inside. Therefore you have to match braces yourself and to include JSON afterwards.
3
  • Huh, that’s so weird that I left out the plus escape, thanks! And the rest is also good to know, I’m surprised about the JSON. What about non-braced values? I’ll look later and see for myself, of course
    – D. Ben Knoble
    Aug 7, 2021 at 11:20
  • @D.BenKnoble jsonBraces is actually transparent fold without any contained elements. So its contents match upper level syntax elements. Obviously, it only works if json is the main syntax.
    – Matt
    Aug 7, 2021 at 12:25
  • This is actually non-trivial because the syntax/json.vim does not really implement the json grammar... I haven’t quite figured it out, but it seems to only want key-value pairs in objects or arrays. Bare stuff (numbers, strings) aren’t highlighted at the very top level, only inside containers. I think I’m going to have to re-implement the parts I need, since my language allows the exact JSON spec.
    – D. Ben Knoble
    Aug 8, 2021 at 1:57
1

Apologies, dear reader, for the XY problem presented here.

I can't accept Matt's answer, despite its help pointing out a problem in my syntax, because it didn't answer the underlying question (which I didn't correctly ask). Ultimately I wanted to embed any JSON values in the syntax highlighting. The syntax/json.vim that ships with vim doesn't do this according to the spec! For example, foo.json containing the number 123 or string "abc" has no highlighting. That syntax file assumes everything is an object or array, AFAICT.

So instead, I have implemented the grammar of the spec (sort of) in my syntax file. I include the full code here, but the most up-to-date version is in the repo at syntax/jsond.vim.

" Vim syntax file
" Language:     #lang jsond

if exists("b:current_syntax")
  finish
endif

" Hack: the syntax/json.vim file only permits numbers followed by blanks
" followed by some kind of object or array delimiter
" JSON the spec says a JSON is an element is a whitespace-delimited value, which
" can be any of an object, an array, a string, a number, or the keywords
" true/false/null
" Ref: https://www.json.org/json-en.html
syntax match jsonNumber "-\=\<\%(0\|[1-9]\d*\)\%(\.\d\+\)\=\%([eE][-+]\=\d\+\)\=\>"
syntax keyword jsonBoolean true false
syntax keyword jsonNull null

syntax cluster json contains=jsonObject,jsonArray,jsonNumber,jsonStringMatch,jsonBoolean,jsonNull

syntax region jsonArray matchgroup=jsonBraces start=/\[/ end=/]/ contains=@json fold
syntax region jsonObject matchgroup=jsonBraces start=/{/ end=/}/ contains=jsonKeyMatch fold

syntax match jsonKeyMatch /"\([^"]\|\\\"\)\+"[[:blank:]\r\n]*:/ contains=jsonKey nextgroup=@json skipwhite
syntax region jsonKey start=/"/ end=/"\ze[[:blank:]\r\n]*:/ contained

syntax match jsonStringMatch /"\([^"]\|\\\"\)\+"/ contains=jsonString
syntax region jsonString oneline start=/"/ skip=/\\\\\|\\"/ end=/"/ contains=jsonEscape contained

syntax match jsonEscape ,\\["\\/bfnrt], contained
syntax match jsonEscape /\\u\x\{4\}/ contained

" syntax iskeyword 33,35-39,42-58,60-90,94,95,97-122,126,_
" converted from decimal to char
" :s/\d\+/\=submatch(0)->str2nr()->nr2char()/g
" but corrected to remove duplicate _, move ^ to end
" also exclude comma, for JSON
syntax iskeyword @,!,#-',*-:,<-Z,a-z,~,_,^,,^
" expanded
" syntax iskeyword !,#,$,%,&,',*,+,,,-,.,/,0-9,:,<,=,>,?,@,A-Z,_,a-z,~,^,,^

syntax match jsondSyntax "#lang " nextgroup=jsondLang
syntax keyword jsondLang jsond

syntax keyword jsondName #:name nextgroup=jsondVarName skipwhite skipempty
syntax match jsondVarName ,\<\k\+\>, contained nextgroup=@json skipwhite skipempty

" Comments
syntax match jsondComment /;.*$/ contains=jsondTodo,jsondNote,@Spell
syntax region jsondMultilineComment start=/#|/ end=/|#/ contains=jsondMultilineComment,jsondTodo,jsondNote,@Spell

syntax keyword jsondTodo FIXME TODO XXX contained
syntax match jsondNote /\CNOTE\ze:\?/ contained

highlight default link jsondSyntax Statement
highlight default link jsondName Type

highlight default link jsondComment Comment
highlight default link jsondMultilineComment Comment

highlight default link jsondTodo Todo
highlight default link jsondNote SpecialComment

highlight default link jsonNumber Number
highlight default link jsonBoolean Boolean
highlight default link jsonNull Constant
highlight default link jsonString String
highlight default link jsonEscape Special
highlight default link jsonKey Label

let b:current_syntax = "jsond"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.