Why my visual selection buffer contains ^@

Question

I am writing a function to reformat a visual selection:

function! FormatText() range
  " Calling this function has ended visual mode, so it must be started
  " again before the selection can be yanked into the unnamed register.
  normal! gvy
  let selection = @"
  echomsg 'Z'.selection.'Z'
  let cmd = 'sort'
  let output = system(cmd, selection)
  echomsg 'X'.output.'X'
  silent exe "'<,'>s/.*/" . output . "/"
endfunction

The examination of @" value reveals that spaces and new lines are randomly replaced by ^@.

Any idea why this is happening and how to handle this? Is there a better approach to get the context of the visual selection?

UPDATE: After the feedback I know now that ^@ is a new line replacement. I also provided the actual function. I have to withdraw the statement about spaces, only new lines are replaced by ^@. The spaces were replaced by external program because it could not handle 0 bytes. To eliminate the external program effect I am using the sort to illustrate what's happening.

Debugging steps:

1. open a file with the following content: z line a line

2. visually select both line and trigger the function by the following mapping: vnoremap f :call FormatText()

3. echomsg output (works as expected) Zzline^@a line^@Z Xaline^@z line^@X

4. The selection in file changed to: aline^@z line^@ aline^@z line^@

I have more questions now.

a) Why is the selection contents doubled? Is this approach wrong:

  silent exe "'<,'>s/.*/" . output . "/"

I tried \%V

  silent exe "'<,'>s/\\%V.*\\%V/" . output . "/"

but the result is the same.

b) Is there a way to get the visual selection as it is, without transformation?

c) How does built-in gq with 'formatexpr' works? Does it convert ^@ to new lines first? Would it be awkward to do it for all type of file formats?

Answer 1

Vim translates the new line (^J, "\n") into the null character (^@ that you can insert with Ctrl vx00) when moving the clipboard into a variable (@")

The echom method output the variable using the ^ syntax where echo output the variable using the conversion (^@ -> <CR>)

When you copy several lines you'll get the new line character (^J) translated into the null character (^@)

The setreg method convert back the null character (^@) into the new line character (^J)

Remark: If you look at the content of the register (:reg ") you can verify that the register contains the end of the line character (^J)

In order to replace the selection I would use the paste method:

function! FormatText() range
  " Calling this function has ended visual mode, so it must be started
  " again before the selection can be yanked into the unnamed register.
  normal! gvy
  let selection = @"
  echomsg 'Z'.selection.'Z'
  let cmd = 'sort'
  let output = system(cmd, selection)
  echomsg 'X'.output.'X'

  " silent exe "'<,'>s/.*/" . output . "/"
  call setreg('"', output)
  normal! gvp
endfunction

Answer 2

Just in case someone else needs the solution I found with the help of contributors. The following function passes VIM visual selection to STDIN of the external program (sort) and replaces the current selection with the STDOUT of that program.

function! FormatSQL() range
  normal! gvy
  let selection = getreg('"', 1, 1)
  let cmd = 'sort'
  let output = system(cmd, selection)
  call setreg('"', output)
  normal! gvp
endfunction

getreg('"', 1, 1) copies the selection register into a list. This allows not to worry about ^@.

setreg('"', output) transparently copies the output into the selection register without meddling with NL or ^@.