I can't reproduce the behavior you mention without a plugin.
The csv.vim plugin provides the behavior you describe in the introduction of your question.
This plugin use only four colors (two for the header and two for the body):
hi CSVDelimiter guifg=#616E88 guibg=#2E3440
hi CSVColumnEven guifg=#D8DEE9 guibg=#434c5e
hi CSVColumnOdd guifg=#D8DEE9 guibg=#2E3440
hi CSVColumnHeaderEven guifg=#D8DEE9 guibg=#434c5e
hi CSVColumnHeaderOdd guifg=#D8DEE9 guibg=#2E3440
The rainbow_csv plugin provide the behavior you would like to have.
This plugin use an arbitrary number of colors (I use ten):
let g:rcsv_colorpairs = [
\ ['magenta', '#B48EAD'],
\ ['red', '#BF616A'],
\ ['lightred', '#D08770'],
\ ['yellow', '#EBCB8B'],
\ ['green', '#A3BE8C'],
\ ['cyan', '#8FBCBB'],
\ ['lightblue', '#88C0D0'],
\ ['blue', '#5E81AC'],
\ ['grey', '#616E88'],
\ ['NONE', 'NONE']
\ ]