(Doesn't look like @filbranden or @AndrewHo-Lee are planning to answer this, so adding a Community Wiki answer with the diagnosis they figured out.)
The problem turned out to be that 'compatible' was set when the mapping was defined.
With this set, the 'cpoptions' setting contains the < flag, which means that Vim will not treat the <Esc> in the ...
Using :substitute to replace end/beginning of line with return got the results OP wanted.
nnoremap o :s/$/\r/ <cr>
nnoremap O :s/^/\r/ <cr>
I can't get just o to be remapped without a recursive error. But a reasonable compromise you might consider from this answer is to use oo for new line. This leaves single o ...