In short, you set
'operatorfunc' to point to the name of a function, then write a mapping that uses the
g@ command will take a motion (in operator-pending mode) and then will call the function set in
'operatorfunc', passing it an argument to indicate whether the motion was char-wise or line-wise. Additionally, it will set the
'] markers pointing to the start and end of the motion.
See section Mapping an Operator in Vim help (also accessible through
:help :map-operator) which will have some more details on that and an example. It will also have pointers to the documentation on
You might also want to read the Grep Operator, Part 2 section of the excellent Learn Vimscript the Hard Way. Actually, you should probably read the whole thing, if Vimscript interests you, but that particular section handles this very specific case of creating a mapping that takes a motion in operator-pending mode.
You might also want to make your mapping work in Visual mode. Both the Vim help and "Learning Vimscript the Hard Way" show examples of writing a function that will work for both, and the appropriate mappings to go along with that.