I want to use different config files for different directories. For example, if I want to use different plugins(via a plugin manager) or use different color themes. Is there a way to do so?
I guess you could use any local vimrc plugin as long as your plugin manager supports on-the-fly loading of plugins.
The almost related Q/A here is missing a few contestants: How to make vimrc settings applicable for a directory tree only. The one on SO is a little bit more complete: https://stackoverflow.com/questions/456792/vim-apply-settings-on-files-in-directory/456889#456889
Note that we cannot use different colorthemes simultaneously in a same vim session. At best the colortheme would change every time you jump into a buffer from a different project -- it would quickly become annoying IMO.