I found a fix for it with extensive googling. All I had to do was type
/nickserv set keepmodes on and the problem seems to have gone away, haha
I have no idea why that is a solution, but it worked for someone else with same problem.
Thanks for your input guys. I appreciate it.
