Refactoring is not an activity you should set time aside to do. You refactor because you want to do something else and refactoring helps you with that other thing.
So, when should you refactor:
The rule of three The third time you do something similar, you should refactor it.
When you add a function You want to add a feature to an existing code you have written or have been written by someone else.