And you're using FastVectorHighlighter... which is supposed to be... well fast right! Well, if you have a document with a lot of potential phrases, you may have a problem. Read this JIRA:
Use hl.phraseLimit. What's happening is that there are too many phrases and it kind of goes out of control. So limit it and your highlighting times will come back to normal.