Question - While paginating over the lower ranked documents the query performance is very bad and the response times are high
Answer - Search engines are built to be able to fetch the top N results very quickly. As a result of that paginating over the lower ranked documents is very slow. This is because to fetch documents 5000 to 5010 the search engine needs to find 5010 documents, rank them and then be able to return the documents between the position 5000 and 5010.
Luckily from Solr 4.7 a new feature was introduced. It's called deep paging. It's an efficient way of scrolling through the documents in a result set.
For more information of how to use the deep paging feature in Solr you could refer to the "Fetching A Large Number of Sorted Results: Cursors" section here - https://cwiki.apache.org/confluence/display/solr/Pagination+of+Results