Cache is accumulating /page/, copies of home page

I recently started using a caching plugin. I noticed it started caching /page/2/, /page/3/, etc., and they are all copies of the home page. If I browse to a new page number, it shows the home page and that new one is cached.

I don’t have any pagination. Shouldn’t such urls give a page not found error? I also wonder why someone (more likely a bot?) would hit those urls.