Nothing to do with what I thought at all. The issue is pagination. Need to add ?per_page=100 to get more than 10. According to a comment on this page if you have more than 100 you need to use pagination.

Just in case anyone makes the same mistake – I had added 20 terms via the UI but only 10 showed up in Rest calls. I added a few more in code wp_insert_term – and these showed up in rest calls. i assumed that was the issue but it seems something to do with ordering – which is why the ones added in code showed up. Actual issue was paginaton.