How to easily exclude first post in pre_get_posts only in queries other than page 1?

In home/category pages (blog), I need to have 4 posts in first page, then 3 posts in all other pages. Problem is, I cannot simply alter posts_per_page, because in following pages I would have wrong posts. I also have to exclude the first one on the same pages.

I did a fast and row test excluding the first post ID manually, and the thing worked properly. But now I need to get that ID programmatically to make it ready for production:

add_action('pre_get_posts','lr_custom_query');
function lr_custom_query($wpquery) {
  if(is_admin() || !$wpquery->is_main_query()) return $wpquery; //do nothing

  if(is_category() || is_home()){
    if($wpquery->query_vars['paged'] == 0){
      $wpquery->set( 'posts_per_page', 4);
    } else {
      $wpquery->set( 'posts_per_page', 3);
      $wpquery->set( 'post__not_in', array(541)); // <<<<<<< see here
    }
  }
}

How to always retrieve the first post? What’s the smarter way?

P.s.
there are other similar questions using offset instead of ‘post__not_in’, but I think using ‘post__not_in’ is a smarter and easier way.