I have created a custom role that only has access to certain pages and their children/parents. I have used map_meta_cap filter to solve this. However even though it goes through my function correctly it will not function properly, return ['do_not_allow']; is not working as intended and I don’t know why, it seemingly does not do anything.

//Custom Roles (stored to db on theme activations)
add_action( 'load-themes.php', 'staby_add_theme_roles');
function staby_add_theme_roles() {
    global $pagenow;

    if ( 'themes.php' == $pagenow && isset($_GET['activated'])) {
        add_role(
            'test',
            __( 'Test', 'staby' ),
            array(
                'pages_id' => array(6911),
            )
        );
        $role = get_role('test');
        $role->add_cap('read');
        $role->add_cap('edit_pages');
        $role->add_cap('edit_published_pages');
        $role->add_cap('edit_others_pages');
        $role->add_cap('publish_pages');
        $role->add_cap('pages_restriction');
    }
}
    //Role cap for different pages
function staby_map_meta_cap( $caps, $cap, $user_id, $args ) {
    // // If the capability being filtered isn't of our interest, just return current value
    
    if ( in_array($cap, ['edit_pages']) ) {
        
        // First item in $args array should be page ID
        if (!staby_role_can_edit( $user_id, $args[0] ) ) {
            // User is not allowed, let's tell that to WP
            return ['do_not_allow'];
        }
    }
    
    // Otherwise just return current value
    return $caps;
}
add_filter( 'map_meta_cap', 'staby_map_meta_cap', 10, 4 );

//See if role can edit correspondent page 
function staby_role_can_edit( $user_id, $page_id) {
    $page = get_post( $page_id );
   // let's find the topmost page in the hierarchy
   while( $page && (int) $page->parent ) {
     $page = get_post( $page->parent );
   }
   
   if ( ! $page ) {
     return false;
   }

   $user = new WP_User($user_id);
   if ($user->allcaps['pages_id']) {
       $user_pages = $user->allcaps['pages_id'];
       if (!in_array($page->ID, $user_pages)) {
            return false;
       }
   }
   
   return true;

}

The only thing I have noticed is that the map_meta_cap filter runs multiple times, and the $args variable is always empty but it somehow still retrieves the $args[0] (page id) in my custom function staby_role_can_edit( $user_id, $page_id ) (but when I call it in that function it’s null?). I have no idea why it’s not working at all.

Any help is very appreciated! Thanks.

Update:

Tried with user_has_cap filter

    function staby_cap_filter( $allcaps, $cap, $args ) {
    
    //Bail out if role does not have restriction
    // Bail out if we're not asking about a post:
    if ( 'edit_post' != $args[0] || !$allcaps['pages_restriction']){
        $allcaps[$cap[0]] = false;
        return $allcaps;
    }
    if (!staby_role_can_edit( $args[1], $args[2] ) ) {
        $allcaps[$cap[0]] = false;
        return $allcaps;
    }
    return $allcaps;
 
}
add_filter( 'user_has_cap', 'staby_cap_filter', 10, 3 );

However even though it finds the correct page I don’t know how to make it restrict access correctly.

$allcaps[$cap[0]] = false;
return $allcaps;

As you can see it sets the primitive capability to false if the user doesn’t have access to the page.