I am loading the latest videos from a YouTube channel to WordPress, using the YouTube API:

function load_feed() {
    $url = add_query_arg(
        array(
            'part' => 'snippet',
            'channelId' => '[YouTube channel ID]',
            'maxResults' => 3,
            'order' => 'date',
            'type' => 'video',
            'key' => '[YouTube API key]'
        ),
        'https://www.googleapis.com/youtube/v3/search'
    );

    $response = wp_remote_get(esc_url_raw($url));

    return json_decode(wp_remote_retrieve_body($response), true);
}

Everything works well when in the API settings on Google Developers I have:

Application restrictions: None

But when I set:

Application restrictions: HTTP referrers (web sites)

Website restrictions: www.mysite.com/*

I get the following response from the API:

{
    "error": {
        "code": 403,
        "message": "Requests from referer https://www.googleapis.com/youtube/v3/search?part=snippet&channelId=[Channel ID]&maxResults=3&order=date&type=video&key=[API key] are blocked.",
        "errors": [
            {
                "message": "Requests from referer https://www.googleapis.com/youtube/v3/search?part=snippet&channelId=[Channel ID]&maxResults=3&order=date&type=video&key=[API key] are blocked.",
                "domain": "global",
                "reason": "forbidden"
            }
        ],
        "status": "PERMISSION_DENIED"
    }
}

It looks as though the API detects the request was made from googleapis.com and not from mysite.com, where I have the WP installation and am running the above code.

Why is this error occurring and what can I do to fix it?