topic Re: Cookie Consent issue in Discussions

Cookie Consent issue

ron_horne — Fri, 09 Jun 2023 18:09:48 GMT

Hi All,

when attempting to open the html code of a webpage i only get the Cookie Consent text rather than the page content.

For example, being in the UK and attempting to open the jmp homepage this is what i get:


//:*/
Page = open("https://www.jmp.com/");

/*:

String( "<html><hea... 2588 total characters ...></html>

" ) assigned.

//:*/
print (Page);
/*:

"<html><head><script type=\!"text/javascript\!">
function callAjax(url, fallbackUrl, callback){
    var xmlhttp = new XMLHttpRequest();
    xmlhttp.onreadystatechange = function(){
        if (xmlhttp.readyState === 4 && xmlhttp.status === 200){
            callback(xmlhttp.responseText);
        }
    }
    xmlhttp.open(\!"GET\!", url, true);
    xmlhttp.timeout = 3000; // time in milliseconds
    xmlhttp.ontimeout = function() {
        console.log(\!"countrycode xhr request timed out\!");
        window.location.replace(fallbackUrl);
    }
    xmlhttp.send();
}

function findInLocales(locales, cookielocale) {
    var i = 0;
    var found = \!"\!";
    while (i < locales.length && !found) {
        if (locales[i].substring(2) === '_' + cookielocale.toLowerCase()) {
            found = locales[i];
        }
        i++;
    }
    return found;
}

function geoResetURL(locales) {
    var windowReplacement = window.location.href;
    var regex = /\/[a-zA-Z][a-zA-Z]_[a-zA-Z][a-zA-Z]\//;
    var regextest =  /[a-zA-Z][a-zA-Z]_[a-zA-Z][a-zA-Z]/;
    windowReplacement = windowReplacement.replace('.geo.', '.');
    var match = document.cookie.match(new RegExp('(^| )usr_locale=([^;]+)'));
    var cookielocale = '';
    if (match != null && match.length > 2) {cookielocale = match[2];}
    if (!cookielocale.match(regextest)) {
        var result;
        var code = Math.random().toString(36).substring(2, 15) + Math.random().toString(36).substring(2, 15);
        callAjax( '/services/countrycode?'+code, windowReplacement,
                function (result) {
                    var jsonresult = result.replace(/[()]/g, '');
                    cookielocale = findInLocales(locales, (JSON.parse(jsonresult)).address.country_code);
                    if (locales.indexOf(cookielocale) != -1) {
                        windowReplacement = windowReplacement.replace(regex, '/' + cookielocale + '/');
                    }
                  window.location.replace(windowReplacement);
                });
    } else {
        if (locales.indexOf(cookielocale) != -1) {
            windowReplacement = windowReplacement.replace(regex, '/' + cookielocale + '/'); 
        }
        window.location.replace(windowReplacement);
    }
}

(function () { 
var locales = ['en_us','en_be','de_at','zh_cn','en_dk','fr_fr','de_de','it_it','ja_jp','ko_kr','en_nl','en_ch','en_gb','en_ca','fr_ca','zh_tw','pt_br','es_mx','es_es','en_au','en_hk','en_my','en_ph','en_in','en_sg','es_ar','es_cl','es_co','es_pe']; 
geoResetURL(locales); 
  }) ();</script></head><body>
<p>Redirecting ...</p></body></html>

"

Many thanks for any suggestions in the right direction.

ron

Re: Cookie Consent issue

jthi — Sun, 30 Jan 2022 15:32:58 GMT

Open Developer Tools in your browser (F12), select Network tab and go to jmp.com. From there take a look at the domains / files and try to find "correct one":

Now try opening that one.

This approach might work or at least get you closer, depending on the website and what you want to get from there.

Re: Cookie Consent issue

ron_horne — Mon, 31 Jan 2022 14:05:01 GMT

Thank you @jthi,

I had a look in the developer tools but didn't manage to crack it yet as a command.

i managed to get the html of http://jmp.com/ by performing the following steps:

1) file>> internet open >> webpage.

2) open jmp.com as a webpage

3) accept / not accept the cookies interactively

4) run the following statement:

page = open ("https://www.jmp.com/en_gb/home.html")

notes: cookie approval only lasts within the same jmp session. After that preferences are not remembered.

This method is useful in my case since only fetch information from one domain in each session. therefore, approving manually is not too difficult. if the scrip crosses domains while fetching data it would be nice to somehow automate this.

this method also worked for the webpage i was actually fetching data from so it is reliable to that extent.

Re: Cookie Consent issue

jthi — Tue, 01 Feb 2022 18:49:08 GMT

Are you trying to open the page in JMP or scrape the website? If you are trying to scrape (I'm not too familiar with web scraping), but you can try something like this:

https://stackoverflow.com/questions/57171353/scraping-a-webpage-using-python-beautiful-soup-that-requires-i-agree-to-cooki then in JMP use New HTTP Request with fields or possibly Cookie (check scripting index for these).

It could even be that GET request with New HTTP Request() is enough without any extra fields

Names Default To Here(1);

request = New HTTP Request(
	URL("https://www.jmp.com/en_us/home.html"),
	Method("Get")
);
data = request << Send;

Re: Cookie Consent issue

ron_horne — Tue, 01 Feb 2022 19:14:12 GMT

Thank you @jthi, this works perfectly.

At this point i do want to scrape the page.