Discussions

ron_horne · Jun 9, 2023 11:09 AM

Hi All,

when attempting to open the html code of a webpage i only get the Cookie Consent text rather than the page content.

For example, being in the UK and attempting to open the jmp homepage this is what i get:


//:*/
Page = open("https://www.jmp.com/");

/*:

String( "<html><hea... 2588 total characters ...></html>

" ) assigned.

//:*/
print (Page);
/*:

"<html><head><script type=\!"text/javascript\!">
function callAjax(url, fallbackUrl, callback){
    var xmlhttp = new XMLHttpRequest();
    xmlhttp.onreadystatechange = function(){
        if (xmlhttp.readyState === 4 && xmlhttp.status === 200){
            callback(xmlhttp.responseText);
        }
    }
    xmlhttp.open(\!"GET\!", url, true);
    xmlhttp.timeout = 3000; // time in milliseconds
    xmlhttp.ontimeout = function() {
        console.log(\!"countrycode xhr request timed out\!");
        window.location.replace(fallbackUrl);
    }
    xmlhttp.send();
}

function findInLocales(locales, cookielocale) {
    var i = 0;
    var found = \!"\!";
    while (i < locales.length && !found) {
        if (locales[i].substring(2) === '_' + cookielocale.toLowerCase()) {
            found = locales[i];
        }
        i++;
    }
    return found;
}

function geoResetURL(locales) {
    var windowReplacement = window.location.href;
    var regex = /\/[a-zA-Z][a-zA-Z]_[a-zA-Z][a-zA-Z]\//;
    var regextest =  /[a-zA-Z][a-zA-Z]_[a-zA-Z][a-zA-Z]/;
    windowReplacement = windowReplacement.replace('.geo.', '.');
    var match = document.cookie.match(new RegExp('(^| )usr_locale=([^;]+)'));
    var cookielocale = '';
    if (match != null && match.length > 2) {cookielocale = match[2];}
    if (!cookielocale.match(regextest)) {
        var result;
        var code = Math.random().toString(36).substring(2, 15) + Math.random().toString(36).substring(2, 15);
        callAjax( '/services/countrycode?'+code, windowReplacement,
                function (result) {
                    var jsonresult = result.replace(/[()]/g, '');
                    cookielocale = findInLocales(locales, (JSON.parse(jsonresult)).address.country_code);
                    if (locales.indexOf(cookielocale) != -1) {
                        windowReplacement = windowReplacement.replace(regex, '/' + cookielocale + '/');
                    }
                  window.location.replace(windowReplacement);
                });
    } else {
        if (locales.indexOf(cookielocale) != -1) {
            windowReplacement = windowReplacement.replace(regex, '/' + cookielocale + '/'); 
        }
        window.location.replace(windowReplacement);
    }
}

(function () { 
var locales = ['en_us','en_be','de_at','zh_cn','en_dk','fr_fr','de_de','it_it','ja_jp','ko_kr','en_nl','en_ch','en_gb','en_ca','fr_ca','zh_tw','pt_br','es_mx','es_es','en_au','en_hk','en_my','en_ph','en_in','en_sg','es_ar','es_cl','es_co','es_pe']; 
geoResetURL(locales); 
  }) ();</script></head><body>
<p>Redirecting ...</p></body></html>

"

Many thanks for any suggestions in the right direction.

ron

jthi · Feb 1, 2022 01:49 PM

Are you trying to open the page in JMP or scrape the website? If you are trying to scrape (I'm not too familiar with web scraping), but you can try something like this:

https://stackoverflow.com/questions/57171353/scraping-a-webpage-using-python-beautiful-soup-that-req... then in JMP use New HTTP Request with fields or possibly Cookie (check scripting index for these).

It could even be that GET request with New HTTP Request() is enough without any extra fields

Names Default To Here(1);

request = New HTTP Request(
	URL("https://www.jmp.com/en_us/home.html"),
	Method("Get")
);
data = request << Send;

-Jarmo

View solution in original post

jthi · Jan 30, 2022 10:32 AM

Open Developer Tools in your browser (F12), select Network tab and go to jmp.com. From there take a look at the domains / files and try to find "correct one":

Now try opening that one.

This approach might work or at least get you closer, depending on the website and what you want to get from there.

-Jarmo

ron_horne · Jan 31, 2022 09:05 AM

Thank you @jthi,

I had a look in the developer tools but didn't manage to crack it yet as a command.

i managed to get the html of http://jmp.com/ by performing the following steps:

1) file>> internet open >> webpage.

2) open jmp.com as a webpage

3) accept / not accept the cookies interactively

4) run the following statement:

page = open ("https://www.jmp.com/en_gb/home.html")

notes: cookie approval only lasts within the same jmp session. After that preferences are not remembered.

This method is useful in my case since only fetch information from one domain in each session. therefore, approving manually is not too difficult. if the scrip crosses domains while fetching data it would be nice to somehow automate this.

this method also worked for the webpage i was actually fetching data from so it is reliable to that extent.

jthi · Feb 1, 2022 01:49 PM

Are you trying to open the page in JMP or scrape the website? If you are trying to scrape (I'm not too familiar with web scraping), but you can try something like this:

https://stackoverflow.com/questions/57171353/scraping-a-webpage-using-python-beautiful-soup-that-req... then in JMP use New HTTP Request with fields or possibly Cookie (check scripting index for these).

It could even be that GET request with New HTTP Request() is enough without any extra fields

Names Default To Here(1);

request = New HTTP Request(
	URL("https://www.jmp.com/en_us/home.html"),
	Method("Get")
);
data = request << Send;

-Jarmo

ron_horne · Feb 1, 2022 02:14 PM

Thank you @jthi, this works perfectly.

At this point i do want to scrape the page.

Discussions

Cookie Consent issue

Re: Cookie Consent issue

Re: Cookie Consent issue

Re: Cookie Consent issue

Re: Cookie Consent issue

Re: Cookie Consent issue

Recommended Articles