topic Re: Use JSL to get data from the web in Discussions

Use JSL to get data from the web

uday_guntupalli — Fri, 28 Oct 2016 13:25:50 GMT

Hello All,

After looking at this discussion (Use JSL to Scrape Data From the Web and Predict Football Wins!) - I got very excited about the # of data sets available online to enhance my JSL skills. However - I haven't had much success after trying the code provided in this post. I tried to write to the author and am still waiting to hear back. If some one has done this before - can someone - please show how to get data from the web .

Lets say For E.g. I want to get the stock price for Twitter , its 52 - week low , 52-week high and current price - how can I do that in JSL .

Any inputs are appreciated.

Best

uday

Re: Use JSL to get data from the web

msharp — Fri, 15 Apr 2016 19:46:52 GMT

There are many methods to get data from the web. The best method depends on the data source. In the example you posted, the user is simply using an Open() command and tells it which data table in the HTML to pull. This is by far the easiest, but not always possible. Some sites, like Twitter have set up API's. This method is slightly more complicated but extremely more powerful.

Have you done your research? Do you know where you want to get the stock price information? What's the data source?

Re: Use JSL to get data from the web

uday_guntupalli — Fri, 15 Apr 2016 20:46:51 GMT

msharp,

I have a couple of options that I have looked at :

1. Bar Chart .com

2. Yahoo Finance

3. Google Finance

But I dont know if we need to ping the api's of these websites or if they are accessible as HTML tables.

Best
uday

Re: Use JSL to get data from the web

Jeff_Perkinson — Fri, 28 Oct 2016 13:25:55 GMT

The Yahoo Finance Fetcher JMP Add-in would be a good place to start.

-Jeff

Re: Use JSL to get data from the web

uday_guntupalli — Fri, 15 Apr 2016 21:01:48 GMT

Hello Jeff ,

Thank you . I have already downloaded the addin. my objective is to be able to learn and understand how to do it in JSL and I dont know how to tap in to the code behind the addin . Also - the addin just seems to open the stock information in a browser against importing the data into JMP (which is what I would like to do) . I wish to import the data from these public repositories to further my analytical skills and coding skills in JSL .

Best

Uday

Re: Use JSL to get data from the web

msharp — Fri, 15 Apr 2016 21:27:33 GMT

So, using yahoo finance as an example I show you the two general methods. If you go to any website you can just use the open command and specify the HTML table. This requires the data to be stored in an HTML table.

Below that, I show you the yahoo finance table.csv api. Adding s=TWTR to the end will create a CSV file for all the twitter data you could then play with in JMP. If you want to know more about the api, this page is helpful.

Yahoo Finance (hidden) API | Ido Green

//Just browsing the web, was able to get these tables

open("http://finance.yahoo.com/echarts?s=TWTR#{\!"allowChartStacking\!":true}", HTML Table(2));

open("http://finance.yahoo.com/q;_ylt=AwrXgiOYWBFXxGUAYiiTmYlQ;_ylu=X3oDMTE0bWJta2VyBGNvbG8DZ3ExBHBvcwMxBHZ0aWQDVklEUUJDS18xBHNlYwNzYw--?s=TWTR", HTML Table(2));

open("http://finance.yahoo.com/q;_ylt=AwrXgiOYWBFXxGUAYiiTmYlQ;_ylu=X3oDMTE0bWJta2VyBGNvbG8DZ3ExBHBvcwMxBHZ0aWQDVklEUUJDS18xBHNlYwNzYw--?s=TWTR", HTML Table(3));

//This is the yahoo finance api interface

open("http://ichart.finance.yahoo.com/table.csv?s=TWTR");

Re: Use JSL to get data from the web

msharp — Fri, 15 Apr 2016 21:29:30 GMT

To access the code behind an Add-in go to View>>Add-Ins...>> Select the Add-in of choice, then select the link next to Home Folder. This will take you to the folder where all the add-in scripts and files are stored on your local hard drive.

Edit: I may have beat Jeff to the punch, but he has a sweet picture that helps clarify.

Re: Use JSL to get data from the web

Jeff_Perkinson — Fri, 15 Apr 2016 21:30:45 GMT

You can get to the JSL for the add-in by going to View->Add-Ins... and then select the Yahoo Fetcher add-in and click on the Home Folder link.

It will open the folder in Explorer (or the Finder if you're on a Mac). Then you can examine the .JSL file that the add-in runs.

-Jeff

Edited to add: msharp beat me to it.

Re: Use JSL to get data from the web

uday_guntupalli — Wed, 19 Oct 2016 02:25:56 GMT

@msharp:

I seem to get the same error as I did when I tried to use the Open command earlier. Does this have anything to do with the settings ?

Re: Use JSL to get data from the web

msharp — Fri, 15 Apr 2016 22:05:19 GMT

I can't say what your problem is, but all those commands worked for me. I would start with basic troubleshooting steps. Are you connected to internet? Try copying and pasting the website into the browser, does it bring up a page? (note: I would try the second page, I had to escape quotation marks in the first web address.)

What version of JMP are you running? If it's an older version, does it not support the functionality? Are you on a mac or windows?

Re: Use JSL to get data from the web

uday_guntupalli — Fri, 15 Apr 2016 22:14:16 GMT

@msharp

I am connected to the internet.

I am able to open the pages in my browser ,

I am using JMP 12

I have Windows.

uday

Re: Use JSL to get data from the web

msharp — Fri, 15 Apr 2016 23:08:11 GMT

Yeah, I figured you were connected to the internet since your posting on the forums

Like I said, I'm not sure what the issue is. I'd just try the normal troubleshooting techniques:

Opening closing JMP.

Restarting your computer.

Clearing cache.

Reinstalling JMP? There is a JMP 12.2 you could update to.

Alternatively you can contact support@jmp.com. They might be able to provide additional support.

Re: Use JSL to get data from the web

Craige_Hales — Sat, 16 Apr 2016 14:19:02 GMT

the 302 error is a redirection. I don't know why you are seeing that; it might be a firewall behavior.

If you want to try to puzzle it out, run this script, a line at a time, and see what each line produces in the JMP log.

site="finance.yahoo.com";

url="http://finance.yahoo.com/echarts?s=TWTR#{\!"allowChartStacking\!":true}";

tCall = Socket();

tCall << connect( site, "80" );

tCall << Send( Char To Blob( "GET " || url || " HTTP/1.0~0d~0a~0d~0a", "ASCII~HEX" ) );

tMessage = tCall << Recv( 1000 );

text = Blob To Char( tMessage[3], "ASCII~HEX" );

Show( text );

tCall << Close();

connect should say: {"connect", "ok"}

send should say: {"Send", "ok", Char To Blob( "", "ascii~hex" )}

recv should say: {"Recv", "ok", Char To Blob(

"HTTP/1.0 200 OK~0D~0AX-Frame-Options: SAMEORIGIN~0D~0AX-Content-Type-Options: nosniff~0D~0AX-XSS-Protection: 1; mode=block~0D~0AContent-Type: text/html~0D~0ASet-Cookie: B=c3r14p5bh4hr3&b=3&s=p2; expires=Sat, 16-Apr-2018 14:08:35 GMT; path=/; domain=.yahoo.com~0D~0ADate: Sat, 16 Apr 2016 14:08:35 GMT~0D~0AServer: ATS~0D~0AVary: X-Ssl~0D~0AVia: http/1.1 media-border115.global.media.bf1.yahoo.com (ApacheTrafficServer [cMsSf ]), http/1.1 media-router4.prod.media.bf1.yahoo.com (ApacheTrafficServer [cMsSf ]), http/1.1 r25.ycpi.dcb.yahoo.net (ApacheTrafficServer [cMsSf ])~0D~0AAge: 0~0D~0AY-Trace: BAEAQAAAAACXg0qXxLFf4AAAAAAAAAAAw514XKmQNkkAAAAAAAAAAAAFMJqpyGJPAAUwmqnLTak2C_LwAAAAAA--~0D~0A~0D~0A<~21DOCTYPE html>~0A<html>~0A<head>~0A <~21-- customizable : anything you expected. -->~0A <title>TWTR Interactive Stock Chart | Yahoo~21 Inc. Stock - Yahoo~21 Finance</title>~0A~0A <meta http-equiv=~22X-UA-Compatible~22 content=~22IE=edge,chrome=1~22 />~0A <meta http-equiv=~22Content-Type~22 content=~22text/html; charset=UTF-8~22 />~0A~0A~0A~0A ~0A <link rel=~22stylesheet~22 type","ascii~hex")}

If you don't get the 200 OK response, maybe you'll get some information about the 302 response.

Re: Use JSL to get data from the web

uday_guntupalli — Mon, 18 Apr 2016 13:48:41 GMT

Craige@JMP :

text = "HTTP/1.1 302 Found~0D~0ALocation: http://tmp1mdf01-fpx01/?cfru=aHR0cDovL2ZpbmFuY2UueWFob28uY29tL2VjaGFydHM/cz1UV1RSI3siYWxsb3dDaGFydFN0YWNraW5nIjp0cnVlfQ==~0D~0ACache-Control: no-cache~0D~0APragma: no-cache~0D~0AContent-Type: text/html; charset=utf-8~0D~0AConnection: close~0D~0AContent-Length: 624~0D~0A~0D~0A<HTML><HEAD>~0A<TITLE>Redirect</TITLE>~0A</HEAD>~0A<BODY>~0A~0A<big></big> ~0A~0A<blockquote>~0A<TABLE border=0 cellPadding=1 width=~2280%~22>~0A<TR><TD>~0A~0A<big>Redirect (authentication_redirect_to_virtual_host)</big>~0A ~0A ~0A~0A</TD></TR>~0A<TR><TD>~0A~0AYou are being redirected to the authentication virtual host.~0A~0A</TD></TR>~0A<TR><TD>~0A~0A~0A~0A</TD></TR>~0A<TR><TD>~0A~0A ~0AFor assistance, contact your network support team.~0A~0A</TD></TR>~0A</TABLE>~0A</blockquote>~0A~0A</BODY></HTML>~0A";

{"Close", "ok"}

This is what I got

Re: Use JSL to get data from the web

msharp — Mon, 18 Apr 2016 14:39:10 GMT

Key information from above:

Redirect (authentication_redirect_to_virtual_host)

You are being redirected to the authentication virtual host.

For assistance, contact your network support team.

Doing a quick search through google, everything seems to be pointing to a proxy issue. This is something your company network support team should be able to easily solve. Have you contacted them?

I might personally try manually setting up the proxy server information in any web browsers you might have, since you are on Windows I'd start with IE. Change proxy server settings in Internet Explorer - Windows Help
However, this will require you to contact the network support team anyways, since I assume you don't know your company's proxy settings.

Re: Use JSL to get data from the web

Craige_Hales — Mon, 18 Apr 2016 14:43:59 GMT

I'm out of my depth. from the messages (which include one saying contact your support team) I think it is a firewall/proxy issue. See:

http://stackoverflow.com/questions/2408964/why-am-i-getting-authentication-redirect-to-virtual-host-error

(which I think might be asking the correct question)

Re: Use JSL to get data from the web

theresagoodyear — Thu, 21 Apr 2016 13:56:16 GMT

Hi Uday.

I was interested in getting football data from the web as well after attending that seminar in San Diego. I created the attached script and got quarterback data from espn.com. I figured out I could get historical data simply be replacing the link with the corresponding year. Once I got historical data and concatenated it, I ran a regression analysis and found that 'passing yards' is the biggest contributor to the quarterback's rank on the ESPN top QB list.

I've modeled other analysis after this one. Hope this helps. Thank you.

Re: Use JSL to get data from the web

uday_guntupalli — Thu, 21 Apr 2016 14:17:27 GMT

theresagoodyear : Thanks for sharing the script and analysis Theresa. I think the primary issue that has plagued me in this context has been that I have network related issues as Craige and MSharp have previously assessed. My network has been redirecting me out of the web when ever I try to access it through JSL. Once I am able to get through to the data - it will be a great experience to start looking at your work and learning what ever is helpful and incorporating those concepts