Detailed Guidance on Using JSL to get data from API

[#1]

All, 

    I would like to get the community's guidance on how to connect and get data from an API into JMP using JSL . I have gone through a couple of posts in the community on this topic : 
 

• https://community.jmp.com/t5/Discussions/JMP-SAS-Programmers-Can-JMP-Access-Twitter-s-API/m-p/11723#...
• https://community.jmp.com/t5/Discussions/How-can-I-import-twitter-data-into-JMP-for-analyzing/m-p/20...

   Additionally , I have reached out to  @Craige_Hales for guidance on the same topic. My question to Craige were : 

   I will be upfront about my ignorance and limited knowledge on the subject so as to not waste your precious time . I know pretty little about API's and have not worked with API's much previously. 

       A brief synopsis of what I understand about the topic within JMP is as follows : 
1. JMP can stream information from API's , with the constraint that the API should be a http and not https - In my case it is https 

2. JMP can connect to R , if streaming data from the API into JMP is not possible , I am hoping to stream into R and pass the data into JMP. However , I am not sure how much of a delay it would introduce into my application . 

3. Socket is an object that allows streaming of information from a connection be it to an API or another computer in the network . Sockets allow to stream and submit information 

       Is this correct? If yes , I will try to build a test script that streams information from my API . If no, can you kindly correct me and guide me to resources which would help me with this process 

       I am dealing with a REST API which can be exposed on https . Authentication Mechanism is based on OAuth2 Guidelines. 
        In the event that , I can only achieve this outside of JMP , I would prefer to work with R / Python 

[#2]

@Craige_Hales, 
           Kindly provided the following guidance :

API is a specification for how to use a service. There isn't much of a standard, even amongst API's that claim to be rest-based. You'll be doing a lot of reading, most likely.

I think you mentioned earlier that you had to authenticate to the service you want to use. The Twitter authentication is quite a complicated process using OAuth2, which I'm not qualified to talk about...I stumbled through it and got it working and it represents a lot of the JSL in the 7-things blog. 

Or you might have a much simpler authentication using a password.

when you connect to a website from a browser, a socket streams the html from the site into the browser. Usually the html is a finite length, but if there is a program on the server generating the html, it is quite possible for it to stream forever. 

JMP's sockets can do that with http. Curl can do that with https (or http). JMP 14 will have curl support builtin.

A rest-based service is likely to hand you json data. (I can't keep up with what goes into each JMP release; I think 13 has some json support and 14 may have more. there is also some jsl Xan and I played with a while back.) json is nested key-value data, not particularly human-friendly, but not too bad either.

part of the magic of streaming data is realizing that curl, or a socket, is going to give you most of the data, up to right now. and maybe in small chunks. so you might have to assemble it into complete chunks and figure out where the chunk boundaries are. and hang on to the last few bytes of the next chunk and wait for the rest to be delivered by the socket, or curl. If you look at the connect example, you'll get a sense of it...a loop that polls the socket and bulds a longer and longer result as the data arrives. Although that example knows the data should be finite and waits until it is all available before proceeding. The twitter example (somewhere in it, its been a long time) uses RunProgram to get a stream from curl, forever. the runprogram readfunction also concatenates data together but occasionally processes the data when it sees it has a complete chunk. With the socket there is an explicit loop. In the twitter runprogram logic the loop is implicit...the runfunction callback is driven by the curl program receiving more data from twitter.

[#3]

@Jeff_Perkinson , @Wendy_Murphrey  , @ms , @pmroz
           Any thoughts ? 

[#4]

Sorry I haven't done anything like this.  Good luck!

[#5 · Accepted solution]

The best guidance I can offer is to wait for just a few months until JMP 14 is released. I think you'll be happy that you did as this will be much easier then.

[#6 · Accepted solution]

As JMP 14 is available for almost 9 months I think I add some reference where this is somehow shown: 

Getting-Started-With-REST-in-JMP/ta-p/64104

and examples-of-http-requests.shtml

HttpRequest is cool :)

Best Martin

[#7]

My go-to is the httr package in R.  It has very mature get/request and parsing capabilities, including json.  The documentation is good and there are a lot of examples out there.  Once installed this is all it takes to get data into JMP from the first API I could find that didn't need login info:

Names default to here( 1 );

R Init( );
R Submit("
library(httr)
httpResponse <- GET(\!"https://swapi.co/api/people/1/?format=json\!")
parsedContent = as.data.frame(content(httpResponse, \!"parsed\!"))
");
dtHidden = R Get( parsedContent );
R Term();

dt = dtHidden << Subset( Link to Original Data Table(0) );