How can I listen for HTTP requests and then fire a callback function?

[#1]

I've been reading the sockets examples, but all I can get JMP to do is acknowledge receiving a request, I can't get the payload or message of the request.  Is there a better way than sockets or scheduler to achieve nonblocking communication?

Everything is on windows.

Basically my workflow is:

1. Render an HTML page with JMP,
2. interact with the data using javascript, html, css
3. pass the result to a python Flask server using Javascript XMLHTTP request
4. Continuously ping the Flask server with jmp via scheduler to see if Flask server has any data waiting for me, if it does then python will return the data and JMP will get it as a response, if it doesn't then jmp gets the string 'no update' and I just return.
5. Update JMP table with the data I got from the server.

I'd like to get rid of steps 3 and 4, and PUSH data directly FROM the browser to jmp using that javascript xmlhttp request with a non-blocking listener in jmp that fires a callback when receiving a request and uses the data in that request to update my table. Currently my non-blocking "listener" is a loop spamming the scheduler.

Things I've read so far:
Scripting Index: Sockets

Detailed Guidance on Using JSL to get data from API 

Socket communication to get and log messages from JMP, R, and python scripts 

I also don't understand this: ((blobdata == Char To Blob( "" )) & (Left(r[2], 11) == "WOULDBLOCK:" ))

what is r[2]? The value of these things keeps indexing to different stuff ? In python I'd be expecting to say something like:

r = request
my_dict.update({'last_jmp_data':pd.DataFrame(request.json)})

That works in python to retrieve my data, but when I try to pass back to JMP, the only thing that prints is the headers, I can't access the JSON data I sent. I'm assuming that my data is in some r[#] item?

I'm trying to make something non blocking that can receive data, right now I'm using a hacky solution with the scheduler to create the workflow and it DOES work, but it's clunky.

I use the scheduler to keep this function scheduled, this hits my flask server, receives back data as JSON and updates my main table the update makes JMP drop all the filters so we log that to a string and then re-apply it so the user doesn't lose what they were doing.

What I'd like is to be able to SEND a request from javascript, and pick it up with JMP:
This is my JavaScript function, I don't need any reply I just want it to go to JMP. Previously I was using a different JavaScript workaround to write a JSON file to my downloads and using my scheduler loop to watch the users download folder for a specific JSON file to update the table with.

This application is a single user app that runs locally on the users PC and does not communicate to any external resource, I just needed to make my GUI/certain data interactions in a browser, but I still want to get that data back into JMP for my chimeric JMP/HTML/JS/CSS application.

Javascript code I use to send data to my python server from browser, I'd like to send this straight to JMP and do my update when it arrives.

            var xhr = new XMLHttpRequest();
                xhr.open("POST", 'http://localhost:8081', true);
                xhr.setRequestHeader('Content-Type', 'application/json');
                xhr.send([my_json_data]);

Jmp scheduler loop

//scheduled loop to ping python local server and see if it's time to update:
get_remote_update = Function({},
	s = Schedule(.25,
		//show("remote update fn");
		//check_downloads_folder(); // old routine that was watching download folder
		check_downloads_over_ip();
		get_remote_update(); //loops by scheduling the event again
	);

jmp call to python server receive JSON data and update if needed.

//****************************************************************
//check_downloads_OVER_IP(); 
//****************************************************************
// EXPLORE THIS dt = Open( Char To Blob( request << Send() ), "JSON" );
// see if we can just get the response back into a table without creating it with the weird JMP thing
check_downloads_over_ip = Function({},
	request = New HTTP Request(
		URL("localhost:8088/get_update"),
		Method("Post"),
		Timeout( .5 ),
	);
	data=request << Send;
	if((data != "No Update" > 0) & ( isempty(data) != 1 ),
		obj = JSON:Parse( data );
		dt = JSON:Make Table(obj);
		dt << show window(0);
		
		temp_dtf = CHAR(dtf <<get script);
		temp_dtf_str = substitute(temp_dtf,"Current Data Table()","summary_jmp_table");
		temp_dtf_parsed = parse(temp_dtf_str);
		summary_jmp_table << Update( With(dt), Match Columns( :col_name = :col_name ));
		(filter_hlb <<Child) << Delete Box;
		//filter_hlb
		ldfExpr = Eval Expr( filter_hlb << Append( dtf =temp_dtf_parsed ));
		Eval( ldfExpr );
		Close( dt, No Save );
	); 
);
//******************************************

Socket Scripts I don't understand especially this part:  //Write whatever is received to the log
It prints the headers in the log, but I can't access the JSON data I sent.

Names default to here( 1 );

//Table to hold received data
dtLog = New Table( "Log",
	New Column( "Time", Numeric, "Continuous", Format( "m/d/y h:m:s", 23, 0 ),
		Input Format( "m/d/y h:m:s", 0 ) ),
	New Column( "Client", Character, "Nominal" ),
	New Column( "Message", Character, "Nominal" )
);

//Wait for and get data from remote machine
receivedata = Function( {c, waittime = 1, waitcount = 10},
	
	//Varaible to store data received from remote machine
	blobdata = Char To Blob( "" );
	
	//Make socket is connected
	r = c << GetPeerName();
	
	//While receiving data or waiting to receive data
	i = 0;
	While( 
		( 
			(r[2] == "ok") | 
			((blobdata == Char To Blob( "" )) & (Left(r[2], 11) == "WOULDBLOCK:" ))
		) & 
		(i < waitcount),
		
		i++;
		
		//Get data
		r = c << recv( 100000 );
		//If no data is received yet, wait 
		If(Left(r[2], 11) == "WOULDBLOCK:", Wait( waittime ) );
		
		//Write whatever is received to the log
		show("writing to log now");
		Write( Blob to Char( r[3] ) );
		show("Done writing to log now");
		//join data to whatever is already received
		blobdata = blobdata || r[3];
	);
	
	return( blobdata );
);

//Get log info from remote machine, store it in a data table
processlogconnection = function( {c},

	//New row in log table
	dtLog << Add Rows( 1 );
	rowout = N Rows( dtLog );
	Column( dtLog, "Time" )[rowout] = Today();
	
	//Get computer name of remote machine
	r = c << GetPeerName();
	Column( dtLog, "Client" )[rowout] = r[3];
	
	//Get message from the remote machine
	Column( dtLog, "Message" )[rowout] = Blob to Char( receivedata( c ) );
	
	//Close the connection
	c << Close;
);

//Function to accept conections from remote machine, calls another
//function to handle communication with that machine
callbackfn = function( {x}, 
	//Call another function with the connection returned from the Accept message
	If( x[2] == "ok", processlogconnection( x[4] ) );
);

//Create a socket
con = Socket();

//Use a specific port on this computer
con << Bind( "localhost", "8081" );

//Tell this socket to be ready for other sockets to connect to it
Write( "\!nlistening on ", (con << getSockName)[3]);
con << Listen();

//Verify that the connection is active
con << getsockname;

//When something connects to this socket, call a function to handle
//the connection.  Accept connections for the next 5 minutes.
con << Accept(callbackfn, 30000);

//Now messages can be sent to this connection.

//When all done, close the connection
//con << close();

[#2]

You could use JMP's HTTP Request to ping the Flask server. That would remove a level of complexity of what you are wanting to do.

You can query the server with HTTP Requests (I assume a GET operation) and read the response directly with the return (like a JSON payload) and then when it's ready, use HTTP Request again (probably as a GET) and process the data from the server. Of course, if you are using sockets, rather than HTTP/HTTPS, then you'll want to stick with your socket approach.

[#3]

Also, JMP's sockets do not support https, only http. They are fairly thin wrappers around low-level socket code and know nothing about http protocol: you have to build it all yourself. There is a complete example in the scripting index:

The "listen" example is self-contained.

It is quite difficult to build a web server in JSL. If you are already using Python from JMP, take a look at Project file for Jelly Video especially the NumberServer. I needed to run multiple copies of JMP to produce the images in the video and I needed a way to hand them the work they needed to do. The number server did that, and provided the instrumentation for a JSL dashboard so I could tell when something needed attention. (That python server was single threaded, I think, which turned out to be a good thing for serving up unique numbers for simultaneous requests. At least I never noticed a duplicate...)

[#4]

On re-reading, I agree with Bryan: you should be using http request to talk to the flask server.

Apparently flask and websockets are good friends too: https://blog.miguelgrinberg.com/post/add-a-websocket-route-to-your-flask-2-x-application

See WebSocket  for an example; a websocket is a bi-directional connection that just stays open...forever...receiving driblets as they come in. It makes a browser accept pushes from a server without polling. That example code uses the Python websocket and JMP's RunProgram to avoid blocking issues.

[#5]

This sounds promising javascript can make websockets natively too.

Can run program get data into JMP in a non blocking manner? I thought while you have the external program running your JMP gui would be blocked?

From that page:

Note: If you use Run Program() to launch a GUI program, JMP is inoperable until you close the program

Edit: I actually did already download this and try to understand it once! I'll give it another shot.

[#6]

Isn't that what I'm currently doing though?

I'm only interested in sockets because it seems to be a way to create a non blocking listener versus my current mechanism of streaming HTTP GET requests from JMP to Flask every half a second to see if the web browser content changed from user interaction. I'm hoping to PUSH an event from flask or javascript to JMP without JMP asking for anything by itself.

check_downloads_over_ip() function //http get

AND the get_remote_update(). // scheduler loop

get_remote_update() keeps the check_downloads function constantly scheduled in the scheduler as a mechanism to make it "real time"

The issue is the scheduler isn't a very elegant way to generate non blocking behavior. 

I'd like JMP to sit listening (in a non blocking manner) and then I can push an http request from javascript directly in the browser to JMP and eliminate the flask server entirely.

The crux is JMP has the tools to PULL data from a server and then work with the response, but can I PUSH an event from the browser or server into JMP without JMp first requesting anything.

I'll strip all the IP out of my app and post a video to make it more clear what I'm going for.

[#7 · Accepted solution]

RunProgram can run N external programs, concurrently and asynchronously. If JMP is otherwise idle, the N RunPrograms can run their ReadFunction() scripts as they receive data on stdin, and they can write data via stdout. If Python is the program that RunProgram is executing, Python's stdin is RunProgram's stdout, and ...stdout...stdin. (N probably can't be bigger than a few 10s.)

RunProgram can run things like excel but that will block. RunProgram is much better for command line programs with stdin and stdout. You might use cmd.exe, or you might use plink to open a shell on a remote Linux machine. Or you might run a C or Python or other language program of your own.

RunProgram has two modes of operation: if it has ReadFunction("text") then it runs in blocking mode, collecting all the stdout and returning a string. Otherwise it runs in non-blocking mode and returns a RunProgram object which you must preserve in a variable (otherwise it will be deleted before the process finishes.)

There are only a few commands you can send to a RunProgram object; you can ask about the connection and send and receive data.

I do not usually use the scheduler for this sort of thing because it wants to have a window open, but see News Feed . Same article shows how to use the CMD start command to run the msedge browser without the blocking behavior.

I have written servers in JSL (same article), but I think it is much better to use a python or python+flask solution. If you use flask, try this: Let RunProgram launch the Python+flask code, and give RunProgram some logic in its readfunction and writefunction to communicate with the python code (stdin/stdout). In the python code, add a thread to talk to JMP. The flask server has its own thread to service http requests. If the server needs to push something to jmp, it needs to signal an event to the JMP thread that will print something to JMP. You might want one more Python thread that is a reader thread, listening for JMP.

[#8]

Thanks! I'll try this approach.