Make a Video in JMP with FFmpeg

[#1]

I've been working on a video project for a while, and finally ran into a wall with VLC. VLC is the best video player. It can also be used as a video generation tool, but has some quirks related to FPS of the output files. I decided to re-investigate FFmpeg as a replacement. I'd used FFmpeg a long time ago, without really reading all the docs and remembered it being complicated. Well, it is. I spent today reading the docs. It is a command line tool with tons of doc (see link). I put this together as a proof-of-concept. There's a lot more it can do.

The JSL below consists of a user defined function that generates images for a video sequence and a RunProgram call that runs FFmpeg to make a video. FFmpeg is told to get images via stdin, and RunProgram provides them by calling the user defined function. I have only run it on Windows, but I believe it will only take some minor changes to make it run on Mac as well.

To run it, you'll need to download FFmpeg. Their website points to another site you can get it from. I'm not sure why they don't have an official windows version. Once installed, this script will run in about 30 seconds to make an 8-second video, then open the video in your default player. It will look better in VLC, or you can upload it to youtube and see it like this:

Read the legal stuff carefully if you are going to do commercial work with it.

// generate images for ffmpeg stdin on the fly to make a video
// using https://ffmpeg.org/download.html#build-windows
//
// You could make a somewhat simpler version, maybe, by writing
// the files to disk and telling ffmpeg to re-read them. I wanted
// to use the stdin pipe, and didn't try a file. I hope this pipe is
// at least as fast as writing and reading the files.
//
// you can't make a 2nd, parallel stream of audio data using RunProgram,
// so you'll need a disk file for audio. Or http as shown...

outputmp4 = ConvertFilePath("$desktop/VideoMadeByJMPandFFMPEG.mp4", "windows");

nrows = 1080 / 2;// keep the image buffer even in both dimensions
ncols = 1920 / 2;// and maybe a multiple of 16 as well

Open Log( 1 ); // where the action is (may need this window open for the text box to work)

imagex = J( nrows, ncols, RGB Color( 1, 1, 1 ) ); // buffer of JSL color values, updated by getNextImage...

// credit the CC music...write this once, into the buffer
musicCredit = Text Box(
"\!"Pump Sting\!" Kevin MacLeod (incompetech.com)
Licensed under Creative Commons: By Attribution 4.0 License
http://creativecommons.org/licenses/by/4.0/
"||char(asdate(today())),
	<<setwrap( ncols ),
	<<backgroundcolor( RGB Color( 1, 1, 1 ) ),
	<<setfontsize( 16 ),
	<<fontcolor( "black" )
);
// grab the pixels from the text box...
creditPixels = (musicCredit << getpicture) << getpixels;
// and stuff them into the buffer
imagex[(N Rows( imagex ) - N Rows( creditPixels ) + 1) :: (N Rows( imagex )), (1 + 20) :: (20 + N Cols( creditPixels ))] = creditPixels;

nn = 0; // what ever state information is needed. here, just a counter.

getNextImage = Function( {},
	If( nn <= 255,
		Try(
			imagex[// make another little square block on each call
				Interpolate( nn, 0, 1, 256, nrows ) :: Interpolate( nn + 1, 0, 1, 256, nrows ), //
				Interpolate( nn, 0, 1, 256, ncols ) :: Interpolate( nn + 1, 0, 1, 256, ncols )//
			] = HLS Color( nn / 256, .5, 1 );// color of block changes
			nn += 1; // update counter/state info
			Matrix To Blob( -imagex, "int", 4, "big" ); // return the argb data (negative JSL color)
		,
			Show( exception_msg ); // oops. what went wrong?
			Empty(); // shut down after error
		)
	,
		Empty() // normal shutdown after all images delivered
	)
);

start = Tick Seconds();
// rp will hold the RunProgram object; below there is a loop to wait for it to complete
rp = Run Program(// downloaded from https://ffmpeg.zeranoe.com/builds/win64/static/ffmpeg-20200626-7447045-win64-static.zip
	executable( "C:\Users\v1\Desktop\ffmpeg-20200626-7447045-win64-static\bin\ffmpeg.exe" ),
	options(
		{// idea from https://stackoverflow.com/questions/34167691/pipe-opencv-images-to-ffmpeg-using-python
		"-y", // overwrite output
		"-f", "rawvideo", // tell ffmpeg the stdin data coming in is
		"-vcodec", "rawvideo", // "raw" pixels, uncompressed, no meta data
		"-s", Eval Insert( "^ncols^X^nrows^" ), // raw video needs this meta data to know the size of the image
		"-pix_fmt", "argb", // the JSL color values are easily converted to argb with MatrixToBlob
		"-r", "20", // input FPS
		"-i", "-", // this is the pipe input. everything before this is describing the input data
		//"-an", // disable audio, or supply a sound file as another input
		"-i", "https://incompetech.com/music/royalty-free/mp3-royaltyfree/Pump%20Sting.mp3", //
		"-vcodec", "mpeg4", // encoding for the output
		"-b:v", "5000k", // bits per second for the output
		outputmp4 // the name of the output
		}
	),
	WriteFunction( // the WriteFunction is called when FFMPEG wants JMP to write some data to FFMPEG's stdin
		Function( {this},
			data = getNextImage(); // getNextImage provides the binary data for an image, or empty() when done
			If( Is Empty( data ), //
				this << writeEOF;// tell FFMPEG there is no more data
			, //else
				this << Write( data );// send the data
			);//
		)
	),
	ReadFunction( // the ReadFunction is called when FFMPEG wants JMP to read some of FFMPEG's stdout status messages
		Function( {this},
			While( rp << canread,
				Write( this << read ); // JMP will copy them to the JMP log window
				Wait( .01 ); // get as much as we can so it doesn't pile up or leave some behind
			)
		)
	)
);

// If you had something else to do, you could do it before this wait loop. FFMPEG closes the
// read pipe after sending its last message, the message is processed by the ReadFunction above.
While( !(rp << isreadeof), Wait( .01 ) );// wait(.01) responds in 1/100 second without burning much CPU.

stop = Tick Seconds();
Write( "\!ndone, used ", Char( stop - start, 7, 1 ), " seconds" ); // without this, it is hard to be sure...

open(outputmp4);