It looks like that exit code is normally from a stack overflow. I concur with @Craige_Hales that examining dt is a good place to start. You might follow that by sending a smaller version of dt to R, and by running each line of the R script separately, or gradually add lines to your script, until you find the exact line that causes the error.