Subscribe Bookmark RSS Feed

Interpolating Large data sets


Super User


Mar 17, 2015

Question on best practice.  Would it be better (faster/more efficient, feel free to define better as you wish) to interpolate using a column formula or matrices?

For example:

I have a table with 2 billion rows with of non integer xs (big_x_vector), with corresponding ys (big_y_vector)

I want to interpolate out 14 through 14,000,000 in .5 unit chunks

range = 14::14000000`;

range = (sort ascending(range|/(range+.5)));

range = range[1::(nrows(range)-1)];//create a .5 interval vector

x=1; //just so I didn't write to log a bajillion characters

//is it better to

interpolated_values = [];

for(i=1, i<=nrows(range), i++, //for through them

       interpolated_values |/ = interpolate(range[i], big_x_vector, big_y_vector)



New Table("Interpolated", //Formula


       New Column("Range", set values(range)),

       New Column("Value", Formula(interpolate(:Range, big_x_vector, big_y_vector)))