Limitation Of HexToNumber()

ravi_sasjmp — Wed, 23 Aug 2017 22:05:39 GMT

I'm using JMP10 and notice for Hex To Number( hextext), hextext is limited to 31-bits. Is there a way around this?

Hex To Number("7FFFFFFF") = 2147483647, Hex To Number("80000000") = -2147483648.

I need to multiply two hex strings (each can be upto 4-bytes) and in the resulting number, depending on state of MSB bit, extract Bits[29:14] or Bits [27:12]. Can this be done in JSL?

Re: Limitation Of HexToNumber()

Craige_Hales — Thu, 24 Aug 2017 13:18:04 GMT

JMP uses 64 bit floating point to represent the JSL numbers; the number of fraction bits available is only 52 or 53. If you multiply two 32 bit integers, you might need 64 bits to represent the answer. If the answer is greater than 52 or 53 bits, you'll lose precision on the low order bits...maybe as many as 11 or 12 bits can be lost.

Here's some JSL that might help.

// crude unsigned hex multiply, poorly tested, no idea what negative numbers are
// do your own testing to make sure this meets your needs
// expects upper case hex ABCDEF, not abcdef
hextonum = [
"0"=>0,"1"=>1,"2"=>2,"3"=>3,
"4"=>4,"5"=>5,"6"=>6,"7"=>7,
"8"=>8,"9"=>9,"A"=>10,"B"=>11,
"C"=>12,"D"=>13,"E"=>14,"F"=>15	
];
numtohex = Associative Array( hextonum << getvalues, hextonum << getkeys );

hexmultiply = Function( {a, b}, // "123AE" and "EF012" for example
    {// locals
    ia,  na=length(a),amat = J( na, 1, 0 ), ib,nb=length(b), bmat = J( nb, 1, 0 ),ir,nr=na+nb,result = J( nr+1, 1, 0 ),r="", carry,remainder},
    a = Reverse( a );
    For( ia = 1, ia <= na, ia++,
        amat[ia] = hextonum[Substr( a, ia, 1 )]
    );
    b = Reverse( b );
    For( ib = 1, ib <= nb, ib++,
        bmat[ib] = hextonum[Substr( b, ib, 1 )]
    );
    for(ia=1,ia<=na,ia+=1,
    	for(ib=1,ib<=nb,ib+=1, // this inner loop might speed up by replacing it with matrix math
            result[ia+ib-1] += amat[ia]*bmat[ib];
    	)
    );
    for(ir=1,ir<=nr,ir++, // do the carry. doing it last is fast, but places some sort of a limit on the total length
        remainder = mod(result[ir],16); 
        carry = floor(result[ir]/16); 
        result[ir] = remainder;
        result[ir+1] += carry;
    	r =  numtohex[result[ir]] || r; // build hex result
    );
    r // return value
);

// example hexmultiply
show( hextonumber(hexmultiply("AE2","3B75")));
show( hextonumber("AE2")*hextonumber("3B75"));

// test code...
ntests=0;
start = tickseconds();
For( num1 = 0, num1 < 2e9, num1 = Floor( num1 * 1.07 + 1 ),
    For( num2 = 0, num2 < 2e9, num2 = Floor( num2 * 1.05 + 1 ),
        If( num1 * num2 > 2 ^ 53,
            Continue(); // because double multiply gets wrong answer
        );
        ntests+=1;
        t1 = hexmultiply( Hex( num1, "integer" ), Hex( num2, "integer" ) );
        If( Hex To Number( Substr( t1, 2 ) ) != num1 * num2,
            Show( num1, num2, t1, Hex To Number( Substr( t1, 2 ) ), num1 * num2 )
        );
    )
);
stop=tickseconds();
show(ntests,stop-start,ntests/(stop-start));

// long test...no promises here...this is not a meaningful test...
show( hexmultiply("00123456789ABCDEF0","00200000000000000000000000000000000000"));

Re: Limitation Of HexToNumber()

Craige_Hales — Tue, 29 Aug 2017 15:02:46 GMT

JMP 10 has some limitations; it looks like the strings of hex digits in JMP 10 need to be an even number of hex characters. The function above seems to work in JMP 10 (do your own testing!), but the test harness is using features from later releases.

topic Re: Limitation Of HexToNumber() in Discussions

Limitation Of HexToNumber()

Re: Limitation Of HexToNumber()

Re: Limitation Of HexToNumber()