Returning an expression from a function differs when namespace is used.

[bculver]

Hello everyone,

I ran into some strange behavior when returning an expression from a function, and could not find any discussions about this already. I am in the habit of using namespaces for pretty much everything. I found that when a function returns an expression, the expression is not evaluated for an unscoped function, but is evaluated if a namespace is used. Does anyone know what's going on here?

This script should show that the variable x is not incremented when the first function is called, but is incremented on the second function call. I am using JMP v17.0.0, so this has possibly been fixed, but I am not able to update right now.

NewNamespace("varNamespace");
varNamespace:x = 0;

myFunc = Function({},
	{DEFAULTLOCAL},
	returnVal = Expr(varNamespace:x++);
	NameExpr(returnVal);
);

myFunc();
Show(varNamespace:x);

NewNamespace("functionTest");
functionTest:myFunc = Function({},
	{DEFAULTLOCAL},
	returnVal = Expr(varNamespace:x++);
	NameExpr(returnVal);
);

functionTest:myFunc();
show(varNamespace:x);

[bculver]

Hello everyone,

I ran into some strange behavior when returning an expression from a function, and could not find any discussions about this already. I am in the habit of using namespaces for pretty much everything. I found that when a function returns an expression, the expression is not evaluated for an unscoped function, but is evaluated if a namespace is used. Does anyone know what's going on here?

This script should show that the variable x is not incremented when the first function is called, but is incremented on the second function call. I am using JMP v17.0.0, so this has possibly been fixed, but I am not able to update right now.

NewNamespace("varNamespace");
varNamespace:x = 0;

myFunc = Function({},
	{DEFAULTLOCAL},
	returnVal = Expr(varNamespace:x++);
	NameExpr(returnVal);
);

myFunc();
Show(varNamespace:x);

NewNamespace("functionTest");
functionTest:myFunc = Function({},
	{DEFAULTLOCAL},
	returnVal = Expr(varNamespace:x++);
	NameExpr(returnVal);
);

functionTest:myFunc();
show(varNamespace:x);

[Craige_Hales · Accepted solution]

Pretty sure you've discovered a bug. Here's a even more simplified demo of two ways to use functions in namespaces:

funcNS = New Namespace(
	"funcNS",
	f1 = Function( {}, 
		expr(1+2)
	)
);

funcNS:f2 = Function( {}, 
	expr(1+2)
);
	
Show( Type( funcNS:f1() ), Type( funcNS:f2() ) );

Type(funcNS:f1()) = "Expression";
Type(funcNS:f2()) = "Number";

f1 is returning the correct answer; f2 is incorrectly evaluating the answer. I prefer the f2 way (except for this bug) because it is faster:

funcNS = New Namespace(
	"funcNS",
	f1 = Function( {}, 
		1
	)
);

funcNS:f2 = Function( {}, 
	1
);
	
nf1 = 0;
For( endtime = HP Time() + 1e6, endtime > HP Time(), nf1++,
	funcNS:f1()
);
Show( nf1 ); // 180K calls
nf2 = 0;
For( endtime = HP Time() + 1e6, endtime > HP Time(), nf2++,
	funcNS:f2()
);
Show( nf2 ); // 260K calls

nf1 = 182239;
nf2 = 258929;

The two issues are probably related. Items specified inside the namespace function call are not constructed until they are used (they are kept as expressions.) That means f1 is not compiled when the namespace is created, but f1 is compiled each time it is used, thus slower.  So there is probably some specialized C++ code to recompile f1 and then evaluate it. My guess is that same code is triggered for f2, but doesn't recompile because f2 is already compiled, and then incorrectly evaluates the result of f2, which most of the time is a primitive value and no one notices. The expr() value that is returned does evaluate into a simpler value, causing you to notice.

Spoiler

code I played with to check my understanding

c=1; d="hi"; e=["key1"=>"val1"];
a = New Namespace( "a",
	w = {c, d, e},
	x = Eval List( {c, d, e} ),
	 // the RHS expression is assigned to x
	y = 1 // this does what you expect because 1 == expr(1)
);
Show( a );
/* The namespace is holding the expressions...
a = New Namespace("a", {
w = {c, d, e},
x = Eval List( {c, d, e} ),
y = 1
});
*/
Show( a:x ); // a:x = {1, "hi", ["key1" => "val1"]};
/////////// change some globals
c=2; d="there"; e["key2"]="newval";
Show( a:x ); // a:x = {2, "there", ["key1" => "val1", "key2" => "newval"]};

a:z = Eval List( {c, d, e} );
show(a);
/*
a = New Namespace("a", {
w = {c, d, e},
x = Eval List( {c, d, e} ),
y = 1,
z = {2, "there", ["key1" => "val1", "key2" => "newval"]}
});
*/

@julian  - pretty sure this is a bug. I don't think there will be many namespace users depending on the wrong behavior.

[bculver]

I have updated to 17.2.0, and can confirm this is still issue still occurs.

Here is a simplified example that should hopefully be easier to read

x = 0;
myFunc = Function({}, {DEFAULTLOCAL},
	Expr(::x++);
);

myFunc(); // Returns unevaluated expression: Expr(::x++)
Show(::x); // ::x=0;

NewNamespace("funcNS");
funcNS:myFunc = Function({},{DEFAULTLOCAL},
	Expr(::x++);
);

funcNS:myFunc(); // Returns evaluated expression;
show(::x); // ::x=1;

[Craige_Hales · Accepted solution]

Pretty sure you've discovered a bug. Here's a even more simplified demo of two ways to use functions in namespaces:

funcNS = New Namespace(
	"funcNS",
	f1 = Function( {}, 
		expr(1+2)
	)
);

funcNS:f2 = Function( {}, 
	expr(1+2)
);
	
Show( Type( funcNS:f1() ), Type( funcNS:f2() ) );

Type(funcNS:f1()) = "Expression";
Type(funcNS:f2()) = "Number";

f1 is returning the correct answer; f2 is incorrectly evaluating the answer. I prefer the f2 way (except for this bug) because it is faster:

funcNS = New Namespace(
	"funcNS",
	f1 = Function( {}, 
		1
	)
);

funcNS:f2 = Function( {}, 
	1
);
	
nf1 = 0;
For( endtime = HP Time() + 1e6, endtime > HP Time(), nf1++,
	funcNS:f1()
);
Show( nf1 ); // 180K calls
nf2 = 0;
For( endtime = HP Time() + 1e6, endtime > HP Time(), nf2++,
	funcNS:f2()
);
Show( nf2 ); // 260K calls

nf1 = 182239;
nf2 = 258929;

The two issues are probably related. Items specified inside the namespace function call are not constructed until they are used (they are kept as expressions.) That means f1 is not compiled when the namespace is created, but f1 is compiled each time it is used, thus slower.  So there is probably some specialized C++ code to recompile f1 and then evaluate it. My guess is that same code is triggered for f2, but doesn't recompile because f2 is already compiled, and then incorrectly evaluates the result of f2, which most of the time is a primitive value and no one notices. The expr() value that is returned does evaluate into a simpler value, causing you to notice.

Spoiler

code I played with to check my understanding

c=1; d="hi"; e=["key1"=>"val1"];
a = New Namespace( "a",
	w = {c, d, e},
	x = Eval List( {c, d, e} ),
	 // the RHS expression is assigned to x
	y = 1 // this does what you expect because 1 == expr(1)
);
Show( a );
/* The namespace is holding the expressions...
a = New Namespace("a", {
w = {c, d, e},
x = Eval List( {c, d, e} ),
y = 1
});
*/
Show( a:x ); // a:x = {1, "hi", ["key1" => "val1"]};
/////////// change some globals
c=2; d="there"; e["key2"]="newval";
Show( a:x ); // a:x = {2, "there", ["key1" => "val1", "key2" => "newval"]};

a:z = Eval List( {c, d, e} );
show(a);
/*
a = New Namespace("a", {
w = {c, d, e},
x = Eval List( {c, d, e} ),
y = 1,
z = {2, "there", ["key1" => "val1", "key2" => "newval"]}
});
*/

@julian  - pretty sure this is a bug. I don't think there will be many namespace users depending on the wrong behavior.

[KasperSchlosser]

Has there been any update to this?
As far as i can tell this behavior still exists in jmp 19.

It definitely seems like the return value gets evaluated twice in the ns:f2 definition:

ns = New Namespace("fns",
	f1 = Function({},
		x = Expr(1+3);
		return(Name Expr(x));
	);
);

f1 = Function({},
	x = Expr(1+3);
	return(Name Expr(x));
);

ns:f2 = Function({},
	x = Expr(1+3);
	return(Name Expr(x));
);

ns:f3 = Function({},
	x = Expr(1+3);
	return(Name Expr(Name Expr(x)));
);

Show(f1(), ns:f1(), ns:f2(), ns:f3());

f1() = 1 + 3;
ns:f1() = 1 + 3;
ns:f2() = 4;
ns:f3() = 1 + 3;

f1() = 1 + 3;
ns:f1() = 1 + 3;
ns:f2() = 4;
ns:f3() = x;

If anyone else encounters this problem the quick workaround is as the example suggest; Wrap the return value in "Name Expr" until the desired result 

This does not work. I had a x defined in the global/here scope from previous runs. There seem to be no way to return an expression from a function defined like ns:f2

This definitely seem like a bug, i dont think anyone would expect a ns:f2 to return a different value compared to ns:f1 or f1; the function code is exactly the same.

[Craige_Hales]

I'll make a report through the support team. 

TS-00235798

[KasperSchlosser]

I also made a report: TS-00235515

The response from support was that this was not considered a bug, even if the behavior was unexpected.

Root cause was as you described; Basically the ns:f2 function gets evaluated twice.

This seem to also affect everything defined outside the initial namespace creation, As can be seen in the updated example below.

The proposed solution is to wrap everything in an expression - ns:f2 = Expr(Function(....))

Names Default to Here(1);
delete symbols();

ns = New Namespace("fns",
	{
	f1 = Function({},{Default Local},
		x = Expr(1+3);
		return(Name Expr(x));
	),
	e1 = Name Expr(expr(1 + a))
	}
);

f1 = Function({},{Default Local},
	x = Expr(1+3);
	return(Name Expr(x));
);

ns:f2 = Function({},{Default Local},
	x = Expr(1+3);
	return(Name Expr(x));
);

ns:f3 = Expr(Function({},{Default Local},
	x = Expr(1+3);
	return(Name Expr(x));
));

ns:e2 = Name Expr(expr(1 + a));
ns:e3 = Expr(Name Expr(expr(1 + a)));

show(ns:e1);
show(try(ns:e2, exception_msg));
show(ns:e3);

Show(f1(), ns:f1(), ns:f2(), ns:f3());

ns:e1 = 1 + a;
Try(ns:e2, exception_msg) = {"Name Unresolved: a"(1, 2, "a", a /*###*/)};
ns:e3 = 1 + a;
f1() = 1 + 3;
ns:f1() = 1 + 3;
ns:f2() = 4;
ns:f3() = 1 + 3;

[Craige_Hales]

That workaround is...unfortunate. Fortunately, most users are not trying to return expressions and don't need it.

[Craige_Hales]

Also, most of the unfortunate aspect can be alleviated by returning 

expr(expr(1+1))

rather than making the entire function be

expr(function(...

funcNS = New Namespace(
	"funcNS",
	f1 = Function( {}, 
		expr(1+2)
	)
);

funcNS:f2 = Function( {}, 
	expr(expr(1+2)) // <<<< extra expr wrapper
);
	
Show( Type( funcNS:f1() ), Type( funcNS:f2() ) ); // expr, expr

[KasperSchlosser]

That solution is indeed more elegant, however it seem to only work when you can directly write the expression to return.

If the expression is stored in a variable and we wrap that in an Expr() the value returned is an expression with the variable name

In this case it seems the only solution seems to be to wrap the return value in a Name Expr() and the entire function in an expr

funcNS:f5 = Expr(Function( {}, {Default Local},
	x = Expr(1+2);
	Name Expr(x)
));

Show( funcNs:f5() ); // 1+2

One big danger with this us that the second evaluation seem to happen outside the local scope of the function.

A global variable might shadow the return value, which would not give an error.

Names Default to Here(1);
Delete Symbols(); // Only use if you dont have variables you need

funcNS = New Namespace("funcNS");

funcNS:f1 = Function( {}, {Default Local},
	x = Expr(1+2);
	Expr(x);
);

funcNS:f2 = Function( {}, {Default Local},
	x = Expr(1+2);
	Expr(Expr(x));
);

funcNS:f3 = Expr(Function( {}, {Default Local},
	x = Expr(1+2);
	x;
));

funcNS:f4 = Expr(Function( {}, {Default Local},
	x = Expr(1+2);
	Expr(x);
));

funcNS:f5 = Expr(Function( {}, {Default Local},
	x = Expr(1+2);
	Name Expr(x)
));
	
Show( Try(funcNs:f1(), exception_msg) ); // Name Unresolved
Show(funcNs:f2()); // X
Show(funcNs:f3()); // 3
Show(funcNs:f4()); // x
Show(funcNs:f5()); // 1+2

// Second Evaluation happens outside Function scope
x = Expr(Expr(1+10));
Show(funcNs:f1()); // 1+10

[hogi]

wow!

added to Tiny Traps in Jmp and JSL