I'm working on a zeek package/library where I'd like users to be able to define a use case (using a function like register_usecase(...)) that has specific expiration timers for internal data structures. for example, I'd like it to automatically handle some basic worker-level batching and on expiration push to the proxies to fully aggregate. there's not a one-size-fits-all interval that will work for all use cases, so I need to be able to let users choose their own values. since the use cases will likely be defined in other packages using the register function API, these values will be local parameters, which causes problems.
&create_expire (and I assume the other &*_expires) expect a literal/const/global to behave properly, because its expression is fetched rather than its value. I assume this was by design, as it allows a running zeek instance to modify the value at runtime and the expiration is updated accordingly. unfortunately, it prevents us from dynamically registering a use case at run time.
at the end is some code that kinda emulates the workflow and shows it works for globals, fails for locals (a function parameter), and segfaults for a lambda that captures a parameter (I thought I was clever but sadly am not). generated by Claude but the code references mostly seem to check out.
so what to do here? the best workaround I've found so far is a single global flattened table and some extra bookkeeping to manually handle expiration, but that's kind of gross. it would be nice to be able to force an expiration timer to be a Val instead of an Expr somehow, but that might be a bit onerous on script authors since it requires some internals knowledge. is it possible to know at runtime if a variable is local vs. global and the Expr will no longer evaluate successfully at a later date? if so, we could Do The Right Thing™ and treat it as a value if this is the case, and an Expr otherwise, but I could also see that causing confusion later on. would be interested in hearing more clever workarounds if you have them.
I was hoping to dig into eval_in_isolation in more detail but I accidentally posted the issue already so I'll leave it as-is and update if I find more interesting details. happy to work on this one, but I'd like know what y'all think first.
# &create_expire breaks when its expression references a local: silently
# ignored for a plain local, SIGSEGV for a capturing lambda.
#
# zeek expire-attr-locals-bug.zeek # case 2 crashes
#
# Expected: all three "ok".
# Actual: case 2 segfaults; commenting it out, case 3 prints BUG.
#
# The attribute keeps the expression rather than its value (Val.cc:1805) and
# evaluates it later with a null frame: GetExpireTime (Val.cc:2787) ->
# eval_in_isolation (Expr.cc:5053) -> Eval(nullptr).
#
# - plain local: NameExpr::Eval (Expr.cc:451) throws, swallowed to -1,
# which disables expiration with no error.
# - capturing lambda: LambdaExpr::Eval -> CreateCaptures(nullptr) ->
# Frame::GetElementByID derefs it. KERN_INVALID_ADDRESS at 0x30:
#
# #0 Frame::GetElementByID
# #1 ScriptFunc::CreateCaptures
# #2 LambdaExpr::Eval
# #3 CallExpr::Eval
# #4 eval_in_isolation
# #5 TableVal::GetExpireTime
# #6 TableVal::DoExpire
# #7 TableValTimer::Dispatch
#
# What we want is case 3: a library where each registered use case gets its
# own table with its own expiry, passed in as a parameter. There seems to be
# no way to express that, since the value is only reachable while the frame
# is alive. &expire_func has the same problem -- a factory returning a
# closure for it never runs.
redef exit_only_after_terminate = T;
redef table_expire_interval = 1 sec;
type Pending: table[string] of count;
function flush(t: Pending, key: string): interval
{
return 0 secs;
}
type UseCase: record { pending: Pending &optional; };
global use_cases: table[string] of UseCase;
# 1. From a global: works.
global g_expiry = 2 sec;
global from_global: Pending &create_expire=g_expiry &expire_func=flush;
# 2. Capturing the parameter causes a segfault.
function register_lambda(name: string, expiry: interval)
{
local uc = UseCase();
uc$pending = table()
&create_expire=(function [expiry] (): interval { return expiry; })()
&expire_func=flush;
use_cases[name] = uc;
}
# 3. Straight from the parameter: never expires.
function register_param(name: string, expiry: interval)
{
local uc = UseCase();
uc$pending = table() &create_expire=expiry &expire_func=flush;
use_cases[name] = uc;
}
event report()
{
print fmt("1. global: %s",
|from_global| == 0 ? "ok" : "BUG: never expired");
print fmt("2. capturing lambda: %s",
|use_cases["lambda"]$pending| == 0 ? "ok" : "BUG: never expired");
print fmt("3. parameter: %s",
|use_cases["param"]$pending| == 0 ? "ok" : "BUG: never expired");
terminate();
}
event zeek_init()
{
from_global["a"] = 1;
register_lambda("lambda", 2 sec);
use_cases["lambda"]$pending["b"] = 1;
register_param("param", 2 sec);
use_cases["param"]$pending["c"] = 1;
schedule 6 sec { report() };
}here's a run of this, followed by a run where the lambda-specific bits are commented out:
$ zeek testing/expire-attr-locals-bug.zeek
zsh: segmentation fault zeek testing/expire-attr-locals-bug.zeek
$ zeek testing/expire-attr-locals-bug.zeek
1. global: ok
3. parameter: BUG: never expired
received termination signal
I'm working on a zeek package/library where I'd like users to be able to define a use case (using a function like
register_usecase(...)) that has specific expiration timers for internal data structures. for example, I'd like it to automatically handle some basic worker-level batching and on expiration push to the proxies to fully aggregate. there's not a one-size-fits-all interval that will work for all use cases, so I need to be able to let users choose their own values. since the use cases will likely be defined in other packages using the register function API, these values will be local parameters, which causes problems.&create_expire(and I assume the other&*_expires) expect a literal/const/global to behave properly, because its expression is fetched rather than its value. I assume this was by design, as it allows a running zeek instance to modify the value at runtime and the expiration is updated accordingly. unfortunately, it prevents us from dynamically registering a use case at run time.at the end is some code that kinda emulates the workflow and shows it works for globals, fails for locals (a function parameter), and segfaults for a lambda that captures a parameter (I thought I was clever but sadly am not). generated by Claude but the code references mostly seem to check out.
so what to do here? the best workaround I've found so far is a single global flattened table and some extra bookkeeping to manually handle expiration, but that's kind of gross. it would be nice to be able to force an expiration timer to be a
Valinstead of anExprsomehow, but that might be a bit onerous on script authors since it requires some internals knowledge. is it possible to know at runtime if a variable is local vs. global and theExprwill no longer evaluate successfully at a later date? if so, we could Do The Right Thing™ and treat it as a value if this is the case, and anExprotherwise, but I could also see that causing confusion later on. would be interested in hearing more clever workarounds if you have them.I was hoping to dig into
eval_in_isolationin more detail but I accidentally posted the issue already so I'll leave it as-is and update if I find more interesting details. happy to work on this one, but I'd like know what y'all think first.here's a run of this, followed by a run where the lambda-specific bits are commented out: