Skip to content

Conversation

@aviatesk
Copy link
Collaborator

There can be cases where a callback key is removed from
user_callbacks_by_key but it still remains in user_callbacks_queue.

The problem may be illustrated with the following MRE.

Say, with the following diff

diff --git a/src/callbacks.jl b/src/callbacks.jl
index bff5256..865bb24 100644
--- a/src/callbacks.jl
+++ b/src/callbacks.jl
@@ -159,6 +159,7 @@ function entr(f::Function, files, modules=nothing; 
all=false, postpone=false, pa
         sleep(pause)
         f()
     end
+    @info "added $(key)"
     try
         while true
             wait(revision_event)
@@ -168,7 +169,7 @@ function entr(f::Function, files, modules=nothing; 
all=false, postpone=false, pa
         isa(err, InterruptException) || rethrow(err)
     finally
         remove_callback(key)
+        @info "removed $(key)"
     end
     nothing
 end
-

and we run the following script:

entr.jl

using Revise

const TARGET_FILE = "watched.jl"

let
    Revise.includet(TARGET_FILE)

    interrupted = false
    while !interrupted
        try
            entr([TARGET_FILE]) do
                # do something ...
            end
            interrupted = true
        catch err
            # handle expected errors, keep running

            # sync errors
            if isa(err, LoadError) ||
               (isa(err, ErrorException) && startswith(err.msg, 
"lowering returned an error")) ||
               isa(err, Revise.ReviseEvalException)
                @warn "handling sync error" err
                continue
            end

            # async errors
            if isa(err, CompositeException)
                errs = err.exceptions
                i = findfirst(e->isa(e, TaskFailedException), errs)
                if i !== nothing
                    tfe = errs[i]::TaskFailedException
                    res = tfe.task.result
                    if isa(res, LoadError) ||
                       (isa(res, ErrorException) && startswith(res.msg, 
"lowering returned an error")) ||
                       isa(res, Revise.ReviseEvalException)
                        @warn "handling async error" res
                        continue
                    end
                end
            end
            rethrow(err)
        end
    end
end

where watched.jl has something like below:

foo(a) = a # first state
# foo(a) = end # second state

Now, boot up REPL and include("entr.jl"), and then modify watch.jl
so that we alternately comment-in/out the lines of first/second state.
It will result in something like below on the current master:

julia> include("entr.jl")
[ Info: added ##245
[ Info: removed ##245
┌ Warning: handling sync error
│   err =
│    LoadError: "unexpected \"end\""in expression starting at 
/Users/aviatesk/julia/packages/Revise/watched.jl:2
└ @ Main ~/julia/packages/Revise/entr.jl:22
[ Info: added ##246
[ Info: removed ##246
ERROR: LoadError: KeyError: key Symbol("##245") not found
Stacktrace:
  [1] getindex(h::Dict{Any, Any}, key::Symbol)
    @ Base ./dict.jl:482
  [2] macro expansion
    @ ~/julia/packages/Revise/src/callbacks.jl:116 [inlined]
  [3] macro expansion
    @ ./task.jl:382 [inlined]
  [4] process_user_callbacks!(keys::Set{Any}; throw::Bool)
    @ Revise ~/julia/packages/Revise/src/callbacks.jl:115
  [5] revise(; throw::Bool)
    @ Revise ~/julia/packages/Revise/src/packagedef.jl:816
  [6] entr(f::var"#3#5", files::Vector{String}, modules::Nothing; 
all::Bool, postpone::Bool, pause::Float64)
    @ Revise ~/julia/packages/Revise/src/callbacks.jl:166
  [7] entr(f::Function, files::Vector{String}, modules::Nothing)
    @ Revise ~/julia/packages/Revise/src/callbacks.jl:156
  [8] top-level scope
    @ ~/julia/packages/Revise/entr.jl:11
  [9] include(fname::String)
    @ Base.MainInclude ./client.jl:451
 [10] top-level scope
    @ REPL[1]:1
in expression starting at 
/Users/aviatesk/julia/packages/Revise/entr.jl:5

, which will be fixed with this PR.


I think we can pack this into a test, but I feel it will be complicated and may be not so robust, so I've not done that yet.

There can be cases where a callback key is removed from
`user_callbacks_by_key` but it still remains in `user_callbacks_queue`.

The problem may be illustrated with the following MRE.

Say, with the following diff
```diff
diff --git a/src/callbacks.jl b/src/callbacks.jl
index bff5256..865bb24 100644
--- a/src/callbacks.jl
+++ b/src/callbacks.jl
@@ -159,6 +159,7 @@ function entr(f::Function, files, modules=nothing; 
all=false, postpone=false, pa
         sleep(pause)
         f()
     end
+    @info "added $(key)"
     try
         while true
             wait(revision_event)
@@ -168,7 +169,7 @@ function entr(f::Function, files, modules=nothing; 
all=false, postpone=false, pa
         isa(err, InterruptException) || rethrow(err)
     finally
         remove_callback(key)
+        @info "removed $(key)"
     end
     nothing
 end
-
```

and we run the following script:
> entr.jl
```julia
using Revise

const TARGET_FILE = "watched.jl"

let
    Revise.includet(TARGET_FILE)

    interrupted = false
    while !interrupted
        try
            entr([TARGET_FILE]) do
                # do something ...
            end
            interrupted = true
        catch err
            # handle expected errors, keep running

            # sync errors
            if isa(err, LoadError) ||
               (isa(err, ErrorException) && startswith(err.msg, 
"lowering returned an error")) ||
               isa(err, Revise.ReviseEvalException)
                @warn "handling sync error" err
                continue
            end

            # async errors
            if isa(err, CompositeException)
                errs = err.exceptions
                i = findfirst(e->isa(e, TaskFailedException), errs)
                if i !== nothing
                    tfe = errs[i]::TaskFailedException
                    res = tfe.task.result
                    if isa(res, LoadError) ||
                       (isa(res, ErrorException) && startswith(res.msg, 
"lowering returned an error")) ||
                       isa(res, Revise.ReviseEvalException)
                        @warn "handling async error" res
                        continue
                    end
                end
            end
            rethrow(err)
        end
    end
end
```

where `watched.jl` has something like below:
```julia
foo(a) = a # first state
# foo(a) = end # second state
```

Now, boot up REPL and `include("entr.jl")`, and then modify `watch.jl`
so that we alternately comment-in/out the lines of first/second state.
It will result in something like below on the current master:
```julia
julia> include("entr.jl")
[ Info: added #timholy#245
[ Info: removed #timholy#245
┌ Warning: handling sync error
│   err =
│    LoadError: "unexpected \"end\""
│    in expression starting at 
/Users/aviatesk/julia/packages/Revise/watched.jl:2
└ @ Main ~/julia/packages/Revise/entr.jl:22
[ Info: added #timholy#246
[ Info: removed #timholy#246
ERROR: LoadError: KeyError: key Symbol("#timholy#245") not found
Stacktrace:
  [1] getindex(h::Dict{Any, Any}, key::Symbol)
    @ Base ./dict.jl:482
  [2] macro expansion
    @ ~/julia/packages/Revise/src/callbacks.jl:116 [inlined]
  [3] macro expansion
    @ ./task.jl:382 [inlined]
  [4] process_user_callbacks!(keys::Set{Any}; throw::Bool)
    @ Revise ~/julia/packages/Revise/src/callbacks.jl:115
  [5] revise(; throw::Bool)
    @ Revise ~/julia/packages/Revise/src/packagedef.jl:816
  [6] entr(f::var"timholy#3#5", files::Vector{String}, modules::Nothing; 
all::Bool, postpone::Bool, pause::Float64)
    @ Revise ~/julia/packages/Revise/src/callbacks.jl:166
  [7] entr(f::Function, files::Vector{String}, modules::Nothing)
    @ Revise ~/julia/packages/Revise/src/callbacks.jl:156
  [8] top-level scope
    @ ~/julia/packages/Revise/entr.jl:11
  [9] include(fname::String)
    @ Base.MainInclude ./client.jl:451
 [10] top-level scope
    @ REPL[1]:1
in expression starting at 
/Users/aviatesk/julia/packages/Revise/entr.jl:5
```
, which will be fixed with this PR.
@timholy
Copy link
Owner

timholy commented Feb 24, 2021

Sorry I failed to notice this when it was submitted. It does sound like this will be hard to test.

@timholy timholy merged commit de80914 into timholy:master Feb 24, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants