Skip to content
bougueilPublic

About

an in-memory logger for spatial and temporal search

Resources

Stars

1 star

Watchers

1 watching

Forks

Latest commit

 

History

19 Commits

Folders and files

Repository files navigation

JOP (Journal of Operations)

CI

Jop logs in memory, spatially and temporally, key value events.

  • A Jop instance can be shared during logging by multiple processes.
  • Several Jop instances can be started and run concurrently.

These events, generated by multiple processes, are then flushed to disk for analysis (e.g. to detect locks).

Warning {: .warning}

Since Jop uses the memory while logging, its usage should be reserved for short time investigations.

Usage

  iex> "myjop"
  ...> |> Jop.init() # init the instance "myjop"
  ...> |> Jop.log("key_1", :any_term_112)
  ...> |> Jop.log("key_2", :any_term_133)
  ...> |> Jop.flush()
log stored in jop_myjop.2020_05_12_21.42.49_dates.gz
log stored in jop_myjop.2020_05_12_21.42.49_keys.gz
#Jop<myjop:uninitialized>

Basic example

# prepare for logging and return a handle
myjop = Jop.init("myjop")

# log with handle event "key_1", :any_term_112
Jop.log myjop, "key_1", :any_term_112
Process.sleep 12

# clear logs
Jop.clear myjop

Jop.log myjop, "key_2", :any_term_113
Process.sleep 12

Jop.log myjop, "key_1", :any_term_112
Process.sleep 12

Jop.log myjop, "key_2", :any_term_113
# flush to disk and erase the log
Jop.flush myjop

log stored in jop_myjop.2020_05_12_21.42.49_dates.gz
log stored in jop_myjop.2020_05_12_21.42.49_keys.gz
#Jop<myjop:uninitialized>

will generate both a temporal (by date) and a spatial (by key) log files:

Examining the temporal log file

# list all operations by date
zcat jop_myjop.2020_05_12_21.42.49_dates.gz

00:00:00_000.482 "key_2": :any_term_113
00:00:00_014.674 "key_1": :any_term_112
00:00:00_028.568 "key_2": :any_term_113

Examining the spatial (by key) log file

# list all operations by key :
zcat jop_myjop.2020_05_12_21.42.49_keys.gz

"key_1": 00:00:00_014.674 :any_term_112
"key_2": 00:00:00_000.482 :any_term_113
"key_2": 00:00:00_028.568 :any_term_113

Real life: concurrent process logging

Processes log in myjop as follow:

# Handle can be saved in process state
myjop = Jop.ref("myjop")

# Log if logging is activated
Jop.log myjop, "key_1", :any_term_112

Console activate / deactivate the logging:

# Activate the logging
# Starts the logging
myjop = Jop.init("myjop")

# Do some queries while the logging is on
inspect myjop # see how many records
Enum.count mylog
Enum.member? mylog, "mykey"
 ...

# Flush the logs on disk, keep on logging
Jop.flush myjop, :nostop

# Clear the logs, keep on logging
Jop.clear myjop  # clear all entries and continue logging

# Flush the logs on disk and deactivate the logging
Jop.flush myjop

# Start logging again
Jop.init("myjop")

Installation

Add jop to your list of dependencies in mix.exs:

def deps do
  [
    {:jop, "~> 0.1"}
  ]
end

Performance

Excerpt from a run of the unit test :

throughput 840812 logs/s. (1 process)
throughput 3851427 logs/s. (concurrency: 8 cores)

About

an in-memory logger for spatial and temporal search

Resources

Stars

1 star

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages