Local Test are OK

I see travis failing. I do not know what is happening.

In Travis details - this is one of the errors and it does not make sense 😕

  Uncaught TypeError: allowedExtensions.includes is not a function
    at Function.check (lib/files/datafile.js:18:45)

Found something related to a harmony flag nodejs/node#5715

. . .

After one failure with lodash.includes(), ended up changing .includes() to .indexOf() > -1 😞

@zeke
This commit includes the check part in each file type like this:

sample of lib/files/image.js

. . .
  // calls back with a boolean indicating whether this class should process the given file.
  static check (filename, callback) {
    const extension = path.extname(filename).toLowerCase()
    const allowedExtensions = ['.jpeg', '.jpg', '.svg', '.png', '.gif']
    var isSvgFont = false
    if (extension === 'svg') {
      isSvgFont = fs.readFileSync(filename, 'utf8').includes('</font>')
    }
    return callback(null,
      allowedExtensions.includes(extension) && !isSvgFont
    )
  }
. . .

The parse part is already working for jus.

Changes

• Break down stylesheet type in each file extension handler like stylesheetCSS, stylesheetLESS , stylesheetSASS and so on...
• Modified some "stylesheet" tests a little bit

@zeke

The lib/files directory looks like "plugins" (it can even be renamed plugins if you think so).

About the parse part in the "plugins":

I think the "plugins" should do some extra work to be really autonomous and extensibles.

1. Have isSqueezable() inside
2. Have isRemovable() inside. This is new in the code, take a look.
3. Handle its own targetExt

The above are NON-breaking changes for jus

They imply more than just parse and here I see an strategic decision in front of us. contextualizer does not need to write. It only needs to build the context. But applications use context to write.

I think we can let the writing be baked-in contextualizer but optionally be left-out by default.

What do I mean?

The user will be coding "magic" inside the plugin already. As well he/she could code "writing magic" inside or outside the plugin. So let them choose. This way, the casual coder could use contextualizer writing framework with minimum effort (...like we may do for jus ).

Understand that this come at a price of having more than two methods check and parse.

...well...

About a using jus as a testing app for contextualizer.

I think we are about to hit the ceiling, the following are breaking changes

It will be desirable to:

1. Remove pluralize()
2. Remove file.type.toLowerCase()

This looked nice for jus but they have no use for contextualizer.

This decision can be delayed but... Maybe it would be a good idea to define a new jus version so we can code without the "breaking" limits.

Changes

• Added method .use() to load "plugins"
• Removed unused file type definition logic (typer.js and extensions.js)

jus.js

...
module.exports = function jus (sourceDir, targetDir) {
  sourceDir = sourceDir || path.normalize(process.cwd())
  targetDir = targetDir || path.normalize(tmp.dirSync().name)
  const emitter = ee()

  // TODO: Add bulk file type handlers load in opts
  const sentinel = new Contextualizer({'targetDir': targetDir})

  const dir = path.relative(process.cwd(), __dirname)
  const filesDir = path.resolve(dir, 'files')

  // This has to be synchronous
  // ... we do not want watcher to start without all file type handlers loaded
  fs.readdirSync(filesDir).forEach((file) => {
    // Ignore file `file.js` (because it is not a type handler, it contains...
    // ... parent class File) and files without '.js' extension
    if (path.basename(file, '.js') === 'file' || path.extname(file) !== '.js') return
    sentinel.use(path.resolve(filesDir, file))
  })

  sentinel.watcher(sourceDir, {'ignored': ignored(sourceDir, targetDir)})
    .on('adding', (file) => {emitter.emit('file-add', file)})
    .on('updating', (file) => {emitter.emit('file-update', file)})
    .on('deleting', (file) => {emitter.emit('file-delete', file)})
    .on('contexting', (files) => {emitter.emit('squeezing', files)})
    .on('contexted', (ctx) => {emitter.emit('squeezed', ctx)})

  process.nextTick(function() {
    emitter.emit('started')
  })

  return emitter
}

Coding Modularize within jus

• Still not breaking anything. jus works as usual ! 😄
• Pass all test. Note: Some test edited to avoid "false negatives"

Changes

• Move write logic out of contextualizer "plugins"

@zeke

Some thoughts about "plugins" having only two functions check and parse

• I see the modularity benefits of having only "read-like" behavior
• This may bit back when coding for "write-like" and "serving-like" behaviors. For example: Filling file.path.target will be out of the "plugins" so it will have to be implemented after parse, not in parse
• I am noting "duplicated" code in similar "plugins" (a price to pay for modularity)

...so, I am sold. I am working on pruning the "plugins" to end only in two functions

This may bit back when coding for "write-like" and "serving-like" behaviors. For example: Filling file.path.target will be out of the "plugins" so it will have to be implemented after parse, not in parse

As I recall, that file.path.target thing is kind of a mess... sorry!

Would it make sense to make each plugin's parse method responsible for declaring the output extension?

For example:

parse('path/to/foo.sass', function(err, result) {
   console.log(result)
)

And result would look like this:

{
  extension: '.css',
  output: 'transpiled CSS here...'
}

Maybe not a good idea.. just thinking out loud. What do you think? Maybe better to leave this as something for jus to handle internally?

Would it make sense to make each plugin's parse method responsible for declaring the output extension?

For now, I think it is going to be necessary. So we can "untangle" the parsing behavior build in jus.

Later, I think we could manage to have it out. It could be managed by groups of "plugins" like stylesheets that will have .css output.

I am thinking of a directory named plugins (instead of the actual files) that will have sub directories named after those groups. That way it will be simple to have rules about behaviors without having to "hack" them inside the "plugins".

...well, that is possible solution... will see.

By the way, how do you feel about the rename of files directory per plugins ?

Grouped "write-like" behavior in class constructor

this.setTargetPathsAndHref(sourceDir, targetDir, '.html')

page.js

module.exports = class Page extends File {
  constructor(filepath, sourceDir, targetDir) {
    super(filepath, sourceDir)
    this.setTargetPathsAndHref(sourceDir, targetDir, '.html') // force target extension
    // Overwrite `file.href` because "index" is a special routing case
    this.setIndexHref()
    this.isRenderable = true // means, will be render before written in targetDir
  }

  // calls back with a boolean indicating whether this class should process the given file.
  static check (filename, callback) {
    const extension = path.extname(filename).toLowerCase()
    const allowedExtensions = ['.html', '.md', '.mdown', '.markdown', '.handlebars', '.hbs']
    return callback(null, allowedExtensions.indexOf(extension) > -1)
  }

  setIndexHref() {
    var tail = this.isIndex ? this.path.dir : path.join(this.path.dir, this.path.name)
    this.href = path.join(process.env.JUS_BASEDIR, tail)
  }
. . .

• Still pass all test
• Plugins work OK

@zeke
I did a lot of changes but still back version compatible. Nothing broken!

I just tested with two datafile plugins. One for JSON and one for YAML.

(...take a look at the defaultPlugins for the plugin object API)

@zeke Found out that the word parse is too limiting. It can not solve all the modular opportunities for jus. It works perfect for datafiles like JSON and YAML but it does not fit for all other file types.

I suggest a plugin object with more words. Words like render or mine from "mining" (like in data mining). The word that better fit the main function performed over a particular file type.

• Main functions
  • parse for datafiles
  • mine for images
  • render for pages, scripts and stylesheets

There are other posible functions performed over files. I can think of transform or maybe translate.

The plugin could have many words, all optional, including parse. With check as the only mandatory function. This way, if you want a plugin with JUST check and parse, you have it. But, there is room for many more functions to be creative.

Underlaying approach

1. Description

• There are many file types. Each one has its own class that extends File.
• Each plugin is designed specifically for one file type only.
• There is only one check function per plugin.
• Each file type could have many plugins. It depends on "how narrow" the check function is. It could select based on more characteristics than the file extension (see the defaultPlugins/images.js that filters based on content of files .svg)

2. Mechanics

• Load 7 defaultPlugins as failovers. One for each file type
• Load custom plugins, assigning the file type each serve (there are three optional way to do it)
• The check selects which plugin
• The plugin instantiate its only possible file type class
• The "payload" of the plugin is "injected" in the file instance
• ...everything else continues in jus as usual (even tests)

Plugin possibilities

Now you can change markdown processor, template engine, add decorators, etc...

Plugin Modules work OK

• Pass all original tests
• Still nothing broken. I just ran serve and compile OK for:
  • jus.js.org
  • zeke.sikelianos.com
  • acrophony

@zeke just built a css-next plugin to replace myth and also a marky-markdown-lite plugin to replace marky-markdown. They work great. The future has no limits!

The plugins have a mandatory check(), optional parse(), render()and toExtension() .... and some optional auxiliaries name and filetype.

So, if you just want to use check() and parse only, it's OK. You can ignore all the other functions and parse will still work as originally planned.

@zeke @wmhilton @jdormit @djfdev I know it is far, but... What's the next step for the plugins toward master? ...or... Should we go all the way toward Electron?

How to try the plugins?

Some intro and instructions here

1. I suggest you first try the branch as it is. You will be using the "standard" plugins with the original jus features and dependencies. But running in the plugins
2. ...then, add both datafileJSON.js and datafileYAML.js plugins to override the "standard". Just drag and drop them from lib/plugins/stored to lib/plugins/use. That's it. You will see a console.log() message saying Using plugin in <plugin filename> for file <file.href>. This message is just a proof convenience. It will be deleted in future commits.
3. ...finally, follow the instructions mentioned above to test the css-next plugin. Be aware that there is a bug in npm caniuse dependencies. But, there is also an easy workaround for you guys to test (I am not planning to release this particular plugin yet. It is just my way to "show off" the new plugins capabilities)

Whoa! You couldn't have made lots of tiny snack sized PRs? It'll take me an hour to read all this... 😭 Hehe

@wmhilton You are correct. Everytime I do not follow your advice on frequent small chunks. I have to start all over.

Before I do that.

• This plugin stuff solves the Modularizing issue Modularizing jus #61 and some others indirectly like the replacement of myth issue Consider replacing myth with cssnext #24 (...and good help in others like minify JS when building #7 and minify CSS when building #8)
• @zeke and I decided to do a PR just for easy review and commenting. Not really planning a merge to master (that is why de WIP tag). So, @wmhilton please do not feel like I want to get it to master. I said that but my intended meaning was different

It was more like:

What do you guys think about this plugin development? Should it go to master in the far future or should it go in parallel toward something new like Electron?

The questions are in general form, like vision. Not specific in the sense of merge approval.

I am aware of the time limitations we all have. I do not want to be a burden. I only need some direction hints on what would be nice for jus org

electron-jus could be a new project or maybe contextualizer ...or jus version 1.0 ...

For instance, if we end up toward jus version 1.0, then I will start a new branch from master and will follow the @wmhilton advice, frequent small chunks for easy review. It will be kind of simple because we already know the end result will be something like this WIP PR

For me will be an opportunity to "polishing" the code

Added Handlebars Partials support

New plugin page-partials.js (already in lib/plugins/use)

• Nothing broken
• Original tests OK

It uses the layouts file type as partials. So the names of the partials should be like layout-mypartial.hbs

• Nested partials work OK
• Context is available down to the deepest
• Helpers work as always

It is a plugin so it can be removed with drag and drop to lib/plugins/stored

@elingerojo Ah, OK. I will skim the code and your detailed comments and try to get the gist of what this is about. :)

OK, I will try to summarize it and you can tell me if I understand it or not. The most helpful thing for me was reading #61 in which the word "contextualizer" is (first?) coined. That helped explain a lot... knowing the context of the contextualizer.

The goal with the contextualizer is to super-charge chokidar with the ability to extract metadata from files and make a kind of live-updating tree of the file system. At first, I am concerned we might be reinventing the wheel - are we just recreating gulp or broccolijs? - but then I realized the important difference. Gulp, Webpack, Browserify... all are explicit e.g. they trace require() dependencies or you must add files with glob patterns. The contextualizer tree is implicit in that it has no end goal in mind yet, just gets as much information as possible out of the directory and presents it to you. That seems novel... that is I'm not aware of any existing tool designed for this... broccolijs comes closest because it is a build system where file trees are the fundamental unit. We might be able to pilfer code or ideas from there.

This could be a very interesting thing. For instance, I use a file system watcher in my ransomware detector. It might not be suited for that particular use case because its watching the entire file system and not trying to hold it in memory. But my point being that "filesystem watcher + super parser powers" is a generally great and useful idea. We might even want to make that its own project, and then jus would be an application that takes this virtual filesystem object and does opinionated compilation / build transforms on it before writing it to disk.

Obviously a key component is the File class. Should we use/extend an existing virtual file package? I haven't used any, but I've researched them before and thought that vfile looked quite nice. Like, actually really nice and only one file with < 300 LOC, and seems to do one thing really well. It has some very interesting metadata it tacks onto files too: an optional error message with line/col for associating linting errors, and a .history property that tracks file renames (I believe) which could be super cool for tracking file transforms & concat/merge operations. Again, I haven't actually used it... but that's life! So much to do, so little time.

I think we should come up with a neat name for it, better than 'contextualizer'. Maybe ReactFS or something?

The goal with the contextualizer is to super-charge chokidar with the ability to extract metadata from files and make a kind of live-updating tree of the file system.

@wmhilton Yeah, description is spot on

But my point being that "filesystem watcher + super parser powers" is a generally great and useful idea...

vfile looks awesome (thanks to bringing it to the table, wasn't aware of it).

I think "filesystem watcher + configurable virtual files + super parser powers" is dynamite.

... We might even want to make that its own project, and then jus would be an application that takes this virtual filesystem object and does opinionated compilation / build transforms on it before writing it to disk.

Again, right on spot!

• contextualizer or ReactFS (...or maybe reactFS) as new project
• jus v1.0 as "by-product" of testing the above feasibility. jus will be the real app litmus test that core is clean and useful. I visualize the contextualizer (or whatever we call it, ...damn!) README saying: "...used by jus..."

It will be two birds with one stone

So the steps forward will be...

1. Extract contextualizer (...we should have a code name while deciding on the public name... will use contexter). So, continuing... Extract contexter from this alpha-dynamic-class branch
2. Create a temporary new repository until we decide to put it in jus org or not
3. Create a new branch from master to develop the jus transition to using contexter. This branch will have the snack sized commits in a new WIP PR
4. Close the present WIP PR

Closed in favor of #70 where the "modularizing" stuff is extracted in npm contexter for convenience