this “project” consists of several “sub-projects” whose only unifying theme is that they are somehow of interest (maybe) to the DTF group chat.

although the sub-projects are thematically disparate from one another, we keep them here in one repository for reasons.

the specifics of development will be determined by the technologies employed by the particular sub-project.

however, these principles apply across the sub-projects:

• see the below guidelines on version control.

• see installing and deploying python if yours is one of the several sub-projects that uses python.

• also for python, see our [#010] extra-conventional conventions.

• passing all the tests MUST generally be considered a prerequisit for pushing to the master branch. (we may explore allowable exceptons later.)

• TODO.stack

  • one line per item
  • see [#004] using the todo stack

• bin/

  • of course don't put actual binary files here. this is for executable files written in scripting languages.

• doc/

  • typically this is for static, semi-human readable documents (like markdown, graph-viz)

  • you will need GraphViz to see the dotfiles (*.dot)

  • the numbered documents in this directory appear in the node table below.

• script/

  • this is like bin/ but for little development utilities and one-offs.

• static-html/

  • as the name implies, this is for the rare static (i.e versioned, probably not generated) html

• «sub-project»

• «sub-project» (etc)

The repository that this document lives in has many directories at its top level. The remainder of this section describes the charactersitics these directories tend to share, and how they tend to relate to each other; all towards reaching an understanding of the "architecture" of this repository:

• Each of these directories is frequently (but not always) some sort of software module. So in essence, this repository is mainly a collection of software modules.

• These software modules do not necessarily exist in the same universe with each other; that is, they are not necessarily intended to all be in the same "runtime". Some of the modules may be written in python; others in ruby. Some may be written with a compiled language, others may be for Angular or React.

• (However, there's no reason we can't hypothetically chain together several scripts meaningfully across "platform" boundaries (for exmaple, piping the output of a ruby script into a python script).)

• We'll call these directories "sub-projects", not "modules": calling them all "modules" may mistakenly confer the idea that (one) they are all software (more below on this) and (two) they all exist in the same universe with each other. Ditto "packages".

• We'll avoid the term "project" because without context, it's ambiguous whether it means "the one big project" or a particular sub-project. Rather, we call this one big directory "the mono-repo".

• A sub-project can be as small as a single-file one-off or as large as a full-stack application.

• A sub-project isn't necessarily software at all: it may simply be documentation (e.g., reading notes) or other text-adjacent assets we want to keep in version contorl for whatever reason.

• For those sub-projects that do depend on others here, the rule is that these dependencies must not cycle (they must be acyclic): any given module here that depends on other modules here must do so as a "stack", with the module in question at the "top" and its depended-upon modules "below" it.

• as for why in the world we maintain this one big repository and not several smaller ones, here is the rationale: we want the sub-projects to "feel" like part of a greater whole in the hopes that this will encourge them to share dependency, architecture and testing decisions whenever it's prudent to do so; in an arrangement where both the "library" and "application" codebases are moving targets under active development. also:

• it is our experimental belief that in its way, maintaining a project of sub-projects can help combat "software erosion" (an idea mentioned in 12 factor, explained well in this old heroku blog, and defined more formally in a wikipedia page.) the fear (wrought from experience) is that if a smaller project is off in its own little corner, it's easy to forget it exists for a few years. then when it's re-discovered and dusted off, the cost of getting it to work again is high; sometimes prohibitively so. keeping small projects close together in a larger repository keeps them all at the fore, so that the decision of whether or not to continue to maintain a small project is one made intentionally, rather than as a circumstantial afterthought. but all of this comes with this next provision:

• certainly if a sub-project grows to any degree of usefulness and maturity (or just sheer size in terms of SLOC), it should fork from this project (repository), be sunsetted here, and git-filter-branch (or similar) used to prune its extraneous history there.

• our only real provision for sub-projects is that if they contain more than one file that they live at the top of this directory tree. (i.e., there are no sub-sub-projects, etc.)

For a sub-project called "foo_bar", run the corresponding test suite "foo_bar_test" with pud foo_bar_test (using an alias described the below section on aliases. For example, for the "grep_dump" sub-project:

pud grep_dump_test

That's all.

We're currently experimenting with different ways we run the whole test suite universe-wide (something we don't usually do at each commit).

The below uses an alias described in the relevant section.

This is the current sketch:

cat mono-repo.test-these.list | while read line; do >&2 echo -n " $line "; 2>&1 pud -fq "$line"; if [ $? -ne 0 ]; then; >&2 echo "ERRRORRED"; break; fi; done | awk '/^Ran ([0-9]+) test/ { printf "+%d", $2}'

This gives progressive output only of each sub-project name as it starts and doesn't show a stack trace. (If the test suite X failed, just do pud -vf X to see details).

We use the python module called coverage. Consult their documentation.

But here's an example that worked at the moment of writing:

coverage run --source pho.cli.commands.issues pho_test/test_3925_issues/test_3975_CLI.py -vf
coverage html
open htmlcov/index.html

testing the API server of the "upload bot" is more involved. it its own terminal:

upload_bot_test/script/run-web-server

(if the server is already running, the PID of the process is displayed; otherwise the server starts in that terminal. stopping and starting the server is necessary whenever relevant code changes, like when you're jumping version. (Ctrl-c stops the server.))

then in your 'main' terminal:

node_modules/newman/bin/newman.js run upload_bot_test/test_700_web/test_100_intro.postman_collection.json

(this presupposes that npm install was run at some point.)

This is out of scope for this readme, but apparently we don't have a dedicated document on the subject yet, so we're stowing this away in here for now: Whirlwind tour of how to doctest:

In the asset code:

def my_ting():
    """
    >>> my_ting(('A', 'B', 'C'))
    'A, B and C'
    """

At the bottom of the asset file:

def _run_doctest():
    from doctest import testmod as func
    func()


if __name__ == "__main__":
    _run_doctest()

And, to get these to run from some proximal unit-test file:

def load_tests(loader, tests, ignore):  # (this is a unittest API hook-in)
    from doctest import DocTestSuite
    tests.addTests(DocTestSuite(subject_module()))
    return tests

These aliases are now installed by the "dotfiles" sub-project (see).

alias py='python3 -W error::Warning::0'

• At #history-A.2 we had to let warnings thru for tatsu, but this may change at any moment
• Wanting to exit with an error at the first warning is peak OCD

alias pud='py -m unittest discover'

alias pi='py pho/cli/commands/issues.py'

• Can also be reached by pho issues, but the above saves a little overhead
• (run setup.py to get the pho executble)

• please make sure you don't leave trailing whitespace on to the lines of any files unless you mean to.

• when renaming files (i.e moving them), please commit the move(s) in a dedicated commit separate from commiting any edits to those files that moved. (reading edits to files in a diff can be ugly when it is paired with a move of the file, depending..)

• write commit messages following these git commit message guidelines; namely that the first line of the message start with a capital letter, not have ending punctuation, be in the imperative mood, etc.

(this table is explained at [#002] using the node table.)

• #history-B.2
• #history-A.2
• spike node table
• #born.