Merge topic/dockerperms to address issue cisagov#137 (cisagov#138) (c…

…isagov#139)

This pull request adds the some new environment variables for Malcolm to address cisagov#137

* `PUID` and `PGID`
  * Docker runs all of its containers as the privileged `root` user by default. For better security, Malcolm immediately drops to non-privileged user accounts for executing internal processes wherever possible. The `PUID` (**p**rocess **u**ser **ID**) and `PGID` (**p**rocess **g**roup **ID**) environment variables allow Malcolm to map internal non-privileged user accounts to a corresponding [user account](https://en.wikipedia.org/wiki/User_identifier) on the host.

Additionally, this pull request additionally moves all remaining process that can be run non-privileged to run as non-privileged.

Each docker container now has the following in its Dockerfile (this example is from the zeek container, they're all similar but may have different specific values):

```
ARG DEFAULT_UID=1000
ARG DEFAULT_GID=1000
ENV DEFAULT_UID $DEFAULT_UID
ENV DEFAULT_GID $DEFAULT_GID
ENV PUSER "zeek"
ENV PGROUP "zeek"
ENV PUSER_PRIV_DROP true
```

The entrypoint of each docker comtainer is now [docker-uid-gid-setup.sh](https://github.com/idaholab/Malcolm/blob/master/shared/bin/docker-uid-gid-setup.sh), which does the following:

1. changes the UID and GID of the default (1000:1000) user to match the PUID:PGID provided
2. finds any files *inside* the docker image owned by those IDs and chown them
3. if required, execs the container command by dropping privileges to the unprivileged user

Additionally, control.py (used for start, restart, etc.) will now error out it run as root rather than just running with a bunch of errors. Malcolm should not be run as a root user.

Merge remote-tracking branch 'idaholab/development'

Merge branch 'topic/llvm10'

Merge remote-tracking branch 'downstream/master'

minor updates to the brew isntructions for macOS

Squashed commit of the following:

commit 192e5c3
Author: SG <13872653+mmguero@users.noreply.github.com>
Date:   Tue May 5 13:03:33 2020 -0600

    bump version to 2.0.3

commit 2c2de9f
Author: SG <13872653+mmguero@users.noreply.github.com>
Date:   Tue May 5 12:51:14 2020 -0600

    create ca-trust directory on boot

commit 4c24aff
Author: SG <13872653+mmguero@users.noreply.github.com>
Date:   Tue May 5 11:55:59 2020 -0600

    remove unused nginx PAM module, and comment setting up initial crt symlinks

commit 04b12ce
Author: SG <13872653+mmguero@users.noreply.github.com>
Date:   Mon May 4 14:32:16 2020 -0600

    for idaholab#128, provide directory to put CA certificates for trusted LDAP servers

make sure patch for idaholab#122 is also applied to sensor-iso (hedge…

…hog)

Malcolm v2.0.1 (from topic/zeek304) (cisagov#124)

* update zeek to 3.0.5 to address a security vulnerability

v2.0.0 development (cisagov#106)

Tons of stuff.

* bump moloch to 2.2.0

* reduce log noise

* bump version for development to 1.8.2

* set elastalert index settings for a single node

* fix issue cisagov#97, when using tcpdump the capture files are named .pcap.pcap

* check moloch viewer status page periodically for docker container health check

* fix docker-compose log verbosity

* made kibana_index_refresh.py more robust as suggested by @fabrie in issue cisagov#100

use a _find API to get the index ID for a given index name instead of just the saved_objects/index-pattern API

Example in test environment:

Before adding new fields (dry run):
```
/home/user/devel/github/malcolm/kibana/scripts/kibana_index_refresh.py
Arguments: ['-v', '-n', '-k', 'http://192.168.0.11:5601/kibana']
Arguments: Namespace(debug=True, dryrun=True, index='sessions2-*', url='http://192.168.0.11:5601/kibana')
Kibana version is 7.5.1
Index ID for sessions2-* is sessions2-*
sessions2-* would have 465 fields
success (dry run only, no write performed)
```

After adding new fields (dry run):
```
/home/user/devel/github/malcolm/kibana/scripts/kibana_index_refresh.py
Arguments: ['-v', '-n', '-k', 'http://192.168.0.11:5601/kibana']
Arguments: Namespace(debug=True, dryrun=True, index='sessions2-*', url='http://192.168.0.11:5601/kibana')
Kibana version is 7.5.1
Index ID for sessions2-* is sessions2-*
sessions2-* would have 481 fields
success (dry run only, no write performed)
```

After adding new fields (update index mapping):
```
/home/user/devel/github/malcolm/kibana/scripts/kibana_index_refresh.py
Arguments: ['-v', '-k', 'http://192.168.0.11:5601/kibana']
Arguments: Namespace(debug=True, dryrun=False, index='sessions2-*', url='http://192.168.0.11:5601/kibana')
Kibana version is 7.5.1
Index ID for sessions2-* is sessions2-*
sessions2-* would have 481 fields
success
```

* added plugin for detecting cve-2020-0601

* work on issue cisagov#102, log access to Malcolm web interface(s) to Elasticsearch for analysis in Kibana

* nginx/php adjustments for issue cisagov#101, uploading very large pcap files may fail

* fix a few of the control bash scripts to use GNU coreutils where applicable (issue cisagov#103)

A few uses of "grep" and "find" use flags unique to the GNU versions of those tools. As GNU coreutils was already required for a few other utilities I've done the same thing to detect and use ggrep and gfind when needed.

As suggested, it might be a good idea to rewrite these to use Python instead to be more portable (although I'll have to take care to make them work with both python 2/3 for various platforms).

* bump malcolm version to 1.9.0, moloch version to 2.2.1

* work on implementing control scripts (start,stop,restart,wipe,logs) in python rather than bash for portability (see issue cisagov#103). have not as of yet removed the bash versions, as I am still testing these new implementations. for now the auth_setup.sh and malcolm_appliance_packager.sh are still in Bash as well

* have ISO use new scripts

* bump elastic to 7.5.2

* update iso build scripts to use new python scripts for install

* compatibility fixes for scripts under linux

* don't source missing files

* more reworking of scripts from bash -> python (not complete yet, may be in a broken state)

* more work on auth_setup

* remove reference to files we're not longer using

* Revert "bump elastic to 7.5.2"

This reverts commit 440c859.

* fix default for external password question

* handle missing python package in windows

* documentation updates

* tweak some codenames

* use specified path (rather than absolute path) for compose file

* make python3 the default

* Revert "make python3 the default"

This reverts commit 52e53f4.

* fixes for new control scripts

* fix logs script hanging

* pin filters by default in kibana

* put a hack/fix in for vagrant not liking dhcp nat in 6.1

* create a zeek.service_version field to track protocol version in a single place; also, move password up to the zeek root level

* added security overview dashboard wip

* added freq.Dockerfile to detecting string entropy

* if designated by the FREQ_LOOKUP (true) environment variable, look up DNS query hostnames using freq_server.py

* update docker ignore ifle

* ask about string freq lookup in install.py

* added security overview dashboard wip

* added security overview dashboard wip

* use a ruby block rather than an http filter in order to better handle arrays

* fix volume mapping for local.zeek in docker-compose.yml for testing

* fix volume mapping for local.zeek in docker-compose.yml for testing

* clean up symlinks as well

* initial code for generating and parsing smb_cmd.log

* initial code for generating and parsing smb_cmd.log

* more work on smb command mapping

* more work on smb

* more work on smb

* bump version to 2.0.0

* some field normalization for 2.0.0

- restored kibana swimlane visualization as it has been fixed for 7.5.x
- remove some unused fields from records (agent.ephemeral_id, input.type, path portion of log.file.path)
- remove "_jsonparsefailure" tag on cleanup
- change some places where we were doing calculations to get count values when we already know the count is "1"
- normalization of "action" or "command" values to "zeek.action" field
- normalization of mime type, file names, fuids, and service version

* fixes to SMB action mapping

* remove useless prefix before smb action

* exclude some domains from freq. analysis

* utility script to repackage zeek logs for upload:

* fix issue cisagov#111, moloch/etc mount in docker-compose.yml causes custom Zeek fields not to be loaded

* fix Malcolm issue cisagov#110, submitting hunt job crashes viewer unless Zeek logs are filtered out (temporary patch of fix for Moloch issue 1374, arkime/arkime#1374)

* Added smb_cmd fields to WISE

* fix dashboard referring to zeek_smb.action -> zeek.action

* remove tunnel:: prefix from tunnel type

* added 'action' panel to overview

* added security overview dashboard (wip) to directory

* more work on issue cisagov#108, create security overview dashboard in kibana

* working on issue cisagov#109, create ICS security overview dashboard

* added ipv4/ipv6

* working on issue cisagov#109, create ICS security overview dashboard

* working on issue cisagov#109, create ICS security overview dashboard

* added network layer to connections

* fix max font size

* bring sensor local.zeek up to match malcolm's

* fixed spacing of navigation menu

* fix issue cisagov#112, region maps not working because of incorrect redirect

* fix issue cisagov#112, region maps not working because of incorrect redirect

* fix issue cisagov#112, region maps not working because of incorrect redirect

* fix issue cisagov#112, region maps not working because of incorrect redirect

* comments

* ignore logs that have been renamed and are in transit being archived

* updates to dashboards

* fix kibana_index_refresh.py for python2

* fix non-ics/iot protocols dashboard

* bump version to 7.6.0 for elastic

* working with es 7.6, but elastalert had to be temporarily disabled. will work on fixing this next

* don't include known_certs in outdated/insecure protocols

* Tons of work refining dashboards

* tweak connections view

* improved maps

* remove warnings

* improvements to how notices can be used througout the other dashboards

* improvements to how notices can be used througout the other dashboards

* do frequency analysis on zeek_ssl.server_name

* merge src/dst mac/oui fields into network.mac and network.oui arrays, respectively to provide aggregated fields that can be used for asset inventory (issue 113)

* experimenting with creating a merged network.mac_oui field that looks like this:

...
    "network": {
      "type": "ipv4",
      "mac_oui": {
        "00:10:db:ff:10:01": "Juniper Networks",
        "8c:85:90:65:85:8f": "Apple, Inc."
      }
    },
...

however, I may revert this for now because although this works kibana does't really play nicely with the data in visualizations

* Revert "experimenting with creating a merged network.mac_oui field that looks like this:"

This reverts commit 8bdcefa.

* Revert "merge src/dst mac/oui fields into network.mac and network.oui arrays, respectively to provide aggregated fields that can be used for asset inventory (issue 113)"

This reverts commit ae60cf2.

* make installer work better for vms

* build virtualbox guest debs in a clean environment

* fixed vagrantfile for malcolm build

* only keep vmware/virtualbox guest packages in the right environments

* increase build memory requirements

* fix typo

* updating sensor-iso to match malcolm-iso

* fix relative path

* removed docker-gen in nginx container, we're not using it any more

* update software saved search

* Fixed installation of elastalert kibana plugin, but still broken due to this issue: bitsensor/elastalert-kibana-plugin#141

* fix issue cisagov#104, Upload without trailing slash redirects to incorrect hostname and scheme

this fix includes a few things:

- modifying the index.html page to prepend "upload/" before relative HREF/SRC references
- removing some useless code in the file-upload default nginx config
- added the trailing slash to the proxy-pass directive for the upload section of the main nginx proxy
- handle "/server/php" as a separate proxy redirect as that's the XHR where the uploads seem to go
- remove the unused Moloch upload page

* update copyright

* update style of upload screen to match the rest of the app

* working on network diff code (wip)

* work in progress on network time diff, viewer.js not actually used yet

* bump moloch to 2.2.2

* work in progress for network diff

* network diff work in progress

* comments/work in progress

* some test files

* update zeek to 3.0.2

* fix reference to zeek::af_packet

* install zeek::af_packet with zkg

* Revert "install zeek::af_packet with zkg"

This reverts commit a20fa9b.

* added docker files for running moloch regression tests

* added vim to test harnest

* fix af_packet zeek build

* added promotional poster:

* switch test harness branch

* temporarily pull from https://github.com/mmguero-dev/moloch fork for issue cisagov#2 development rather than aol/moloch releases

* update elastic to 7.6.1 for security and bug fixes

* update moloch to 2.2.3

* update moloch to 2.2.3

* update zeek to 3.0.3

* update psutil to fix security alert GHSA-qfc5-mcwq-26q8

* zeek updated website, fix broken link

* restore stuff for generating web documentation

* fix URL for relocated MITRE ATTACK BZAR plugin

* fix broken links for build of Zeek, MITRE ATT&CK BZAR plugin

* fixed duplicate plugin URL in script

* update moloch version in docs to 2.2.3

* remove files no longer needed for testing

* remove files no longer needed for testing and update moloch version in documentation

* should fix issue cisagov#114. I discovered that even though moloch-capture isn't writing the PCAP files, the pcapDir and maxFileSizeG values still matter for viewer to be able to delete managed pcap files.

* should fix issue cisagov#114. I discovered that even though moloch-capture isn't writing the PCAP files, the pcapDir and maxFileSizeG values still matter for viewer to be able to delete managed pcap files.

* proof of concept for a segment mapping form

* work in progress on the segment mapping ui

* more work on the segment mapping ui

* more work on the segment mapping ui

* more work on the segment mapping ui

* more work on the segment mapping ui

* more work on the segment mapping ui

* more work on the segment mapping ui

* apply tooltip for table columns

* scroll back and forth to selected item

* beautify with icons

* basic validation client-side

* more work on the segment mapping ui (integration with malcolm scripts on logstash startup)

* more work on the segment mapping ui (creation of docker image, integration with malcolm's nginx reverse proxy)

* Added new icon to malcolm iso for subnet mapping editor

* documentation updates

* start logstash under supervisord in order to add a process that will watch for changes to the name matching

* more work on name-map-ui, allow uploading of the JSON file so it can be pushed to the docker image volume automatically

* map location of host/subnet mapping to correct location under name-map-ui container

* integrate upload with name-map-ui

* add the ability to signal logstash from the net-map-ui container

* clear out previous maps between restarts

* add ability to save net-map.json from web ui

* basic control for restarting logstash via ui controls

* put save/restart confirmations in UI

* added import button to name map ui

* send save-state post value to restart-logstash.php

* update documentation

* update documentation

* remove unused variable

* documentation updates

* use fonts-symbola instead of fonts-noto-color-emoji

* re-enable swimlane visualization

* update elasticsearch to 7.6.2; also, fix issue cisagov#119

* use default theme in elastalert kibana editor

* update kibana plugin version

* add user to vboxsf group for using shared folders

* the 'run a separate instance of Zeek locally' use case isn't really a big enough use case to have a whole separate docker-compose file for it; especially with the ISO and live capture methods.

* ensure all services have a health check

* reduce verbosity of health checks in logs

fix broken links for build of Zeek, MITRE ATT&CK BZAR plugin