Tons of stuff.
* bump moloch to 2.2.0
* reduce log noise
* bump version for development to 1.8.2
* set elastalert index settings for a single node
* fix issue #97, when using tcpdump the capture files are named .pcap.pcap
* check moloch viewer status page periodically for docker container health check
* fix docker-compose log verbosity
* made kibana_index_refresh.py more robust as suggested by @fabrie in issue #100
use a _find API to get the index ID for a given index name instead of just the saved_objects/index-pattern API
Example in test environment:
Before adding new fields (dry run):
```
/home/user/devel/github/malcolm/kibana/scripts/kibana_index_refresh.py
Arguments: ['-v', '-n', '-k', 'http://192.168.0.11:5601/kibana']
Arguments: Namespace(debug=True, dryrun=True, index='sessions2-*', url='http://192.168.0.11:5601/kibana')
Kibana version is 7.5.1
Index ID for sessions2-* is sessions2-*
sessions2-* would have 465 fields
success (dry run only, no write performed)
```
After adding new fields (dry run):
```
/home/user/devel/github/malcolm/kibana/scripts/kibana_index_refresh.py
Arguments: ['-v', '-n', '-k', 'http://192.168.0.11:5601/kibana']
Arguments: Namespace(debug=True, dryrun=True, index='sessions2-*', url='http://192.168.0.11:5601/kibana')
Kibana version is 7.5.1
Index ID for sessions2-* is sessions2-*
sessions2-* would have 481 fields
success (dry run only, no write performed)
```
After adding new fields (update index mapping):
```
/home/user/devel/github/malcolm/kibana/scripts/kibana_index_refresh.py
Arguments: ['-v', '-k', 'http://192.168.0.11:5601/kibana']
Arguments: Namespace(debug=True, dryrun=False, index='sessions2-*', url='http://192.168.0.11:5601/kibana')
Kibana version is 7.5.1
Index ID for sessions2-* is sessions2-*
sessions2-* would have 481 fields
success
```
* added plugin for detecting cve-2020-0601
* work on issue #102, log access to Malcolm web interface(s) to Elasticsearch for analysis in Kibana
* nginx/php adjustments for issue #101, uploading very large pcap files may fail
* fix a few of the control bash scripts to use GNU coreutils where applicable (issue #103)
A few uses of "grep" and "find" use flags unique to the GNU versions of those tools. As GNU coreutils was already required for a few other utilities I've done the same thing to detect and use ggrep and gfind when needed.
As suggested, it might be a good idea to rewrite these to use Python instead to be more portable (although I'll have to take care to make them work with both python 2/3 for various platforms).
* bump malcolm version to 1.9.0, moloch version to 2.2.1
* work on implementing control scripts (start,stop,restart,wipe,logs) in python rather than bash for portability (see issue #103). have not as of yet removed the bash versions, as I am still testing these new implementations. for now the auth_setup.sh and malcolm_appliance_packager.sh are still in Bash as well
* have ISO use new scripts
* bump elastic to 7.5.2
* update iso build scripts to use new python scripts for install
* compatibility fixes for scripts under linux
* don't source missing files
* more reworking of scripts from bash -> python (not complete yet, may be in a broken state)
* more work on auth_setup
* remove reference to files we're not longer using
* Revert "bump elastic to 7.5.2"
This reverts commit 440c85980b4e58064e164294ee395cf503e93487.
* fix default for external password question
* handle missing python package in windows
* documentation updates
* tweak some codenames
* use specified path (rather than absolute path) for compose file
* make python3 the default
* Revert "make python3 the default"
This reverts commit 52e53f4196cf9221333af69d63ce71bdf12b220d.
* fixes for new control scripts
* fix logs script hanging
* pin filters by default in kibana
* put a hack/fix in for vagrant not liking dhcp nat in 6.1
* create a zeek.service_version field to track protocol version in a single place; also, move password up to the zeek root level
* added security overview dashboard wip
* added freq.Dockerfile to detecting string entropy
* if designated by the FREQ_LOOKUP (true) environment variable, look up DNS query hostnames using freq_server.py
* update docker ignore ifle
* ask about string freq lookup in install.py
* added security overview dashboard wip
* added security overview dashboard wip
* use a ruby block rather than an http filter in order to better handle arrays
* fix volume mapping for local.zeek in docker-compose.yml for testing
* fix volume mapping for local.zeek in docker-compose.yml for testing
* clean up symlinks as well
* initial code for generating and parsing smb_cmd.log
* initial code for generating and parsing smb_cmd.log
* more work on smb command mapping
* more work on smb
* more work on smb
* bump version to 2.0.0
* some field normalization for 2.0.0
- restored kibana swimlane visualization as it has been fixed for 7.5.x
- remove some unused fields from records (agent.ephemeral_id, input.type, path portion of log.file.path)
- remove "_jsonparsefailure" tag on cleanup
- change some places where we were doing calculations to get count values when we already know the count is "1"
- normalization of "action" or "command" values to "zeek.action" field
- normalization of mime type, file names, fuids, and service version
* fixes to SMB action mapping
* remove useless prefix before smb action
* exclude some domains from freq. analysis
* utility script to repackage zeek logs for upload:
* fix issue #111, moloch/etc mount in docker-compose.yml causes custom Zeek fields not to be loaded
* fix Malcolm issue #110, submitting hunt job crashes viewer unless Zeek logs are filtered out (temporary patch of fix for Moloch issue 1374, https://github.com/aol/moloch/issues/1374)
* Added smb_cmd fields to WISE
* fix dashboard referring to zeek_smb.action -> zeek.action
* remove tunnel:: prefix from tunnel type
* added 'action' panel to overview
* added security overview dashboard (wip) to directory
* more work on issue #108, create security overview dashboard in kibana
* working on issue #109, create ICS security overview dashboard
* added ipv4/ipv6
* working on issue #109, create ICS security overview dashboard
* working on issue #109, create ICS security overview dashboard
* added network layer to connections
* fix max font size
* bring sensor local.zeek up to match malcolm's
* fixed spacing of navigation menu
* fix issue #112, region maps not working because of incorrect redirect
* fix issue #112, region maps not working because of incorrect redirect
* fix issue #112, region maps not working because of incorrect redirect
* fix issue #112, region maps not working because of incorrect redirect
* comments
* ignore logs that have been renamed and are in transit being archived
* updates to dashboards
* fix kibana_index_refresh.py for python2
* fix non-ics/iot protocols dashboard
* bump version to 7.6.0 for elastic
* working with es 7.6, but elastalert had to be temporarily disabled. will work on fixing this next
* don't include known_certs in outdated/insecure protocols
* Tons of work refining dashboards
* tweak connections view
* improved maps
* remove warnings
* improvements to how notices can be used througout the other dashboards
* improvements to how notices can be used througout the other dashboards
* do frequency analysis on zeek_ssl.server_name
* merge src/dst mac/oui fields into network.mac and network.oui arrays, respectively to provide aggregated fields that can be used for asset inventory (issue 113)
* experimenting with creating a merged network.mac_oui field that looks like this:
...
"network": {
"type": "ipv4",
"mac_oui": {
"00:10:db:ff:10:01": "Juniper Networks",
"8c:85:90:65:85:8f": "Apple, Inc."
}
},
...
however, I may revert this for now because although this works kibana does't really play nicely with the data in visualizations
* Revert "experimenting with creating a merged network.mac_oui field that looks like this:"
This reverts commit 8bdcefaed5d2be765f994abc0959fd713d0ea451.
* Revert "merge src/dst mac/oui fields into network.mac and network.oui arrays, respectively to provide aggregated fields that can be used for asset inventory (issue 113)"
This reverts commit ae60cf27c82fbb6e3fa814601facbac3e06181b8.
* make installer work better for vms
* build virtualbox guest debs in a clean environment
* fixed vagrantfile for malcolm build
* only keep vmware/virtualbox guest packages in the right environments
* increase build memory requirements
* fix typo
* updating sensor-iso to match malcolm-iso
* fix relative path
* removed docker-gen in nginx container, we're not using it any more
* update software saved search
* Fixed installation of elastalert kibana plugin, but still broken due to this issue: https://github.com/bitsensor/elastalert-kibana-plugin/issues/141
* fix issue #104, Upload without trailing slash redirects to incorrect hostname and scheme
this fix includes a few things:
- modifying the index.html page to prepend "upload/" before relative HREF/SRC references
- removing some useless code in the file-upload default nginx config
- added the trailing slash to the proxy-pass directive for the upload section of the main nginx proxy
- handle "/server/php" as a separate proxy redirect as that's the XHR where the uploads seem to go
- remove the unused Moloch upload page
* update copyright
* update style of upload screen to match the rest of the app
* working on network diff code (wip)
* work in progress on network time diff, viewer.js not actually used yet
* bump moloch to 2.2.2
* work in progress for network diff
* network diff work in progress
* comments/work in progress
* some test files
* update zeek to 3.0.2
* fix reference to zeek::af_packet
* install zeek::af_packet with zkg
* Revert "install zeek::af_packet with zkg"
This reverts commit a20fa9b370711f4117e0f470fba643716cdfda55.
* added docker files for running moloch regression tests
* added vim to test harnest
* fix af_packet zeek build
* added promotional poster:
* switch test harness branch
* temporarily pull from https://github.com/mmguero-dev/moloch fork for issue #2 development rather than aol/moloch releases
* update elastic to 7.6.1 for security and bug fixes
* update moloch to 2.2.3
* update moloch to 2.2.3
* update zeek to 3.0.3
* update psutil to fix security alert https://github.com/advisories/GHSA-qfc5-mcwq-26q8
* zeek updated website, fix broken link
* restore stuff for generating web documentation
* fix URL for relocated MITRE ATTACK BZAR plugin
* fix broken links for build of Zeek, MITRE ATT&CK BZAR plugin
* fixed duplicate plugin URL in script
* update moloch version in docs to 2.2.3
* remove files no longer needed for testing
* remove files no longer needed for testing and update moloch version in documentation
* should fix issue #114. I discovered that even though moloch-capture isn't writing the PCAP files, the pcapDir and maxFileSizeG values still matter for viewer to be able to delete managed pcap files.
* should fix issue #114. I discovered that even though moloch-capture isn't writing the PCAP files, the pcapDir and maxFileSizeG values still matter for viewer to be able to delete managed pcap files.
* proof of concept for a segment mapping form
* work in progress on the segment mapping ui
* more work on the segment mapping ui
* more work on the segment mapping ui
* more work on the segment mapping ui
* more work on the segment mapping ui
* more work on the segment mapping ui
* more work on the segment mapping ui
* apply tooltip for table columns
* scroll back and forth to selected item
* beautify with icons
* basic validation client-side
* more work on the segment mapping ui (integration with malcolm scripts on logstash startup)
* more work on the segment mapping ui (creation of docker image, integration with malcolm's nginx reverse proxy)
* Added new icon to malcolm iso for subnet mapping editor
* documentation updates
* start logstash under supervisord in order to add a process that will watch for changes to the name matching
* more work on name-map-ui, allow uploading of the JSON file so it can be pushed to the docker image volume automatically
* map location of host/subnet mapping to correct location under name-map-ui container
* integrate upload with name-map-ui
* add the ability to signal logstash from the net-map-ui container
* clear out previous maps between restarts
* add ability to save net-map.json from web ui
* basic control for restarting logstash via ui controls
* put save/restart confirmations in UI
* added import button to name map ui
* send save-state post value to restart-logstash.php
* update documentation
* update documentation
* remove unused variable
* documentation updates
* use fonts-symbola instead of fonts-noto-color-emoji
* re-enable swimlane visualization
* update elasticsearch to 7.6.2; also, fix issue #119
* use default theme in elastalert kibana editor
* update kibana plugin version
* add user to vboxsf group for using shared folders
* the 'run a separate instance of Zeek locally' use case isn't really a big enough use case to have a whole separate docker-compose file for it; especially with the ISO and live capture methods.
* ensure all services have a health check
* reduce verbosity of health checks in logs