Tags: ReedCStone/Malcolm
Tags
changes for Release 2.6.0 (new ICSNPP Zeek parsers) (cisagov#157)
Merge development for Malcolm 2.4.2 release (cisagov#156) Main changes: * Added code to allow periodic updates of [Yara](https://github.com/Neo23x0/signature-base) and [Capa](https://github.com/fireeye/capa-rules/) rules * Bump to [Arkime](https://arkime.com/) (Moloch up until recently) [2.7.1](https://github.com/arkime/arkime/blob/v2.7.1/CHANGELOG#L22-L30) and all possible related user-facing code/documentation changed * Bump kernel to 5.9.0 for ISO installer Individual commits: * restore web generation * update sha for malcolm iso * Allow multiple file scanners (VirusTotal, ClamAV, Malass) to be run instead of having to just select one. * working on idaholab#148, adding Yara as a file scanner * working on idaholab#148, some fixes to signatures dashboard/parsing * revert dashboard change from previous commit * working on idaholab#148, change file pipeline from push/pull to publish/subscribe (but still screwy for multiple engines) * working on idaholab#148, change file pipeline from push/pull to publish/subscribe (but still screwy for multiple engines) * working on idaholab#148, more work on allowing multiple file scanners to be able to run nicely * working on idaholab#148, adding Yara as a file scanner to the hedgehog iso * working on idaholab#148, adding Yara as a file scanner to the hedgehog iso * update pip3 python versions * fix configure-capture for multiple scanners * fix configure-capture for multiple scanners * fix configure-capture for multiple scanners * bump version for 2.3.0 * bump moloch to 2.4.0 * fix moloch 2.4.0 build * adjust footer for malcolm * fix footer * fix footer * readme updates * fix idaholab#150 by decreasing the value of COMPOSE_HTTP_TIMEOUT slightly * reduce image size for freq * remove old kernels when done with installs * remove old kernels when done with installs * update kbuild version for iso * added new environment variables for disabling certain zeek features to hedgehog * fix issue with zeek_init in my local.zeek not having priority set so it executes prior to the parser plugins being loaded * don't depend on scanner for file-monitor health check * update shas * provide more than just filename to scanners from carved file watcher (additionally send size and mime type) * add capa (https://github.com/fireeye/capa) binary to file-monitor docker image * working on idaholab#152, implement capa file scanner for EXE Files * working on idaholab#152, implement capa file scanner for EXE Files * working on idaholab#152, implement capa file scanner for EXE Files * improve Capa detection by logging mitre att&ck techniques by default * fix verbose capa results * improve Capa detection by logging mitre att&ck techniques by default, but allow original rules as well * update zeek to 3.0.10 (https://github.com/zeek/zeek/blob/v3.0.10/NEWS) * update version to 2.4.0 * fix bug idaholab#24, install.py won't prompt to change ownership of extracted directory correctly if run as root * fix kbuild install * update capa version * retain some build stuff * keep spicy build available in ISO * added detection for Zerologon (CVE-2020-1472) via corelight zeek script https://github.com/corelight/zerologon * this *should* fix the issues with the installer not working in BIOS. it should work in both bios and efi. fixes issue idaholab#26. testing in progress * added menu entries for bios installer * install capa from pip rather than binary * bump moloch to 2.4.1; https://github.com/aol/moloch/blob/v2.4.1/CHANGELOG#L21-L42 * fix url for zeek-EternalSafety github repo * fix issue where capa rules directory is not specified correctly * fix lgtm python alerts for unused variable and exception handling * fix issue running spicy where noexec is defined for /tmp; related somewhat to issue zeek/spicy#521 * fix ufw/iptables/docker networking issue * fix ufw/iptables/docker networking issue * fix ufw/iptables/docker networking issue (specify default docker address pool) * change documentation to reflect resolution of issue idaholab#26 * fix BIOS installer to use preseed correctly * in malcolm installer, make /var partition larger to give more room for docker images * tweak spicy HLTO TMP location so it doesn't try to use capture path on hedgehog ISO * make sure file doesn't get moved out from underneath capa scanner by making move logic more robust * make sure that capa-rules matches the capa git rules submodule revision * updated download shas * fix logo url * resolve issue idaholab#27, by capping max swap size at 12GB * resolve issue idaholab#27, by capping max swap size at 16GB * change ISO shas for DL of 2.4.0.1 * minor updates to generic slides * update version to 2.4.1 development * 2.4.1 version bumps: supercronic - 0.1.11 nginx (docker) - 1.19.3 bison (zeek build) - 3.7.2 cmake (zeek build) - 3.18.4 zeek - 3.0.11 kernel (ISOs) - 5.8 * stick with 5.7 kernel for now * stick with 5.7 kernel for now * update kernel to 5.8 * added Zeek plugin for corelight/CVE-2020-16898 * fix protologbeat build * fix protologbeat build * make sure we check for all of the plugins we're installing * added SHAs for 2.4.1 ISOs * fix some upgrade documentation * first pass at renaming moloch->arkime, should be visual/user-facing changes onlyu * fix build errors * update to Arkime 2.7.0 https://github.com/arkime/arkime/blob/v2.7.0/CHANGELOG#L22-L31 * remove logo customization * remove password change file * bump version to 2.4.2 for arkime update * fix moloch deb build * added update procedure for capa and yara rules * update Arkime to 2.7.1 https://github.com/arkime/arkime/blob/v2.7.1/CHANGELOG#L22-L30 * run update scripts explicitly with bas * add vagrant-reload plugin as a requirement in the documentation * update kernel
Malcolm v2.4.1 development (cisagov#155) - Version bumps - supercronic (for Docker images) [0.1.11](https://github.com/aptible/supercronic/releases/tag/v0.1.11) - nginx [1.19.3](https://nginx.org/en/CHANGES) - bison (for Zeek compile) 3.7.2 - cmake (for Zeek compile) 3.18.4 - Zeek [3.0.11](https://github.com/zeek/zeek/releases/tag/v3.0.11) - Moloch [2.4.1](https://github.com/aol/moloch/blob/v2.4.1/CHANGELOG#L21-L42) - Linux Kernel (for ISOs) 5.8.0 - Zeek - added [plugin](https://github.com/corelight/CVE-2020-16898) to detect "bad neighbor" (CVE-2020-16898)
Malcolm 2.4.0 release development (cisagov#154) Malcolm 2.4.0 release development Extracted file scanning added Capa as an optional extracted file scanner multiple file scanners can now be enabled Version updates updated Moloch to 2.4.1 updated Zeek to 3.0.10 updated Linux Kernel for ISO installers to 5.7 Zeek plugins added Corelight's Zerologon plugin to detect CVE-2020-1472 Tweaks and bug fixes Don't allow docker to mess with firewall rules in Malcolm ISO Fix idaholab#26, ISO installers result in blank screen when booting with BIOS Fix idaholab#24, install.py won't prompt to change ownership of extracted directory correctly if run as root Leave some development packages in place in Hedgehog ISO so that Spicy plugins can be compiled * update sha for malcolm iso * Allow multiple file scanners (VirusTotal, ClamAV, Malass) to be run instead of having to just select one. * working on idaholab#148, adding Yara as a file scanner * working on idaholab#148, some fixes to signatures dashboard/parsing * revert dashboard change from previous commit * working on idaholab#148, change file pipeline from push/pull to publish/subscribe (but still screwy for multiple engines) * working on idaholab#148, change file pipeline from push/pull to publish/subscribe (but still screwy for multiple engines) * working on idaholab#148, more work on allowing multiple file scanners to be able to run nicely * working on idaholab#148, adding Yara as a file scanner to the hedgehog iso * working on idaholab#148, adding Yara as a file scanner to the hedgehog iso * update pip3 python versions * fix configure-capture for multiple scanners * fix configure-capture for multiple scanners * fix configure-capture for multiple scanners * bump version for 2.3.0 * bump moloch to 2.4.0 * fix moloch 2.4.0 build * adjust footer for malcolm * fix footer * fix footer * readme updates * fix idaholab#150 by decreasing the value of COMPOSE_HTTP_TIMEOUT slightly * reduce image size for freq * remove old kernels when done with installs * remove old kernels when done with installs * update kbuild version for iso * added new environment variables for disabling certain zeek features to hedgehog * fix issue with zeek_init in my local.zeek not having priority set so it executes prior to the parser plugins being loaded * don't depend on scanner for file-monitor health check * update shas * provide more than just filename to scanners from carved file watcher (additionally send size and mime type) * add capa (https://github.com/fireeye/capa) binary to file-monitor docker image * working on idaholab#152, implement capa file scanner for EXE Files * working on idaholab#152, implement capa file scanner for EXE Files * working on idaholab#152, implement capa file scanner for EXE Files * improve Capa detection by logging mitre att&ck techniques by default * fix verbose capa results * improve Capa detection by logging mitre att&ck techniques by default, but allow original rules as well * update zeek to 3.0.10 (https://github.com/zeek/zeek/blob/v3.0.10/NEWS) * update version to 2.4.0 * fix bug idaholab#24, install.py won't prompt to change ownership of extracted directory correctly if run as root * fix kbuild install * update capa version * retain some build stuff * keep spicy build available in ISO * added detection for Zerologon (CVE-2020-1472) via corelight zeek script https://github.com/corelight/zerologon * this *should* fix the issues with the installer not working in BIOS. it should work in both bios and efi. fixes issue idaholab#26. testing in progress * added menu entries for bios installer * install capa from pip rather than binary * bump moloch to 2.4.1; https://github.com/aol/moloch/blob/v2.4.1/CHANGELOG#L21-L42 * fix url for zeek-EternalSafety github repo * fix lgtm python alerts for unused variable and exception handling * fix issue where capa rules directory is not specified correctly * fix issue where capa rules directory is not specified correctly * fix lgtm python alerts for unused variable and exception handling * fix issue running spicy where noexec is defined for /tmp; related somewhat to issue zeek/spicy#521 * fix issue running spicy where noexec is defined for /tmp * fix ufw/iptables/docker networking issue * fix ufw/iptables/docker networking issue * fix ufw/iptables/docker networking issue (specify default docker address pool) * fix ufw/iptables/docker networking issue (specify default docker address pool) * change documentation to reflect resolution of issue idaholab#26 * change documentation to reflect resolution of issue idaholab#26 * fix BIOS installer to use preseed correctly * fix BIOS installer to use preseed correctly * in malcolm installer, make /var partition larger to give more room for docker images * tweak spicy HLTO TMP location so it doesn't try to use capture path on hedgehog ISO * make sure file doesn't get moved out from underneath capa scanner by making move logic more robust * make sure that capa-rules matches the capa git rules submodule revision
v2.3.0 development (cisagov#151) * Carved file scanning improvements * Multiple file scanners can now be enabled concurrently (previously only one at a time was allowed) * [Yara](https://github.com/VirusTotal/yara) [added](idaholab#148) as carved file scanner feeding signatures.log with [Florian Roth](https://github.com/Neo23x0)'s [Signature-Base](https://github.com/Neo23x0/signature-base) Yara ruleset enabled by default and the ability to provide other yara signatures under `yara/rules` under the Malcolm directory (see cisagov#148 and cisagov#14) * Bumped versions * Moloch [v2.4.0](https://github.com/aol/moloch/blob/v2.4.0/CHANGELOG#L21-L42) * Bug fixes * cisagov#150 docker-compose having issues with start and logs under macOS * Hedgehog was missing new environment variables for finer control of Zeek local policy behavior * miscellaneous tweaks to Docker and ISO images (mainly for file size)
Malcolm v2.2.0 development (cisagov#145) * Zeek: - Update Zeek to [3.0.8](https://github.com/zeek/zeek/releases/tag/v3.0.8) - Include [Spicy](https://github.com/zeek/spicy) - Added ability to disable certain zeek features/parsers using environment variables - Added [Wireguard parser](https://github.com/theparanoids/spicy-noise) - Added a few Corelight plugins: + Corelight's [callstranger-detector](https://github.com/corelight/callstranger-detector) plugin + Corelight's [ripple20](https://github.com/corelight/ripple20) plugin + Corelight's [SIGred](https://github.com/corelight/SIGred) plugin * Logstash: - Added parsing for Zeek Wireguard (noise.log) - Initial work towards mapping Zeek log fields to Elastic Common Schema (see issue cisagov#79) + Disabled by default, can be enabled with `LOGSTASH_TO_ECS : 'true'` in `x-logstash-variables` in `docker-compose.yml` + not 100% complete. Good first effort, more will be done in the future - Some fixes to the JA3 signature mapping generation * ISOs - Updated Hedgehog and Malcolm ISOs to use 5.6 kernel - Get virtualbox guest VM debs from [unofficial backport](https://people.debian.org/~lucas/virtualbox-buster/) rather than building for VM installs * Documentation - Documentation, scripts, Vagrantfiles and sample configurations for using Beats to forward host logs to Malcolm
Merge topic/dockerperms to address issue cisagov#137 (cisagov#138) (c… …isagov#139) This pull request adds the some new environment variables for Malcolm to address cisagov#137 * `PUID` and `PGID` * Docker runs all of its containers as the privileged `root` user by default. For better security, Malcolm immediately drops to non-privileged user accounts for executing internal processes wherever possible. The `PUID` (**p**rocess **u**ser **ID**) and `PGID` (**p**rocess **g**roup **ID**) environment variables allow Malcolm to map internal non-privileged user accounts to a corresponding [user account](https://en.wikipedia.org/wiki/User_identifier) on the host. Additionally, this pull request additionally moves all remaining process that can be run non-privileged to run as non-privileged. Each docker container now has the following in its Dockerfile (this example is from the zeek container, they're all similar but may have different specific values): ``` ARG DEFAULT_UID=1000 ARG DEFAULT_GID=1000 ENV DEFAULT_UID $DEFAULT_UID ENV DEFAULT_GID $DEFAULT_GID ENV PUSER "zeek" ENV PGROUP "zeek" ENV PUSER_PRIV_DROP true ``` The entrypoint of each docker comtainer is now [docker-uid-gid-setup.sh](https://github.com/idaholab/Malcolm/blob/master/shared/bin/docker-uid-gid-setup.sh), which does the following: 1. changes the UID and GID of the default (1000:1000) user to match the PUID:PGID provided 2. finds any files *inside* the docker image owned by those IDs and chown them 3. if required, execs the container command by dropping privileges to the unprivileged user Additionally, control.py (used for start, restart, etc.) will now error out it run as root rather than just running with a bunch of errors. Malcolm should not be run as a root user.
PreviousNext