Henry Develop notes Zola 2024-12-02T00:00:00+00:00 https://yinheli.com/atom.xml shell 脚本中的 `trap` 命令 2024-12-02T00:00:00+00:00 2024-12-02T00:00:00+00:00 Unknown https://yinheli.com/blog/2024/trap/ <p>shell 脚本中有类似编程里的 defer, finally, register_shutdown_function 处理效果吗？有的，就是 <code>trap</code> 命令。</p> <h2 id="1-shen-me-shi-trap">1. 什么是 <code>trap</code>？</h2> <p><code>trap</code> 是 Shell 提供的一个内置命令，用于捕获 UNIX 信号（Signals）或者特殊的运行时事件，并在触发时执行特定命令。它的基本用法如下：</p> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>trap </span><span style="color:#d69d85;">&#39;commands&#39;</span><span> signals </span></code></pre> <ul> <li><strong><code>commands</code></strong>：在捕获指定信号或事件时执行的一段命令。</li> <li><strong><code>signals</code></strong>：触发 <code>commands</code> 执行的信号列表，可以是信号名称（如 <code>EXIT</code> 或 <code>INT</code>）或信号编号（如 <code>15</code> 或 <code>2</code>）。</li> </ul> <p>通过 <code>trap</code> 命令，我们可以在脚本运行的期间甚至退出时，优雅地完成清理、日志记录以及错误处理等任务。</p> <hr /> <h2 id="2-chang-jian-xin-hao-yu-zuo-yong">2. 常见信号与作用</h2> <p>在 UNIX 系统中，信号是用于与进程通信的机制。下表列出了常见信号以及它们的作用：</p> <table><thead><tr><th>信号名称</th><th>信号编号</th><th>描述</th><th>触发方式</th></tr></thead><tbody> <tr><td><code>SIGHUP</code></td><td><code>1</code></td><td>挂起信号，通常在终端关闭时发送。</td><td>关闭 SSH 会话或者挂起终端</td></tr> <tr><td><code>SIGINT</code></td><td><code>2</code></td><td>中断信号，一般由用户按下 <code>Ctrl+C</code> 产生。</td><td>用户主动中断</td></tr> <tr><td><code>SIGQUIT</code></td><td><code>3</code></td><td>退出信号，通常由用户按下 <code>Ctrl+\</code> 产生，用于生成调试 core dump。</td><td>强制退出</td></tr> <tr><td><code>SIGTERM</code></td><td><code>15</code></td><td>终止信号，通常通过 <code>kill</code> 命令发送，表示希望优雅终止。</td><td>执行 <code>kill PID</code></td></tr> <tr><td><code>SIGKILL</code></td><td><code>9</code></td><td>强制终止信号，无法被捕获，是一个直接杀死进程的信号。</td><td>执行 <code>kill -9 PID</code></td></tr> <tr><td><code>SIGSTOP</code></td><td><code>19</code></td><td>暂停信号，无法被捕获或忽略，用于暂停进程。</td><td>暂停的快捷方式</td></tr> <tr><td><code>EXIT</code></td><td>-</td><td>在脚本退出时触发（无论正常还是错误）。</td><td>脚本执行完成或者使用 <code>exit</code> 语句</td></tr> </tbody></table> <blockquote> <p>你可以通过 <code>kill -l</code> 查看当前系统支持的完整信号列表。</p> </blockquote> <hr /> <h2 id="3-ji-ben-yong-fa">3. 基本用法</h2> <h3 id="shi-li-1-qing-li-lin-shi-wen-jian">示例 1：清理临时文件</h3> <p>当脚本中需要执行一些中间操作时，可能会产生临时文件。如果用户中途中断脚本，这些临时文件可能会遗留在文件系统中。例如：</p> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;">#!/bin/bash </span><span> </span><span>temp_file=</span><span style="background-color:#282828;color:#d69d85;">&quot;/tmp/my_temp_file&quot;</span><span> </span><span>trap </span><span style="color:#d69d85;">&quot;rm -f $</span><span>temp_file</span><span style="color:#d69d85;">; echo &#39;临时文件已清理&#39;&quot;</span><span> EXIT </span><span> </span><span style="color:#608b4e;"># 模拟创建临时文件并操作 </span><span>touch $temp_file </span><span>echo </span><span style="color:#d69d85;">&quot;处理中...&quot; </span><span>sleep 5 </span><span> </span><span style="color:#608b4e;"># 脚本结束后，自动触发清理 </span><span>echo </span><span style="color:#d69d85;">&quot;执行完成。&quot; </span></code></pre> <p>运行后，无论脚本是成功执行还是中途退出（如被 <code>Ctrl+C</code> 中断），<code>trap</code> 都会捕获 <code>EXIT</code> 信号，执行清理命令 <code>rm -f $temp_file</code>。</p> <hr /> <h3 id="shi-li-2-bu-huo-yong-hu-zhong-duan-xin-hao-sigint">示例 2：捕获用户中断信号 (<code>SIGINT</code>)</h3> <p>有时候我们希望让用户按下 <code>Ctrl+C</code> 时，脚本能优雅退出，而非直接中止。例如：</p> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;">#!/bin/bash </span><span> </span><span>trap </span><span style="color:#d69d85;">&quot;echo &#39;优雅退出中...&#39;; cleanup; exit&quot;</span><span> SIGINT </span><span> </span><span>cleanup() { </span><span> echo </span><span style="color:#d69d85;">&quot;正在清理资源...&quot; </span><span>} </span><span> </span><span>echo </span><span style="color:#d69d85;">&quot;运行脚本，按 Ctrl+C 中断。&quot; </span><span style="color:#569cd6;">while </span><span>true</span><span style="color:#569cd6;">; do </span><span> sleep 1 </span><span style="color:#569cd6;">done </span></code></pre> <p>在脚本中，用户按下 <code>Ctrl+C</code> 会发送 <code>SIGINT</code> 信号。通过 <code>trap</code>，可以拦截该信号并优雅退出，同时完成清理操作。</p> <hr /> <h3 id="shi-li-3-cuo-wu-chu-li-err-xin-hao">示例 3：错误处理 (<code>ERR</code> 信号)</h3> <p>在 Shell 脚本中，如果某个命令失败，往往需要及时处理或终止脚本以避免影响后续操作。通过 <code>trap</code> 捕获 <code>ERR</code> 信号，可以实现全局错误处理：</p> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;">#!/bin/bash </span><span>set -e </span><span style="color:#608b4e;"># 任何命令失败时立即退出 </span><span>trap </span><span style="color:#d69d85;">&quot;echo &#39;发生错误，终止脚本&#39;; cleanup; exit 1&quot;</span><span> ERR </span><span> </span><span>cleanup() { </span><span> echo </span><span style="color:#d69d85;">&quot;清理完成。&quot; </span><span>} </span><span> </span><span style="color:#608b4e;"># 模拟正常命令 </span><span>echo </span><span style="color:#d69d85;">&quot;开始执行脚本...&quot; </span><span>ls /nonexistent_folder </span><span style="color:#608b4e;"># 强制触发错误 </span><span>echo </span><span style="color:#d69d85;">&quot;这条命令不会被执行。&quot; </span></code></pre> <p>当某条命令（如 <code>ls /nonexistent_folder</code>）出错时，会触发 <code>ERR</code> 信号，调用 <code>cleanup</code> 函数清理资源并退出脚本。</p> <hr /> <h2 id="4-gao-ji-yong-fa">4. 高级用法</h2> <h3 id="bu-huo-duo-ge-xin-hao">捕获多个信号</h3> <p>你可以通过 <code>trap</code> 同时捕获多个信号，并为它们定义相同的行为：</p> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>trap </span><span style="color:#d69d85;">&quot;echo &#39;收到信号，正在优雅退出...&#39;; exit&quot;</span><span> SIGINT SIGTERM </span></code></pre> <p>无论是按下 <code>Ctrl+C</code>（触发 <code>SIGINT</code>）还是通过 <code>kill</code> 命令（触发 <code>SIGTERM</code>），都会执行同样的命令。</p> <hr /> <h3 id="hu-lue-mou-xie-xin-hao">忽略某些信号</h3> <p>如果希望某些信号不影响脚本运行，可以让 <code>trap</code> 执行空操作：</p> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>trap </span><span style="color:#d69d85;">&quot;&quot;</span><span> SIGINT SIGTERM </span><span>echo </span><span style="color:#d69d85;">&quot;信号被忽略，按 Ctrl+C 无法中断脚本。&quot; </span><span style="color:#569cd6;">while </span><span>true</span><span style="color:#569cd6;">; do </span><span> sleep 1 </span><span style="color:#569cd6;">done </span></code></pre> <p>需要注意的是，部分信号（例如 <code>SIGKILL</code> 和 <code>SIGSTOP</code>）无法被忽略或捕获。</p> <hr /> <h3 id="diao-shi-jiao-ben-liu-debug-xin-hao">调试脚本流（<code>DEBUG</code> 信号）</h3> <p><code>trap</code> 的 <code>DEBUG</code> 信号使你可以在脚本执行每条命令之前插入调试信息，非常适用于调试复杂脚本。</p> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>trap </span><span style="color:#d69d85;">&#39;echo &quot;正在执行命令：$BASH_COMMAND&quot;&#39;</span><span> DEBUG </span><span> </span><span>echo </span><span style="color:#d69d85;">&quot;执行第一步&quot; </span><span>ls /nonexistent_file </span><span style="color:#608b4e;"># 触发错误 </span><span>echo </span><span style="color:#d69d85;">&quot;执行下一步&quot; </span></code></pre> <p>当脚本运行时，每条命令（包括未定义的或出错的命令）都会在执行前打印到屏幕上，便于跟踪脚本行为。</p> <hr /> <h3 id="bu-huo-han-shu-de-fan-hui-zhuang-tai-return-xin-hao">捕获函数的返回状态（<code>RETURN</code> 信号）</h3> <p>当 Shell 函数返回时，可以通过捕获 <code>RETURN</code> 信号执行额外逻辑，例如记录函数的返回值或完成特定清理操作。</p> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>trap </span><span style="color:#d69d85;">&#39;echo &quot;函数返回了。&quot;&#39;</span><span> RETURN </span><span> </span><span>test_function() { </span><span> echo </span><span style="color:#d69d85;">&quot;正在执行函数任务...&quot; </span><span>} </span><span>test_function </span></code></pre> <hr /> <h2 id="5-qian-zai-wen-ti-yu-zhu-yi-shi-xiang">5. 潜在问题与注意事项</h2> <ol> <li> <p><strong>部分信号无法捕获</strong>：<code>SIGKILL</code> 和 <code>SIGSTOP</code> 是特殊信号，无法被 <code>trap</code> 捕获或阻止。这是 UNIX 系统的设计策略，无法绕过。</p> </li> <li> <p><strong>并发信号处理</strong>：当多个信号几乎同时触发时，Shell 可能会按顺序依次处理它们，但这可能引发意外行为，建议测试关键脚本。</p> </li> <li> <p><strong>复杂命令需注意转义</strong>：<code>trap</code> 中的命令如果包含 <code>$</code>、<code>&quot;</code> 等特殊字符，可能需要正确转义。例如：</p> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>trap </span><span style="color:#d69d85;">&#39;echo &quot;命令失败: $BASH_COMMAND&quot;&#39;</span><span> ERR </span></code></pre> </li> </ol> <hr /> <h2 id="6-zong-jie">6. 总结</h2> <p><code>trap</code> 是 Shell 脚本中功能强大且不可或缺的工具。无论是资源清理、安全退出、错误处理还是调试跟踪，它都能为我们的脚本带来极大的灵活性与可靠性。</p> <p>通过结合信号特性与实际需求，合理设计脚本中的信号处理逻辑，可以避免意外问题的发生，提升脚本的健壮性和安全性。</p> web3 几个安全相关名词 2024-06-19T00:00:00+00:00 2024-06-19T00:00:00+00:00 Unknown https://yinheli.com/blog/2024/web3-security-sss-tss-mpc-vss/ <h1 id="shamir-s-secret-sharing-sss">Shamir's secret sharing (SSS)</h1> <p>是一种基于多项式插值的秘密共享方案。它将秘密信息分割成多个部分，每个部分称为“秘密份额”。只有收集到足够数量的秘密份额，才能使用多项式插值方法还原原始秘密。</p> <p><strong>主要特点:</strong></p> <ul> <li><strong>阈值方案:</strong> 需要达到一定的阈值数量（k）的秘密份额才能恢复秘密。</li> <li><strong>灵活:</strong> 可以根据需要调整阈值数量，以控制恢复秘密的难易程度。</li> <li><strong>安全:</strong> 即使部分秘密份额丢失或泄露，也不会影响秘密的安全性。</li> </ul> <p><strong>区别:</strong></p> <ul> <li>SSS 主要用于秘密信息的共享和恢复，重点在于保护秘密本身。</li> <li>SSS 不涉及签名或验证的功能。</li> </ul> <h1 id="threshold-signature-schemes-tss">Threshold Signature Schemes (TSS)</h1> <p> 是一种分布式签名方案，它将签名权力分散到多个参与者中，需要满足一定的阈值条件才能生成有效的签名。</p> <p><strong>主要特点:</strong></p> <ul> <li><strong>分布式签名:</strong> 签名权力分散在多个参与者中，而不是集中在单个实体。</li> <li><strong>阈值要求:</strong> 需要达到一定的阈值数量的参与者才能生成有效的签名。</li> <li><strong>安全:</strong> 防止单个参与者控制签名权力，提高了签名方案的安全性。</li> </ul> <p><strong>区别:</strong></p> <ul> <li>TSS 主要用于生成和验证签名，而不是秘密共享。</li> <li>TSS 的重点在于控制签名权力，而不是保护秘密本身。</li> </ul> <h1 id="multi-party-computation-mpc">Multi-Party Computation (MPC)</h1> <p>是一种允许多个参与者合作进行计算，而无需向彼此公开其私有输入信息的协议。</p> <p><strong>主要特点:</strong></p> <ul> <li><strong>隐私保护:</strong> 参与者可以进行联合计算，同时保护其私有输入信息的隐私。</li> <li><strong>安全:</strong> 即使一些参与者恶意行为，也不会影响计算结果的正确性。</li> <li><strong>广泛应用:</strong> 应用于多个领域，包括加密货币、隐私保护计算、安全多方计算等。</li> </ul> <p><strong>区别:</strong></p> <ul> <li>MPC 的重点在于进行安全的联合计算，而不是秘密共享或签名。</li> <li>MPC 适用于各种类型的计算，而不仅仅是秘密共享或签名。</li> </ul> <h1 id="multi-signature-multisig">Multi-Signature (Multisig)</h1> <p>是一种需要多个签名才能完成交易的机制。它通常用于提高交易安全性，防止单个签名者的恶意行为。</p> <p><strong>主要特点:</strong></p> <ul> <li><strong>多重签名要求:</strong> 需要多个签名者签署才能完成交易。</li> <li><strong>安全:</strong> 提高了交易安全性，防止单个签名者控制交易。</li> <li><strong>应用:</strong> 常用于加密货币交易、数字资产管理等领域。</li> </ul> <p><strong>区别:</strong></p> <ul> <li>Multisig 主要用于控制交易执行，而不是秘密共享或签名。</li> <li>Multisig 的重点在于防止单个参与者控制交易，而不是保护秘密本身。</li> </ul> <h1 id="verifiable-secret-sharing-vss">Verifiable Secret Sharing (VSS)</h1> <p>是一种结合了秘密共享和可验证性特性的方案。它允许参与者验证他们收到的秘密份额是否正确，并确保秘密不会被恶意参与者篡改。</p> <p><strong>主要特点:</strong></p> <ul> <li><strong>可验证性:</strong> 参与者可以验证他们收到的秘密份额是否正确。</li> <li><strong>容错性:</strong> 即使部分参与者恶意行为，仍然可以恢复原始秘密。</li> <li><strong>应用:</strong> 常用于安全多方计算、分布式共识等领域。</li> </ul> <p><strong>区别:</strong></p> <ul> <li>VSS 既包含秘密共享功能，也包含可验证性功能，两者相结合。</li> <li>VSS 主要用于提高秘密共享方案的安全性，防止恶意行为。</li> </ul> <h1 id="reference">Reference</h1> <ul> <li>https://en.wikipedia.org/wiki/Shamir%27s_secret_sharing</li> <li>https://cryptonews.net/news/other/29241788/</li> <li>https://en.wikipedia.org/wiki/Threshold_cryptosystem</li> </ul> ucloud CDN 日志解析 2024-04-20T00:00:00+00:00 2024-04-20T00:00:00+00:00 Unknown https://yinheli.com/blog/2024/ucloud-cdn-log-parser/ <p>前几天遇到 ucloud CDN 服务的带宽报警，而他们的后台对流量分析这块太弱了，于是看了下文档对照写了个<a href="https://github.com/yinheli/ucloud-cdn-log-parser">日志解析</a>，主要是练手 rust，又有一年多没有做 rust 的项目了。</p> <p>这个工具只是简单的用正则对日志文件做了解析，具体的分析需要再配合其他工具处理，我这里比较推荐使用 duckdb 或者 clickhouse (local)。将数据解析转换为 parquet 后压缩率很高。</p> <p>顺便发现 duckdb 的 csv/tsv 数据导入 parquet 比 clickhouse 支持的更好，感觉数据类型的推理更智能。</p> <p>具体使用姿势参考项目的 <a href="https://github.com/yinheli/ucloud-cdn-log-parser?tab=readme-ov-file#usage">readme</a> 。</p> 用抢占式节点做视频转码 2024-04-01T00:00:00+00:00 2024-04-01T00:00:00+00:00 Unknown https://yinheli.com/blog/2024/transcode-with-spot/ <p>最近有一批阿里云OSS存储的视频要做视频转码，量比较大，有2千多万个视频视频文件，平均时长 17s 左右。我们需要转出多个版本 &amp; 水印这样转码时长就成倍增加，如果用阿里云的转码服务，叠加折扣券，预算下来要40多万，最后自己写了个服务，用 ACK + 抢占式 ECI，不到总花费不到一万，省了一大笔。</p> <p>推荐 ecs.hfc6.large 机型，起初用的 ecs.c6.large 后来发现 hfc6 不仅有 5%~7%左右性能提升，价格却还能节省 62% ，当然抢占式机型的价格是浮动的，不过短期内浮动不是很大。</p> postgresql 大版本升级 2023-12-22T00:00:00+00:00 2023-12-22T00:00:00+00:00 Unknown https://yinheli.com/blog/2023/postgresql-upgrade/ <p>gitea 的 helm 升级连带升级了 postgresql , postgresql 的大版本升级比较麻烦，这里记录下。</p> <ul> <li>首先把老的版本数据文件夹备份</li> <li>老的 bin 文件夹备份</li> <li>然后通过 <code>pg_upgrade</code> 来升级</li> <li>拷贝一个老的或者新的 pod 的 yaml 配置，并且设置为 root 用户启动，command 改为 <code>tail -f /dev/null</code></li> </ul> <p>以上都做完，可以停掉老的服务，用以上备份的 k8s pod yaml 改为新版本镜像，挂载老的数据盘，启动一个实例</p> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;"># 创建 postgres 用户 </span><span>adduser --uid 1001 postgres </span><span> </span><span style="color:#608b4e;"># 老版本bin /bitnami/postgresql/oldbin </span><span style="color:#608b4e;"># 老版本data /bitnami/postgresql/olddata </span><span> </span><span>mkdir /bitnami/postgresql/data </span><span> </span><span>su postgres </span><span> </span><span style="color:#608b4e;"># 初始化 </span><span>initdb -E UTF8 -U postgres -D /bitnami/postgresql/data </span><span> </span><span style="color:#608b4e;"># 升级 </span><span>pg_upgrade -b /bitnami/postgresql/oldbin -B /opt/bitnami/postgresql/bin -d /bitnami/postgresql/olddata -D /bitnami/postgresql/data </span></code></pre> <p>如果此前的老版本没有正常退出，报错</p> <pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>The source cluster was not shut down cleanly </span></code></pre> <p>可以用老的 bin 重新启动关闭一下，确保它是正常的退出，然后在尝试执行上面的升级操作</p> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>/bitnami/postgresql/oldbin/pg_ctl start -w -D /bitnami/postgresql/olddata </span><span>/bitnami/postgresql/oldbin/pg_ctl stop -s -m fast -D /bitnami/postgresql/olddata </span></code></pre> <h2 id="can-kao-zi-liao">参考资料</h2> <ul> <li>https://github.com/bitnami/charts/issues/8025</li> </ul> 本地运行 loki 查询 2023-12-12T00:00:00+00:00 2023-12-12T00:00:00+00:00 Unknown https://yinheli.com/blog/2023/loki-local/ <p>loki 的 <a href="https://grafana.com/docs/loki/latest/query/">LogQL</a> 使用体验还是很好的，一方面能过滤查询，另一方面还能看到日志的趋势。</p> <p>我的场景是要临时分析 CDN 的某个域名的日志</p> <h2 id="zhun-bei-docker-compose-yun-xing-huan-jing">准备 docker compose 运行环境</h2> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>vi loki.yaml </span></code></pre> <pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">auth_enabled</span><span>: </span><span style="color:#569cd6;">false </span><span> </span><span style="background-color:#282828;color:#569cd6;">server</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">http_listen_port</span><span>: </span><span style="color:#b5cea8;">3100 </span><span> </span><span style="background-color:#282828;color:#569cd6;">grpc_server_max_recv_msg_size</span><span>: </span><span style="color:#b5cea8;">41943040 </span><span> </span><span style="background-color:#282828;color:#569cd6;">grpc_server_max_send_msg_size</span><span>: </span><span style="color:#b5cea8;">41943040 </span><span> </span><span style="background-color:#282828;color:#569cd6;">common</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">path_prefix</span><span>: </span><span style="background-color:#282828;color:#d69d85;">/loki</span><span> </span><span> </span><span style="background-color:#282828;color:#569cd6;">storage</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">filesystem</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">chunks_directory</span><span>: </span><span style="background-color:#282828;color:#d69d85;">/loki/chunks</span><span> </span><span> </span><span style="background-color:#282828;color:#569cd6;">rules_directory</span><span>: </span><span style="background-color:#282828;color:#d69d85;">/loki/rules</span><span> </span><span> </span><span style="background-color:#282828;color:#569cd6;">replication_factor</span><span>: </span><span style="color:#b5cea8;">1 </span><span> </span><span style="background-color:#282828;color:#569cd6;">ring</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">kvstore</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">store</span><span>: </span><span style="background-color:#282828;color:#d69d85;">inmemory</span><span> </span><span> </span><span style="background-color:#282828;color:#569cd6;">schema_config</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">configs</span><span>: </span><span> - </span><span style="background-color:#282828;color:#569cd6;">from</span><span>: </span><span style="color:#b4cea8;">2020-10-24 </span><span> </span><span style="background-color:#282828;color:#569cd6;">store</span><span>: </span><span style="background-color:#282828;color:#d69d85;">boltdb-shipper</span><span> </span><span> </span><span style="background-color:#282828;color:#569cd6;">object_store</span><span>: </span><span style="background-color:#282828;color:#d69d85;">filesystem</span><span> </span><span> </span><span style="background-color:#282828;color:#569cd6;">schema</span><span>: </span><span style="background-color:#282828;color:#d69d85;">v11</span><span> </span><span> </span><span style="background-color:#282828;color:#569cd6;">index</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">prefix</span><span>: </span><span style="background-color:#282828;color:#d69d85;">index_</span><span> </span><span> </span><span style="background-color:#282828;color:#569cd6;">period</span><span>: </span><span style="background-color:#282828;color:#d69d85;">24h</span><span> </span><span style="background-color:#282828;color:#569cd6;">limits_config</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">reject_old_samples</span><span>: </span><span style="color:#569cd6;">false </span><span> </span><span style="background-color:#282828;color:#569cd6;">ingestion_rate_mb</span><span>: </span><span style="color:#b5cea8;">1024 </span><span> </span><span style="background-color:#282828;color:#569cd6;">per_stream_rate_limit</span><span>: </span><span style="background-color:#282828;color:#d69d85;">512MB</span><span> </span></code></pre> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>vi docker-compose.yaml </span></code></pre> <pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">version</span><span>: </span><span style="color:#d69d85;">&quot;3&quot; </span><span> </span><span style="background-color:#282828;color:#569cd6;">services</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">loki</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">image</span><span>: </span><span style="background-color:#282828;color:#d69d85;">docker.mirrors.sjtug.sjtu.edu.cn/grafana/loki</span><span> </span><span> </span><span style="background-color:#282828;color:#569cd6;">user</span><span>: </span><span style="background-color:#282828;color:#d69d85;">root</span><span> </span><span> </span><span style="background-color:#282828;color:#569cd6;">volumes</span><span>: </span><span> - </span><span style="background-color:#282828;color:#d69d85;">/data/loki:/loki</span><span> </span><span> - </span><span style="background-color:#282828;color:#d69d85;">./loki.yaml:/etc/loki/local-config.yaml</span><span> </span><span> </span><span style="background-color:#282828;color:#569cd6;">ports</span><span>: </span><span> - </span><span style="color:#d69d85;">&quot;3100:3100&quot; </span><span> </span><span> </span><span style="background-color:#282828;color:#569cd6;">grafana</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">image</span><span>: </span><span style="background-color:#282828;color:#d69d85;">docker.mirrors.sjtug.sjtu.edu.cn/grafana/grafana</span><span> </span><span> </span><span style="background-color:#282828;color:#569cd6;">ports</span><span>: </span><span> - </span><span style="color:#d69d85;">&quot;3000:3000&quot; </span></code></pre> <h2 id="dao-ru-ri-zhi">导入日志</h2> <p>可以用 promtail 或者如果数据有自己的预处理需求，就直接写个脚本，我这里写的 python 脚本来处理</p> <pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#9b9b9b;">import </span><span>gzip </span><span style="color:#9b9b9b;">import </span><span>json </span><span style="color:#9b9b9b;">import </span><span>os </span><span> </span><span style="color:#9b9b9b;">import </span><span>httpx </span><span> </span><span style="color:#608b4e;"># https://console.bce.baidu.com/cdn/#/cdn/customlog </span><span>fields = </span><span style="color:#d69d85;">&#39;remote_addr remote_ip remote_port jvip time_local scheme host format_time request status bytes_sent body_bytes_sent sent_http_content_length http_content_length request_time request_time_ms connection udf_hit http_range http_referer http_cookie http_user_agent http_x_forwarded_for ssl_protocol sent_http_content_type sent_http_content_range server_protocol http_accept log_level req_len url request_method uri http_ver log_timestamp_ms log_timestamp_s&#39;</span><span>.split(</span><span style="color:#d69d85;">&#39; &#39;</span><span>) </span><span> </span><span style="color:#569cd6;">def </span><span>iter_log(batch=</span><span style="color:#b5cea8;">1000</span><span>): </span><span> items = [] </span><span> </span><span style="color:#608b4e;"># list dir gzip iter all gz files </span><span> </span><span style="color:#569cd6;">for </span><span>file </span><span style="color:#569cd6;">in </span><span>sorted(os.listdir(</span><span style="color:#d69d85;">&#39;.&#39;</span><span>)): </span><span> </span><span style="color:#569cd6;">if </span><span>file.endswith(</span><span style="color:#d69d85;">&#39;.gz&#39;</span><span>): </span><span> </span><span style="color:#569cd6;">with </span><span>gzip.open(file, </span><span style="color:#d69d85;">&#39;rt&#39;</span><span>) </span><span style="color:#569cd6;">as </span><span>f: </span><span> </span><span style="color:#569cd6;">for </span><span>line </span><span style="color:#569cd6;">in </span><span>f: </span><span> line = line.strip() </span><span> </span><span style="color:#569cd6;">if not </span><span>line: </span><span> </span><span style="color:#569cd6;">continue </span><span> line = line.split(</span><span style="color:#d69d85;">&#39;</span><span style="color:#e3bbab;">\t</span><span style="color:#d69d85;">&#39;</span><span>) </span><span> </span><span style="color:#569cd6;">if </span><span>len(line) != len(fields): </span><span> </span><span style="color:#569cd6;">continue </span><span> item = dict(zip(fields, line)) </span><span> items.append([str(int(item[</span><span style="color:#d69d85;">&#39;log_timestamp_ms&#39;</span><span>]) * </span><span style="color:#b5cea8;">1000000</span><span>), json.dumps(item, ensure_ascii=</span><span style="color:#569cd6;">False</span><span>)]) </span><span> </span><span style="color:#569cd6;">if </span><span>len(items) == batch: </span><span> </span><span style="color:#569cd6;">yield </span><span>items </span><span> items = [] </span><span> </span><span style="color:#569cd6;">if </span><span>items: </span><span> </span><span style="color:#569cd6;">yield </span><span>items </span><span> </span><span> </span><span style="color:#608b4e;"># https://grafana.com/docs/loki/latest/reference/api/ </span><span style="color:#569cd6;">for </span><span>items </span><span style="color:#569cd6;">in </span><span>iter_log(batch=</span><span style="color:#b5cea8;">1000</span><span>): </span><span> data = { </span><span> </span><span style="color:#d69d85;">&quot;streams&quot;</span><span>: [ </span><span> { </span><span> </span><span style="color:#d69d85;">&quot;stream&quot;</span><span>: { </span><span> </span><span style="color:#d69d85;">&quot;cdn&quot;</span><span>: </span><span style="color:#d69d85;">&quot;baidu&quot; </span><span> }, </span><span> </span><span style="color:#d69d85;">&quot;values&quot;</span><span>: items </span><span> } </span><span> ] </span><span> } </span><span> r = httpx.post(</span><span style="color:#d69d85;">&#39;http://127.0.0.1:3100/loki/api/v1/push&#39;</span><span>, json=data) </span><span> </span><span style="color:#569cd6;">if </span><span>r.status_code != </span><span style="color:#b5cea8;">204</span><span>: </span><span> print(r.status_code, r.text) </span></code></pre> pytorch 镜像 2023-08-08T00:00:00+00:00 2023-08-08T00:00:00+00:00 Unknown https://yinheli.com/blog/2023/pytorch-mirror/ <p>pytorch 仅有一个 wheels 下载站 https://download.pytorch.org/whl ,当然 pip 是可以识别 <code>find-links</code> 类型的，但是在国内的网络环境下，下载却非常慢，尝试用上海交大的镜像，能临时解决问题，不过我们要在项目中构建 Docker 镜像，还是用免费的 nexus 私服构建代理比较合适。</p> <p>而 nexus 私服的 pypi 只支持代理 pypi 的上游，不支持 <code>find-links</code> 规范，故曲线救国，先手动把 whl 包下载下来，再用 <code>twine</code> 上传/发布到私服。</p> <p>先到华中科大的<a href="https://mirror.sjtu.edu.cn/pytorch-wheels/cu118/?mirror_intel_list">镜像站</a>下载需要的 whl 包 , 留意下载地址的那个参数，必传否则跳转到官方，可能机器学习大热华中科大的流量也是比较惊人吧。</p> <ol> <li>在浏览器的 console 执行下面脚本，拷贝所有 linux 的发行包（按照你的需要调整） ```</li> </ol> <pre data-lang="js" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-js "><code class="language-js" data-lang="js"><span>copy($$(</span><span style="color:#d69d85;">&#39;a&#39;</span><span>).map(it</span><span style="color:#569cd6;">=&gt;</span><span>it.href).filter(it</span><span style="color:#569cd6;">=&gt;</span><span>it.indexOf(</span><span style="color:#d69d85;">&#39;linux&#39;</span><span>)</span><span style="color:#569cd6;">&gt;</span><span style="color:#b5cea8;">0</span><span>).join(</span><span style="color:#d69d85;">&#39;</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">&#39;</span><span>)) </span></code></pre> <ol start="2"> <li>配置 <code>~/.pypirc</code></li> </ol> <pre data-lang="ini" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-ini "><code class="language-ini" data-lang="ini"><span style="color:#569cd6;">[distutils] </span><span>index-servers </span><span style="color:#569cd6;">= </span><span> nexus-torch-cu118 </span><span> </span><span style="color:#569cd6;">[nexus-torch-cu118] </span><span>repository </span><span style="color:#569cd6;">= </span><span style="color:#b4cea8;">https://nexus-server/repository/pytorch-cu118</span><span style="color:#569cd6;">/ </span><span>username </span><span style="color:#569cd6;">=</span><span> user </span><span>password </span><span style="color:#569cd6;">=</span><span> pass </span></code></pre> <ol start="3"> <li>安装好 <code>twine</code> ，挨个包执行上传/发布</li> </ol> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>twine upload --repository nexus-torch-cu118 torch-2.0.0+cu118-cp39-cp39-linux_x86_64.whl </span></code></pre> <ol start="4"> <li>记得在 pypi group 里加上发布的那个 self-host 仓库</li> </ol> nginx url decode issue 2023-02-07T00:00:00+00:00 2023-02-07T00:00:00+00:00 Unknown https://yinheli.com/blog/2023/nginx-url-decode-issue/ <p>公司的 bitbucket 已使用多年，最近计划升级，主要期望解决 nextjs 项目中文件名带有括号时 bitbucket 无法在 web 端查看代码的问题，发现升级完成后问题依旧，而相同版本，本地实验环境则没有发现这个问题，突然想到是不是中间代理层的问题，直连后 bitbucket (tomcat) 服务后发现果然没有这个问题，没想到困扰了这么久的问题居然是 nginx 导致的……</p> <p>尝试了多种方式，包括 rewrite 等，<code>proxy_pass</code> 依旧转发的内容会先做 url decode，这导致了后端server无法处理，最后本想用 lua 来试试，不过代理网关那边是 <code>apt</code> 直接安装的 nginx，改为 openresty 改动还有点大，遂试试 apt 安装 <a href="https://nginx.org/en/docs/njs/install.html"><code>nginx-module-njs</code></a>.</p> <p>修改 nginx.conf, 添加 <code>ngx_http_js_module</code> 加载</p> <pre data-lang="conf" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-conf "><code class="language-conf" data-lang="conf"><span>load_module </span><span style="color:#569cd6;">/</span><span>usr</span><span style="color:#569cd6;">/</span><span>lib</span><span style="color:#569cd6;">/</span><span>nginx</span><span style="color:#569cd6;">/</span><span>modules</span><span style="color:#569cd6;">/</span><span>ngx_http_js_module</span><span style="color:#569cd6;">.</span><span>so; </span></code></pre> <p>修改有关 bitbucket 段的反向代理配置，导入 git.js，用 auth_request 每次请求都执行 <code>/urlencode</code> 块 </p> <pre data-lang="conf" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-conf "><code class="language-conf" data-lang="conf"><span>js_import </span><span style="color:#569cd6;">/</span><span>etc</span><span style="color:#569cd6;">/</span><span>nginx</span><span style="color:#569cd6;">/</span><span>conf</span><span style="color:#569cd6;">.</span><span>d/git.js; </span><span>js_set $plain_uri </span><span style="color:#d69d85;">&#39;empty&#39;</span><span>; </span><span> </span><span style="color:#569cd6;">location ^~ /</span><span> { </span><span> </span><span style="color:#608b4e;"># ... </span><span> auth_request </span><span style="color:#569cd6;">/</span><span>urlencode; </span><span> proxy_pass </span><span style="color:#b4cea8;">http://10.180.9.229:7990$plain_uri</span><span>; </span><span>} </span><span> </span><span style="color:#569cd6;">location /</span><span>urlencode { </span><span style="color:#569cd6;"> internal</span><span>; </span><span> js_content git</span><span style="color:#569cd6;">.</span><span>urlencode; </span><span>} </span></code></pre> <p>其中 <code>git.js</code> 如下</p> <blockquote> <p>这里没有用 <code>replaceAll</code> 是因为在 <code>njs</code> 中并不支持</p> </blockquote> <pre data-lang="js" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-js "><code class="language-js" data-lang="js"><span style="color:#569cd6;">function </span><span>urlencode(r){ </span><span> </span><span style="color:#569cd6;">let </span><span>uri = r.variables.request_uri; </span><span> r.variables.plain_uri = uri.replace(</span><span style="color:#d69d85;">/</span><span style="color:#e3bbab;">\[</span><span style="color:#d69d85;">/</span><span style="color:#569cd6;">g</span><span>, </span><span style="color:#d69d85;">&#39;%5B&#39;</span><span>).replace(</span><span style="color:#d69d85;">/</span><span style="color:#e3bbab;">\]</span><span style="color:#d69d85;">/</span><span style="color:#569cd6;">g</span><span>, </span><span style="color:#d69d85;">&#39;%5D&#39;</span><span>); </span><span> r.return(</span><span style="color:#b5cea8;">200</span><span>); </span><span>} </span><span style="color:#569cd6;">export default </span><span>{urlencode}; </span></code></pre> kubernetes node shell plugin 2023-01-29T00:00:00+00:00 2023-01-29T00:00:00+00:00 Unknown https://yinheli.com/blog/2023/k8s-node-shell/ <p><code>kubectl exec</code> 可以进到 pod 容器内获取 shell，有时候需要 shell 到 node，而原生的 <code>kubectl debug</code> 虽然支持但是不支持特权模式，拿不到 root 权限，也不能用宿主机上的大部分工具，发现 <code>Lens</code> 上有这个功能，把核心的部分拷贝出来，做成脚本，作为 <code>kubectl</code> 插件。</p> <p>将 <code>kubectl-shell</code> 放到 <code>PATH</code> 即可， <code>kubectl shell &lt;node-name&gt;</code></p> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>vi kubectl-shell </span></code></pre> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;">#!/bin/bash </span><span>set -e </span><span> </span><span>node=</span><span style="background-color:#282828;color:#d69d85;">${</span><span style="background-color:#282828;color:#dcdcdc;">1</span><span style="background-color:#282828;color:#d69d85;">}</span><span> </span><span>nodeName=</span><span style="background-color:#282828;color:#d69d85;">$(</span><span style="background-color:#282828;color:#dcdcdc;">kubectl</span><span style="background-color:#282828;color:#d69d85;"> get node ${</span><span style="background-color:#282828;color:#dcdcdc;">node</span><span style="background-color:#282828;color:#d69d85;">}</span><span style="background-color:#282828;color:#dcdcdc;"> -o</span><span style="background-color:#282828;color:#d69d85;"> template</span><span style="background-color:#282828;color:#dcdcdc;"> --template=</span><span style="background-color:#282828;color:#d69d85;">&#39;{{index .metadata.labels &quot;kubernetes.io/hostname&quot;}}&#39;)</span><span> </span><span>nodeSelector=</span><span style="background-color:#282828;color:#d69d85;">&#39;&quot;nodeSelector&quot;: { &quot;kubernetes.io/hostname&quot;: &quot;&#39;${</span><span style="background-color:#282828;color:#dcdcdc;">nodeName:?</span><span style="background-color:#282828;color:#d69d85;">}&#39;&quot; },&#39;</span><span> </span><span>podName=</span><span style="background-color:#282828;color:#d69d85;">node-shell-${</span><span style="background-color:#282828;color:#dcdcdc;">node</span><span style="background-color:#282828;color:#d69d85;">}</span><span> </span><span> </span><span>kubectl run ${podName:?} --restart=Never -it --rm --timeout=5m --pod-running-timeout=5m --image overriden --overrides </span><span style="color:#d69d85;">&#39; </span><span style="color:#d69d85;">{ </span><span style="color:#d69d85;"> &quot;spec&quot;: { </span><span style="color:#d69d85;"> &quot;hostPID&quot;: true, </span><span style="color:#d69d85;"> &quot;hostNetwork&quot;: true, </span><span style="color:#d69d85;"> &#39;&quot;${</span><span>nodeSelector?</span><span style="color:#d69d85;">}&quot;&#39; </span><span style="color:#d69d85;"> &quot;tolerations&quot;: [{ </span><span style="color:#d69d85;"> &quot;operator&quot;: &quot;Exists&quot; </span><span style="color:#d69d85;"> }], </span><span style="color:#d69d85;"> &quot;containers&quot;: [ </span><span style="color:#d69d85;"> { </span><span style="color:#d69d85;"> &quot;name&quot;: &quot;shell&quot;, </span><span style="color:#d69d85;"> &quot;image&quot;: &quot;alpine&quot;, </span><span style="color:#d69d85;"> &quot;command&quot;: [ </span><span style="color:#d69d85;"> &quot;nsenter&quot;, &quot;-t&quot;, &quot;1&quot;, &quot;-m&quot;, &quot;-u&quot;, &quot;-i&quot;, &quot;-n&quot;, </span><span style="color:#d69d85;"> &quot;bash&quot; </span><span style="color:#d69d85;"> ], </span><span style="color:#d69d85;"> &quot;stdin&quot;: true, </span><span style="color:#d69d85;"> &quot;tty&quot;: true, </span><span style="color:#d69d85;"> &quot;securityContext&quot;: { </span><span style="color:#d69d85;"> &quot;privileged&quot;: true </span><span style="color:#d69d85;"> } </span><span style="color:#d69d85;"> } </span><span style="color:#d69d85;"> ] </span><span style="color:#d69d85;"> } </span><span style="color:#d69d85;">}&#39;</span><span> --attach </span><span style="color:#d69d85;">&quot;$</span><span>@</span><span style="color:#d69d85;">&quot; </span></code></pre> <p>原理：</p> <p>创建一个有特权模式的 pod，调度到对应的 node 上运行，然后 <a href="https://man7.org/linux/man-pages/man1/nsenter.1.html"><code>nsenter</code></a> 拿到 root shell</p> <p>P.S.</p> <p>发现已有社区相关<a href="https://github.com/kvaps/kubectl-node-shell">插件</a>了，下次可以先去 <a href="https://krew.sigs.k8s.io/plugins/">krew</a> 找一下 :)</p> kaniko build cli 2023-01-18T00:00:00+00:00 2023-01-18T00:00:00+00:00 Unknown https://yinheli.com/blog/2023/kaniko-build-cli/ <p>造了个 cli 工具，<a href="https://github.com/yinheli/kaniko-build">kaniko-build</a> 用于在 k8s 集群中打包镜像。</p> <p>kaniko 是我们用的比较多的工具，包括 tekton 流水线上也在使用，如果在本地调试，此前要么是本地 docker build 要么就是几个 bash 脚本，构建环境和创建构建资源，这次把这些脚本整理了一下，写成 python 脚本，用 <code>click</code> 这个包很快就做成一个 cli 工具，而相关的 yaml 文件则使用模板渲染丢给 kubectl pipe，觉得清爽方便了很多。</p> <p>借助 kaniko 本身的能力，它除了可以支持挂载 pv ，使用本地 Dockerfile， 也支持远程包括 git， s3 等，具体请参考官方文档吧，我只用到 pv 挂载和 git 远程库两种场景，其他场景未做测试。</p> <h1 id="xiang-guan-zi-liao">相关资料</h1> <ul> <li>kaniko <a href="https://github.com/GoogleContainerTools/kaniko">https://github.com/GoogleContainerTools/kaniko</a></li> </ul> 阿里云 k8s ingress xff 问题 2023-01-17T00:00:00+00:00 2023-01-17T00:00:00+00:00 Unknown https://yinheli.com/blog/2023/aliyun-k8s-ingress-xff-issue/ <p>在排查 IP 问题的时候发现记录的 IP 有明显问题，都是运营商或者机房的，感觉 xff 出问题了，修改 nginx-ingress 的 ConfigMap 解决，以下为推荐配置，请参考官方文档和业务情况适当调整</p> <pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">generate-request-id</span><span>: </span><span style="color:#d69d85;">&quot;true&quot; </span><span style="background-color:#282828;color:#569cd6;">use-forwarded-headers</span><span>: </span><span style="color:#d69d85;">&quot;true&quot; </span><span style="background-color:#282828;color:#569cd6;">enable-real-ip</span><span>: </span><span style="color:#d69d85;">&quot;true&quot; </span><span style="background-color:#282828;color:#569cd6;">proxy-real-ip-cidr</span><span>: </span><span style="color:#d69d85;">&quot;0.0.0.0/0&quot; </span><span style="background-color:#282828;color:#569cd6;">proxy-add-original-uri-header</span><span>: </span><span style="color:#d69d85;">&quot;true&quot; </span><span style="background-color:#282828;color:#569cd6;">compute-full-forwarded-for</span><span>: </span><span style="color:#d69d85;">&quot;true&quot; </span><span style="background-color:#282828;color:#569cd6;">allow-backend-server-header</span><span>: </span><span style="color:#d69d85;">&quot;true&quot; </span><span style="background-color:#282828;color:#569cd6;">enable-underscores-in-headers</span><span>: </span><span style="color:#d69d85;">&quot;true&quot; </span></code></pre> <p>默认的 ingress 对应的负载均衡器为安装时的固定规格，也可以参考 Annotation 配置，调整 nginx-lb 的 service，<a href="https://yinheli.com/blog/2023/aliyun-k8s-reduce-costs/">调整规格</a>，计划方案，如 LCU 计费或包年包月计划等，如果波峰比较大，个人比较推荐 LCU 的计费模式，省钱的同时也能应对突增流量。</p> <h2 id="xiang-guan-zi-liao">相关资料</h2> <ul> <li><a href="https://github.com/kubernetes/ingress-nginx">https://github.com/kubernetes/ingress-nginx</a></li> <li><a href="https://kubernetes.github.io/ingress-nginx/">https://kubernetes.github.io/ingress-nginx/</a></li> <li><a href="https://help.aliyun.com/document_detail/86531.html">https://help.aliyun.com/document_detail/86531.html</a></li> </ul> 阿里云 k8s 节约成本记录 2023-01-11T00:00:00+00:00 2023-01-11T00:00:00+00:00 Unknown https://yinheli.com/blog/2023/aliyun-k8s-reduce-costs/ <h1 id="bei-jing">背景</h1> <p>公司部分阿里云的项目逐步迁入阿里云的 K8S，由于业务场景需求，迁入版本为 ASK，ASK 是构建在 ECI 之上的，所以所有的工作负载，pod 实际上是对应 ECI 的一个容器，如没有明确指定配置或设定 <code>resource</code> 默认为 <code>2cpu,4G</code> 版本，并且按需付费。</p> <p>我们有 3 套环境，测试、 灰度和生产，开发环境则部署在公司办公室环境的本地虚拟服务器中。</p> <h1 id="jie-sheng-fang-an">节省方案</h1> <h2 id="ce-shi-huan-jing">测试环境</h2> <ul> <li>无状态负载配置 <code>Cron HPA</code> 定时关闭（设置副本数为 <code>0</code>）（比如每天凌晨）</li> <li>优化 cpu &amp; 内存配置，尽量使用抢占式实例，突发性能实例能能节省很多，超乎想象</li> <li>尽量使用内网负载均衡器，如果没有外部访问需求，应该创建 <code>ClusterIP</code> 的 service</li> <li>实用 <code>LCU</code> 计费的负载均衡器和 ingress</li> </ul> <h2 id="sheng-chan-huan-jing">生产环境</h2> <ul> <li>优化 cpu &amp; 内存配置，避免浪费，更多副本可能比更高配的机器省钱，更能避免单点故障</li> <li>使用 <code>LCU</code> 计费的负载均衡器，包括 ingress。这比较适合流量峰谷比较明显的场景</li> </ul> <h1 id="shi-ji-xiao-guo">实际效果</h1> <ul> <li>测试环境节省超过 70%</li> <li>生产环境待评估，预计超过 30%</li> </ul> <h1 id="xiang-guan-zi-liao">相关资料</h1> <ul> <li><a href="https://help.aliyun.com/document_detail/154527.htm">ECI 计费文档</a></li> <li>ECS <a href="https://www.aliyun.com/price/product#/ecs/detail/vm">价格表</a></li> <li>更多<a href="https://help.aliyun.com/document_detail/186939.html">注解说明</a>， 如抢占式突发性能实例</li> </ul> hello world, hello rust 2023-01-11T00:00:00+00:00 2023-01-11T00:00:00+00:00 Unknown https://yinheli.com/blog/2023/hello-world-hello-rust/ <p>blog 从 hugo 迁入 <a href="https://github.com/getzola/zola">zola</a> 偏爱 rust。</p> stable diffusion 2022-10-07T00:00:00+00:00 2022-10-07T00:00:00+00:00 Unknown https://yinheli.com/blog/2023/learn-stable-diffusion/ <h1 id="kai-shi">开始</h1> <p>https://github.com/AUTOMATIC1111/stable-diffusion-webui https://github.com/comfyanonymous/ComfyUI</p> <h1 id="ru-men-xue-xi-kai-fa-zhe-jiao-du">入门学习（开发者角度）</h1> <ul> <li>huggingface 的公开课 https://github.com/huggingface/diffusion-models-class</li> <li>相对系统的总结了一些 sd-webui 的使用和基本概念 https://stable-diffusion-art.com/</li> </ul> <h1 id="ji-chu-mo-xing">基础模型</h1> <blockquote> <p>https://huggingface.co/runwayml/stable-diffusion-v1-5</p> </blockquote> <ul> <li><a href="https://huggingface.co/runwayml/stable-diffusion-v1-5/resolve/main/v1-5-pruned-emaonly.ckpt">v1-5-pruned-emaonly.ckpt</a> - 4.27GB, ema-only weight. uses less VRAM - suitable for inference <ul> <li>适合推理</li> </ul> </li> <li><a href="https://huggingface.co/runwayml/stable-diffusion-v1-5/resolve/main/v1-5-pruned.ckpt">v1-5-pruned.ckpt</a> - 7.7GB, ema+non-ema weights. uses more VRAM - suitable for fine-tuning <ul> <li>适合调优</li> </ul> </li> </ul> <blockquote> <p>更推荐下载 <code>safetensors</code> 格式</p> </blockquote> <h2 id="geng-duo-mo-xing">更多模型</h2> <ul> <li>C站 https://civitai.com/</li> <li>哩布哩布 https://www.liblib.ai/</li> </ul> Greenplum 入门 2020-03-11T00:00:00+00:00 2020-03-11T00:00:00+00:00 Unknown https://yinheli.com/blog/2020/greenplum-get-starting/ <p>Greenplum 数据库是 shared nothing 的分析型 MPP 数据库。近期由于老婆的项目组在使用，我也顺便了解学习下。记录下主要的流程步骤备忘。</p> <p>如果你参考我的文档，需要对 Greenplum 的架构有个大概的了解，官网和中文社区是个好去处。</p> <h2 id="huan-jing-zhun-bei">环境准备</h2> <p>资源有限，用 docker 来模拟多个服务器的情况，先准备 docker 环境。 我这里宿主机是 centos 7， 参考 docker 官方文档, 依次安装 docker 和 docker-composer。</p> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;"># setup repository </span><span>yum install -y yum-utils \ </span><span> device-mapper-persistent-data \ </span><span> lvm2 </span><span> </span><span>yum-config-manager \ </span><span> --add-repo \ </span><span> https://download.docker.com/linux/centos/docker-ce.repo </span><span> </span><span>yum-config-manager --disable docker-ce-nightly </span><span> </span><span>yum install -y docker-ce docker-ce-cli containerd.io </span><span> </span><span style="color:#608b4e;"># start docker </span><span>systemctl start docker </span><span> </span><span style="color:#608b4e;"># enable docker auto start </span><span>systemctl enable docker </span><span> </span><span style="color:#608b4e;"># install docker-composer </span><span>curl -L </span><span style="color:#d69d85;">&quot;https://github.com/docker/compose/releases/download/1.25.4/docker-compose-$(</span><span>uname -s</span><span style="color:#d69d85;">)-$(</span><span>uname -m</span><span style="color:#d69d85;">)&quot;</span><span> -o /usr/local/bin/docker-compose </span><span>chmod +x /usr/local/bin/docker-compose </span><span> </span><span style="color:#608b4e;"># test docker and composer </span><span>docker info </span><span>docker-compose version </span></code></pre> <p>要模拟整个安装流程，因此只准备一个基础的镜像，仅安装必要的软件包。<code>Dockerfile</code> 如下：</p> <pre data-lang="dockerfile" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-dockerfile "><code class="language-dockerfile" data-lang="dockerfile"><span style="color:#569cd6;">FROM</span><span> centos:centos7 </span><span> </span><span style="color:#569cd6;">RUN </span><span>yum install -y wget </span><span> </span><span style="color:#608b4e;"># 够用阿里云的镜像，在国内，加速包下载 </span><span style="color:#608b4e;"># 安装基本的软件包，并支持 sshd 服务，greenplum 节点之间通讯需要 </span><span style="color:#569cd6;">RUN </span><span>wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo &amp;&amp; \ </span><span> yum makecache &amp;&amp; \ </span><span> yum install -y which bind-utils net-tools less wget curl zip unzip telnet lsof git &amp;&amp; \ </span><span> yum install -y passwd openssh-clients openssh-server ssh-keygen &amp;&amp; \ </span><span> ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -N </span><span style="color:#d69d85;">&quot;&quot;</span><span> -q &amp;&amp; \ </span><span> ssh-keygen -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key -N </span><span style="color:#d69d85;">&quot;&quot;</span><span> -q &amp;&amp; \ </span><span> ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key -N </span><span style="color:#d69d85;">&quot;&quot;</span><span> -q &amp;&amp; \ </span><span> echo </span><span style="color:#d69d85;">&#39;UseDNS no&#39;</span><span> &gt;&gt; /etc/ssh/sshd_config </span><span> </span><span style="color:#608b4e;"># 安装 greenplum 依赖包 </span><span style="color:#569cd6;">RUN </span><span>yum install -y ed apr apr-util bzip2 krb5-devel libevent libyaml openssl iproute </span><span> </span><span style="color:#608b4e;"># 初始化用户和数据文件 </span><span style="color:#569cd6;">RUN </span><span>useradd -m gpadmin </span><span style="color:#569cd6;">RUN </span><span>mkdir -p /data &amp;&amp; chown -R gpadmin:gpadmin /data </span><span> </span><span style="color:#569cd6;">EXPOSE </span><span>22 </span><span> </span><span style="color:#569cd6;">CMD </span><span>[</span><span style="color:#d69d85;">&quot;/usr/sbin/sshd&quot;</span><span>, </span><span style="color:#d69d85;">&quot;-D&quot;</span><span>] </span></code></pre> <p>build 基于 centos 7 的镜像，名为 <code>mygreenplum</code>，后面我们用到这个镜像。</p> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>docker build . -t mygreenplum </span></code></pre> <p>准备 <code>docker-compose.yaml</code></p> <blockquote> <p>规划 1 个 master 节点，2 个 segment 节点，1 个 ETL 节点（用来做 ETL 任务）</p> <p>节点命名参考官方命名习惯: mdw ( master 节点 )</p> <p>另外注意，需要准备一个内存大一点的服务器，我的这里的试验机为 8G 内存</p> </blockquote> <pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">version</span><span>: </span><span style="color:#d69d85;">&quot;3&quot; </span><span style="background-color:#282828;color:#569cd6;">services</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">master</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">image</span><span>: </span><span style="color:#d69d85;">&quot;mygreenplum&quot; </span><span> </span><span style="background-color:#282828;color:#569cd6;">hostname</span><span>: </span><span style="color:#d69d85;">&quot;mdw&quot; </span><span> </span><span style="background-color:#282828;color:#569cd6;">container_name</span><span>: </span><span style="color:#d69d85;">&quot;master&quot; </span><span> </span><span style="background-color:#282828;color:#569cd6;">ports</span><span>: </span><span> - </span><span style="color:#d69d85;">&quot;2222:22&quot; </span><span> - </span><span style="color:#d69d85;">&quot;5432:5432&quot; </span><span> </span><span style="background-color:#282828;color:#569cd6;">sysctls</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">net.core.somaxconn</span><span>: </span><span style="color:#b5cea8;">1024 </span><span> </span><span style="background-color:#282828;color:#569cd6;">net.ipv4.tcp_syncookies</span><span>: </span><span style="color:#b5cea8;">0 </span><span> </span><span> </span><span style="background-color:#282828;color:#569cd6;">volumes</span><span>: </span><span> - </span><span style="background-color:#282828;color:#d69d85;">.:/root/downloads</span><span> </span><span> </span><span style="background-color:#282828;color:#569cd6;">networks</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">main</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">aliases</span><span>: </span><span> - </span><span style="color:#d69d85;">&quot;master&quot; </span><span> </span><span style="background-color:#282828;color:#569cd6;">segment01</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">image</span><span>: </span><span style="color:#d69d85;">&quot;mygreenplum&quot; </span><span> </span><span style="background-color:#282828;color:#569cd6;">hostname</span><span>: </span><span style="color:#d69d85;">&quot;segment01&quot; </span><span> </span><span style="background-color:#282828;color:#569cd6;">container_name</span><span>: </span><span style="color:#d69d85;">&quot;segment01&quot; </span><span> </span><span style="background-color:#282828;color:#569cd6;">volumes</span><span>: </span><span> - </span><span style="background-color:#282828;color:#d69d85;">.:/root/downloads</span><span> </span><span> </span><span style="background-color:#282828;color:#569cd6;">networks</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">main</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">aliases</span><span>: </span><span> - </span><span style="color:#d69d85;">&quot;segment01&quot; </span><span> </span><span style="background-color:#282828;color:#569cd6;">segment02</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">image</span><span>: </span><span style="color:#d69d85;">&quot;mygreenplum&quot; </span><span> </span><span style="background-color:#282828;color:#569cd6;">hostname</span><span>: </span><span style="color:#d69d85;">&quot;segment02&quot; </span><span> </span><span style="background-color:#282828;color:#569cd6;">container_name</span><span>: </span><span style="color:#d69d85;">&quot;segment02&quot; </span><span> </span><span style="background-color:#282828;color:#569cd6;">volumes</span><span>: </span><span> - </span><span style="background-color:#282828;color:#d69d85;">.:/root/downloads</span><span> </span><span> </span><span style="background-color:#282828;color:#569cd6;">networks</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">main</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">aliases</span><span>: </span><span> - </span><span style="color:#d69d85;">&quot;segment02&quot; </span><span> </span><span style="background-color:#282828;color:#569cd6;">segment03</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">image</span><span>: </span><span style="color:#d69d85;">&quot;mygreenplum&quot; </span><span> </span><span style="background-color:#282828;color:#569cd6;">hostname</span><span>: </span><span style="color:#d69d85;">&quot;segment03&quot; </span><span> </span><span style="background-color:#282828;color:#569cd6;">container_name</span><span>: </span><span style="color:#d69d85;">&quot;segment03&quot; </span><span> </span><span style="background-color:#282828;color:#569cd6;">volumes</span><span>: </span><span> - </span><span style="background-color:#282828;color:#d69d85;">.:/root/downloads</span><span> </span><span> </span><span style="background-color:#282828;color:#569cd6;">networks</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">main</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">aliases</span><span>: </span><span> - </span><span style="color:#d69d85;">&quot;segment03&quot; </span><span> </span><span style="background-color:#282828;color:#569cd6;">networks</span><span>: </span><span> </span><span style="background-color:#282828;color:#569cd6;">main</span><span>: </span></code></pre> <p>github 下载 Greenplum 二进制安装包</p> <pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>wget https://github.com/greenplum-db/gpdb/releases/download/6.4.0/greenplum-db-6.4.0-rhel7-x86_64.rpm </span></code></pre> <p>启动服务</p> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>docker-composer up -d </span><span> </span><span style="color:#608b4e;"># 查看启动情况 </span><span>docker ps -a </span></code></pre> <h2 id="an-zhuang">安装</h2> <p>参考 <a href="https://gpdb.docs.pivotal.io/6-4/install_guide/install_gpdb.html">pivotal 上的安装指导</a>, 由于是在 docker 里，指导里的系统配置部分可以省略，实际生产环境建议遵循指导手册说明，以及相关参数的调优。</p> <p>进入 master 节点，初始化其 gpadmin 用户 ssh 密钥</p> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>docker exec -it master /bin/bash -c </span><span style="color:#d69d85;">&#39;su - gpadmin&#39; </span><span>ssh-keygen </span><span>cat ~/.ssh/id_rsa.pub </span></code></pre> <p>进入 其他节点，把 master 节点的 ssh 公钥写入 gpadmin 用户下</p> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>docker exec -it segment01 /bin/bash -c </span><span style="color:#d69d85;">&#39;su - gpadmin&#39; </span><span>mkdir .ssh </span><span>vi .ssh/authorized_keys </span><span>chmod 700 .ssh </span><span style="color:#569cd6;">&amp;&amp; </span><span>chmod 600 .ssh/authorized_keys </span></code></pre> <p>回到 master 节点，准备 greenplum-db 的安装</p> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>docker exec -it master /bin/bash </span><span style="color:#608b4e;"># root 用户执行安装 </span><span>yum install -y downloads/greenplum-db-6.4.0-rhel7-x86_64.rpm </span><span>chown -R gpadmin:gpadmin /usr/local/greenplum</span><span style="color:#569cd6;">* </span><span> </span><span style="color:#608b4e;"># 切换到 gpadmin 修改 profile </span><span>su - gpadmin </span><span>mkdir -p /data/master </span><span>vi ~/.bash_profile </span></code></pre> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>source /usr/local/greenplum-db/greenplum_path.sh </span><span style="color:#569cd6;">export </span><span>MASTER_DATA_DIRECTORY=</span><span style="background-color:#282828;color:#d69d85;">/data/master/gpseg-1</span><span> </span></code></pre> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;"># 应用 profile </span><span>source ~/.bash_prifile </span><span> </span><span style="color:#608b4e;"># 准备 hostfile_exkeys 将所有的 segment、etl 节点的 hostname 都写到里面 </span><span>vi hostfile_exkeys </span></code></pre> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;"># 验证所有的节点都是可以连接的 </span><span>gpssh -f hostfile_exkeys -e </span><span style="color:#d69d85;">&#39;uname -a&#39; </span></code></pre> <p>进入其他节点，和 master 节点一样，安装 greenplum-db 的 rpm 包</p> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>docker exec -it segment01 /bin/bash </span><span>yum install -y downloads/greenplum-db-6.4.0-rhel7-x86_64.rpm </span><span>chown -R gpadmin:gpadmin /usr/local/greenplum</span><span style="color:#569cd6;">* </span></code></pre> <h2 id="chu-shi-hua">初始化</h2> <h2 id="deng-lu">登录</h2> <h2 id="ce-shi-dao-ru-shu-ju">测试导入数据</h2> <p>参考资料：</p> <ul> <li>Docker: https://docs.docker.com/install/</li> <li>Greenplum: https://github.com/greenplum-db/gpdb &amp; https://greenplum.org/</li> <li>pivotal gpdp: https://gpdb.docs.pivotal.io/6-4/main/index.html</li> </ul> GORM 自定义数据类型的处理 2018-05-17T00:00:00+00:00 2018-05-17T00:00:00+00:00 Unknown https://yinheli.com/blog/2018/gorm-custom-type/ <p>如果有自定的数据类型，关键是实现 sql 的序列和反序列化方法，即: <code>driver.Valuer</code> <code>sql.Scanner</code> 接口。</p> <p>例如有如下数据结构</p> <pre data-lang="go" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-go "><code class="language-go" data-lang="go"><span style="color:#569cd6;">type </span><span style="color:#4ec9b0;">StringArray </span><span>[]string </span><span> </span><span style="color:#569cd6;">type </span><span style="color:#4ec9b0;">RecArticle </span><span style="color:#569cd6;">struct </span><span>{ </span><span> Id uint64 </span><span style="color:#d69d85;">`gorm:&quot;column:id&quot; json:&quot;Id,string&quot;` </span><span> Tags </span><span style="color:#569cd6;">StringArray </span><span style="color:#d69d85;">`gorm:&quot;column:tags&quot; json:&quot;tags&quot;` </span><span> Cates </span><span style="color:#569cd6;">StringArray </span><span style="color:#d69d85;">`gorm:&quot;column:cates&quot; json:&quot;cates&quot;` </span><span>} </span></code></pre> <p>实现上面的接口</p> <pre data-lang="go" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-go "><code class="language-go" data-lang="go"><span style="color:#569cd6;">func </span><span>(t </span><span style="color:#569cd6;">StringArray</span><span>) Value() (driver.</span><span style="color:#569cd6;">Value</span><span>, error) { </span><span> b, _ := json.Marshal(t) </span><span> </span><span style="color:#569cd6;">return </span><span>string(b), </span><span style="color:#569cd6;">nil </span><span>} </span><span> </span><span style="color:#569cd6;">func </span><span>(t </span><span style="color:#569cd6;">*StringArray</span><span>) Scan(src </span><span style="color:#569cd6;">interface</span><span>{}) error { </span><span> </span><span style="color:#569cd6;">if </span><span>src </span><span style="color:#569cd6;">!= nil </span><span>{ </span><span> </span><span style="color:#569cd6;">var </span><span>v []string </span><span> json.Unmarshal(src.([]byte), </span><span style="color:#569cd6;">&amp;</span><span>v) </span><span> </span><span style="color:#569cd6;">*</span><span>t = v </span><span> } </span><span> </span><span style="color:#569cd6;">return nil </span><span>} </span></code></pre> <p>如果对时间戳，json 都需要特定的处理</p> <p>add callbacks</p> <pre data-lang="go" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-go "><code class="language-go" data-lang="go"><span style="color:#608b4e;">// ... init gorm </span><span>DB, err = gorm.Open(</span><span style="color:#d69d85;">&quot;mysql&quot;</span><span>, </span><span style="color:#d69d85;">&quot;DSN&quot;</span><span>) </span><span style="color:#608b4e;">// ... handle error </span><span> </span><span style="color:#608b4e;">// add callbacks refer: http://gorm.io/docs/write_plugins.html </span><span>DB.Callback().Create().After(</span><span style="color:#d69d85;">&quot;gorm:update_time_stamp&quot;</span><span>).Register(</span><span style="color:#d69d85;">&quot;mark_timestamp_normal&quot;</span><span>, markTimestampNormal) </span><span>DB.Callback().Update().After(</span><span style="color:#d69d85;">&quot;gorm:update_time_stamp&quot;</span><span>).Register(</span><span style="color:#d69d85;">&quot;mark_timestamp_normal&quot;</span><span>, markTimestampNormal) </span><span style="color:#608b4e;">// .... </span><span> </span><span style="color:#569cd6;">func </span><span>markTimestampNormal(scope </span><span style="color:#569cd6;">*</span><span>gorm.</span><span style="color:#569cd6;">Scope</span><span>) { </span><span> </span><span style="color:#569cd6;">if </span><span>createdAtField, ok := scope.FieldByName(</span><span style="color:#d69d85;">&quot;CreatedAt&quot;</span><span>); ok { </span><span> createdAtField.IsNormal = </span><span style="color:#569cd6;">true </span><span> } </span><span> </span><span> </span><span style="color:#569cd6;">if </span><span>updatedAtField, ok := scope.FieldByName(</span><span style="color:#d69d85;">&quot;UpdatedAt&quot;</span><span>); ok { </span><span> updatedAtField.IsNormal = </span><span style="color:#569cd6;">true </span><span> } </span><span>} </span></code></pre> <p>alias</p> <pre data-lang="go" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-go "><code class="language-go" data-lang="go"><span style="color:#608b4e;">// alias </span><span style="color:#569cd6;">type </span><span style="color:#4ec9b0;">Timestamp </span><span>time.</span><span style="color:#569cd6;">Time </span><span> </span><span style="color:#608b4e;">// UnmarshalParam echo api @see https://echo.labstack.com/guide/request </span><span style="color:#569cd6;">func </span><span>(t </span><span style="color:#569cd6;">*Timestamp</span><span>) UnmarshalParam(src string) error { </span><span> </span><span style="color:#569cd6;">if </span><span>src </span><span style="color:#569cd6;">!= </span><span style="color:#d69d85;">&quot;&quot; </span><span>{ </span><span> m, err := strconv.ParseInt(src, </span><span style="color:#b5cea8;">10</span><span>, </span><span style="color:#b5cea8;">64</span><span>) </span><span> </span><span style="color:#569cd6;">if </span><span>err </span><span style="color:#569cd6;">!= nil </span><span>{ </span><span> </span><span style="color:#569cd6;">return </span><span>err </span><span> } </span><span> </span><span> ts := time.Unix(</span><span style="color:#b5cea8;">0</span><span>, m</span><span style="color:#569cd6;">*</span><span>int64(time.Millisecond)) </span><span> </span><span style="color:#569cd6;">*</span><span>t = Timestamp(ts) </span><span> } </span><span> </span><span style="color:#569cd6;">return nil </span><span>} </span><span> </span><span style="color:#608b4e;">// MarshalJSON echo api json response </span><span style="color:#569cd6;">func </span><span>(t </span><span style="color:#569cd6;">*Timestamp</span><span>) MarshalJSON() ([]byte, error) { </span><span> </span><span style="color:#569cd6;">if </span><span>t </span><span style="color:#569cd6;">!= nil </span><span>{ </span><span> ts := time.Time(</span><span style="color:#569cd6;">*</span><span>t) </span><span> </span><span style="color:#569cd6;">return </span><span>[]byte(fmt.Sprintf(</span><span style="color:#d69d85;">`&quot;</span><span style="color:#b4cea8;">%d</span><span style="color:#d69d85;">&quot;`</span><span>, ts.UnixNano()</span><span style="color:#569cd6;">/</span><span>int64(time.Millisecond))), </span><span style="color:#569cd6;">nil </span><span> } </span><span> </span><span style="color:#569cd6;">return nil</span><span>, </span><span style="color:#569cd6;">nil </span><span>} </span><span> </span><span style="color:#608b4e;">// for sql log, print readable format </span><span style="color:#569cd6;">func </span><span>(t </span><span style="color:#569cd6;">Timestamp</span><span>) String() string { </span><span> ts := time.Time(t) </span><span> </span><span style="color:#569cd6;">return </span><span>ts.Format(time.RFC3339) </span><span>} </span><span> </span><span style="color:#608b4e;">// insert into database conversion </span><span style="color:#569cd6;">func </span><span>(t </span><span style="color:#569cd6;">Timestamp</span><span>) Value() (driver.</span><span style="color:#569cd6;">Value</span><span>, error) { </span><span> </span><span style="color:#569cd6;">return </span><span>time.Time(t), </span><span style="color:#569cd6;">nil </span><span>} </span><span> </span><span style="color:#608b4e;">// read from database conversion </span><span style="color:#569cd6;">func </span><span>(t </span><span style="color:#569cd6;">*Timestamp</span><span>) Scan(src </span><span style="color:#569cd6;">interface</span><span>{}) error { </span><span> </span><span style="color:#569cd6;">if </span><span>val, ok := src.(time.</span><span style="color:#569cd6;">Time</span><span>); ok { </span><span> </span><span style="color:#569cd6;">*</span><span>t = Timestamp(val) </span><span> } </span><span> </span><span style="color:#569cd6;">return nil </span><span>} </span></code></pre> <p>model</p> <pre data-lang="go" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-go "><code class="language-go" data-lang="go"><span style="color:#569cd6;">type </span><span style="color:#4ec9b0;">App </span><span style="color:#569cd6;">struct </span><span>{ </span><span> Id uint32 </span><span style="color:#d69d85;">`json:&quot;id,string&quot;` </span><span> AppKey string </span><span style="color:#d69d85;">`json:&quot;appKey&quot; validate:&quot;required&quot;` </span><span> AppSecret string </span><span style="color:#d69d85;">`json:&quot;appSecret&quot; validate:&quot;required&quot;` </span><span> AppName string </span><span style="color:#d69d85;">`json:&quot;appName&quot; validate:&quot;required&quot;` </span><span> CreatedAt </span><span style="color:#569cd6;">Timestamp </span><span style="color:#d69d85;">`json:&quot;createdAt&quot;` </span><span> UpdatedAt </span><span style="color:#569cd6;">*Timestamp </span><span style="color:#d69d85;">`json:&quot;updatedAt&quot;` </span><span>} </span></code></pre> <p>Refer:</p> <ul> <li>http://gorm.io/docs/write_plugins.html</li> <li>https://github.com/jinzhu/gorm/issues/1236#issuecomment-389633747</li> <li>https://echo.labstack.com/guide/request</li> </ul> 远程初始化树莓派 2017-09-06T00:00:00+00:00 2017-09-06T00:00:00+00:00 Unknown https://yinheli.com/blog/2017/raspberry-install-without-keyboard/ <p>没有键盘，显示器，只有 wifi 网络的情况下，启用 树莓派, 官方下载树莓派镜像，使用 Etcher 刷入 sd 卡</p> <p>在 sd 卡根目录创建空白 ssh 文件，标记启用 ssh</p> <p>找个 linux 服务器，挂载 sd 卡的 ext 分区，修改：<code>vi /etc/network/interfaces</code></p> <pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>// ... </span><span> </span><span>// 添加 </span><span>allow-hotplug wlan0 </span><span>iface wlan0 inet dhcp </span><span> wpa-ssid &quot;joy&quot; </span><span> wpa-psk &quot;xxxx&quot; // 密码（明文） </span></code></pre> <p><code>vi /etc/wpa_supplicant/wpa_supplicant.conf</code></p> <pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>country=US </span><span>ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev </span><span>update_config=1 </span><span> </span><span>network={ </span><span> ssid=&quot;joy&quot; </span><span> psk=&quot;xxxx&quot; </span><span>} </span></code></pre> <p>配置 wifi 重连</p> <pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>cp /etc/ifplugd/action.d/ifupdown /etc/ifplugd/action.d/ifupdown.bak </span><span>cp /etc/wpa_supplicant/ifupdown.sh /etc/ifplugd/action.d/ifupdown </span></code></pre> iptables 笔记 2015-02-11T00:00:00+00:00 2015-02-11T00:00:00+00:00 Unknown https://yinheli.com/blog/2015/iptables-notes/ <p>看看服务器都已经应用了哪些规则</p> <pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>iptables -L -n </span><span>iptables -Z </span></code></pre> <p>清理掉全部的规则</p> <pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>iptables -F </span><span>iptables -X </span></code></pre> <p>定义自己的规则</p> <blockquote> <p>特别小心, 如果没有开放 ssh 端口就把 INPUT 的全部 DROP, shell 就掉了. 确保有一个 ssh 22 端口是开放的. 另一个保险是确保 iptables 不是启动运行, 这样还可以通过管理后台执行服务器重启来挽救. 否则.... 你的懂的</p> </blockquote> <pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>iptables -I INPUT 1 -m state --state RELATED,ESTABLISHED -j ACCEPT </span><span> </span><span># 各个链的规则 </span><span>iptables -P INPUT DROP </span><span>iptables -P OUTPUT ACCEPT </span><span>iptables -P FORWARD ACCEPT </span><span> </span><span># 允许 ping </span><span>iptables -A INPUT -p icmp -j ACCEPT </span><span># 允许回环地址 </span><span>iptables -A INPUT -i lo -j ACCEPT </span><span> </span><span># ssh 端口记录日志 </span><span>iptables -A INPUT 1 -p tcp --dport 22 -m limit --limit 3/minute --limit-burst 8 -j LOG --log-prefix ssh-burst: </span><span> </span><span># 10s 内, 超过5次请求, 抛弃 </span><span>iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --set </span><span>iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 10 --hitcount 5 -j DROP </span><span>iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 10 --hitcount 5 -j LOG --log-prefix ssh-conn-max-time: </span><span> </span><span>iptables -A INPUT -p tcp --dport 25000 -m state --state NEW -m recent --set </span><span>iptables -A INPUT -p tcp --dport 25000 -m state --state NEW -m recent --update --seconds 10 --hitcount 10 -j DROP </span><span>iptables -A INPUT -p tcp --dport 25000 -m state --state NEW -m recent --update --seconds 10 --hitcount 10 -j LOG --log-prefix app-conn-max-time: </span><span> </span><span># 限制某些端口的最大连接数 </span><span>iptables -A INPUT -p tcp --syn --dport 22 -m connlimit --connlimit-above 5 -j DROP </span><span>iptables -A INPUT -p tcp --syn --dport 25000 -m connlimit --connlimit-above 10 -j DROP </span><span> </span><span># 开放的端口 </span><span>iptables -A INPUT -p tcp --dport 22 -j ACCEPT </span><span>iptables -A INPUT -p tcp --dport 25000 -j ACCEPT </span><span>iptables -A INPUT -p tcp --dport 20000 -j ACCEPT </span><span>iptables -A INPUT -p tcp --dport 9100 -j ACCEPT </span><span> </span><span># 这条规则用在INPUT链默没有DROP的情况, 作用与-P DROP相同, 当前面所有的规则都没匹配时, 自然落到这个 REJECT 上. </span><span>iptables -A INPUT -j REJECT --reject-with icmp-host-prohibited </span><span>iptables -A FORWARD -j REJECT --reject-with icmp-host-prohibited </span></code></pre> <p>删除某条记录</p> <p>方法 1, 把配置的那个 <code>A</code> 换成 <code>D</code></p> <p><code>iptables -L -n --line-numbers</code> 使用 <code>--line-numbers</code> 打印行, 然后删除行 例如: <code>iptables -D INPUT 2</code></p> <p>保存所有改动</p> <pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>/etc/init.d/iptables save </span></code></pre> <!--more--> <pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span># 完整应用脚本, 用于快速初始化服务器 iptables </span><span> </span><span>iptables -L -n </span><span> </span><span>iptables -P INPUT ACCEPT </span><span>iptables -P OUTPUT ACCEPT </span><span>iptables -P FORWARD ACCEPT </span><span> </span><span>iptables -F </span><span>iptables -X </span><span>iptables -Z </span><span> </span><span>iptables -A INPUT -p icmp -j ACCEPT </span><span>iptables -A INPUT -i lo -j ACCEPT </span><span>iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT </span><span> </span><span>iptables -A INPUT 1 -p tcp --dport 22 -m limit --limit 3/minute --limit-burst 8 -j LOG --log-prefix ssh-burst: </span><span> </span><span>iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --set </span><span>iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 10 --hitcount 5 -j DROP </span><span>iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 10 --hitcount 5 -j LOG --log-prefix ssh-conn-max-times: </span><span> </span><span>iptables -A INPUT -p tcp --syn --dport 22 -m connlimit --connlimit-above 30 -j DROP </span><span>iptables -A INPUT -p tcp --dport 22 -j ACCEPT </span><span> </span><span>iptables -P INPUT DROP </span><span>iptables -P OUTPUT ACCEPT </span><span>iptables -P FORWARD ACCEPT </span><span> </span><span> </span><span>iptables -A INPUT -p tcp --dport 25000 -m state --state NEW -m recent --set </span><span>iptables -A INPUT -p tcp --dport 25000 -m state --state NEW -m recent --update --seconds 10 --hitcount 10 -j DROP </span><span>iptables -A INPUT -p tcp --dport 25000 -m state --state NEW -m recent --update --seconds 10 --hitcount 10 -j LOG --log-prefix app-conn-max-time: </span><span> </span><span>iptables -A INPUT -p tcp --syn --dport 25000 -m connlimit --connlimit-above 10 -j DROP </span><span>iptables -A INPUT -p tcp --dport 25000 -j ACCEPT </span><span> </span><span>iptables -A INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above 60 -j DROP </span><span>iptables -A INPUT -p tcp --dport 80 -j ACCEPT </span><span> </span><span>iptables -A INPUT -p tcp --syn --dport 443 -m connlimit --connlimit-above 60 -j DROP </span><span>iptables -A INPUT -p tcp --dport 443 -j ACCEPT </span><span> </span><span>iptables -A INPUT -p tcp --dport 20000 -j ACCEPT </span><span>iptables -A INPUT -p tcp --dport 13307 -j ACCEPT </span><span> </span><span>iptables -A INPUT -j REJECT --reject-with icmp-host-prohibited </span><span>iptables -A FORWARD -j REJECT --reject-with icmp-host-prohibited </span><span> </span><span>iptables -L -n </span><span> </span><span>/etc/init.d/iptables save </span></code></pre> mysql 数据导出和导入 2015-02-10T00:00:00+00:00 2015-02-10T00:00:00+00:00 Unknown https://yinheli.com/blog/2015/mysql-import-export/ <p>导出</p> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>mysqldump -C -u root --opt --flush-logs --quick dbname </span><span style="color:#569cd6;">| </span><span>gzip &gt; file.gz </span></code></pre> <pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>-C 使用压缩协议 </span><span>-u 用户名 </span><span>--opt Same as --add-drop-table, --add-locks, --create-options, 等, 其实默认是开启的 </span></code></pre> <p>导入</p> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>gunzip &lt; file.tgz </span><span style="color:#569cd6;">| </span><span>mysql -u root -p </span><span style="color:#d69d85;">&quot;password&quot;</span><span> dbname </span></code></pre> <p>一个全量备份脚本</p> <pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;">#!/bin/sh </span><span>PATH=</span><span style="background-color:#282828;color:#d69d85;">$</span><span style="background-color:#282828;color:#dcdcdc;">PATH</span><span style="background-color:#282828;color:#d69d85;">:$</span><span style="background-color:#282828;color:#dcdcdc;">HOME</span><span style="background-color:#282828;color:#d69d85;">/bin:/home/mysql/app/mysql/bin</span><span> </span><span style="color:#569cd6;">export </span><span>PATH </span><span>cd `dirname $0` </span><span>db=</span><span style="background-color:#282828;color:#d69d85;">hpos</span><span> </span><span>dd=`date +%Y_%m%d_%H%M` </span><span>sql_file=</span><span style="background-color:#282828;color:#d69d85;">db_backup_$</span><span style="background-color:#282828;color:#dcdcdc;">dd</span><span style="background-color:#282828;color:#d69d85;">.sql.gz</span><span> </span><span>ig=</span><span style="background-color:#282828;color:#d69d85;">&#39;&#39;</span><span> </span><span> </span><span style="color:#608b4e;"># 忽略一些表 </span><span style="color:#569cd6;">for</span><span> i </span><span style="color:#569cd6;">in</span><span> table_ignore_1 table_ignore_2 </span><span style="color:#569cd6;">do </span><span> ig=</span><span style="background-color:#282828;color:#d69d85;">$</span><span style="background-color:#282828;color:#dcdcdc;">ig</span><span style="background-color:#282828;color:#d69d85;">&#39; --ignore-table=&#39;$</span><span style="background-color:#282828;color:#dcdcdc;">db</span><span style="background-color:#282828;color:#d69d85;">.$</span><span style="background-color:#282828;color:#dcdcdc;">i</span><span> </span><span style="color:#569cd6;">done </span><span> </span><span style="color:#608b4e;">#echo $ig </span><span>mysqldump -u root --opt --flush-logs --quick $ig $db </span><span style="color:#569cd6;">| </span><span>gzip &gt; $sql_file </span></code></pre> centos 6 安装 vmware tools 无法启动的问题 2015-01-20T00:00:00+00:00 2015-01-20T00:00:00+00:00 Unknown https://yinheli.com/blog/2015/install-vmware-centos6-issue/ <p>错误信息</p> <pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>vmware-tools-thinprint start/running </span><span>initctl: Job failed to start </span><span>Unable to start services for VMware Tools </span></code></pre> <p>尝试安装 <code>fuse-libs</code></p> <pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>yum install fuse-libs </span></code></pre> <p>手工启动</p> <pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>/etc/vmware-tools/services.sh restart </span></code></pre> <pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>Stopping VMware Tools services in the virtual machine: </span><span> Guest operating system daemon: [确定] </span><span> VMware User Agent (vmware-user): [确定] </span><span> Blocking file system: [确定] </span><span> Unmounting HGFS shares: [确定] </span><span> Guest filesystem driver: [确定] </span><span> VM communication interface socket family: [确定] </span><span> VM communication interface: [确定] </span><span> Checking acpi hot plug [确定] </span><span>Starting VMware Tools services in the virtual machine: </span><span> Switching to guest configuration: [确定] </span><span> VMware Automatic Kmods: [确定] </span><span> VM communication interface: [确定] </span><span> VM communication interface socket family: [确定] </span><span> Guest filesystem driver: [确定] </span><span> Mounting HGFS shares: [确定] </span><span> Blocking file system: [确定] </span><span> Guest operating system daemon: [确定] </span></code></pre> <p>问题解决</p>