Upgrade K3s and infrastructure with selective DNS relay

ansible/.gitignore

@@ -14,4 +14,5 @@ kubeconfig-*.yaml
 # IDE
 .idea/
 .vscode/
+__pycache__/
 

ansible/README.md

@@ -154,4 +154,19 @@ ansible-playbook playbooks/site.yml -vvv
 ansible-playbook playbooks/site.yml --syntax-check
 ```
 
+## 升级已有集群
+
+加载 `ansible/.env` 后运行：
+
+```bash
+source .env
+ansible-playbook playbooks/site.yml --tags control-plane-relay,k3s,status
+```
+
+master 逐台升级，worker 保持并行（默认 20 forks）。已有 server 在安装前保存不压缩的 etcd 快照和恢复用 token；每台 server 等待本地 `/readyz`（包含 etcd 检查），所有节点等待目标版本和 Ready，失败则停止后续阶段。只有通过共享 master 执行的就绪检查限制为 5 并发，以避免 SSH 会话耗尽。快照位于 server 的 snapshots 目录，token 单独保存在 server 目录，权限 0600；恢复仍需另行演练。
+
+选择性中继名单见 `inventory/group_vars/all.yml` 的 `tailscale_control_plane_relay_peers`，说明见 `README_MANUAL.md`。
+
+在安装 Ansible 的 Python 环境运行就绪检查回归：`python -m unittest discover -s tests -v`。
+
 

ansible/README_MANUAL.md

@@ -120,10 +120,10 @@ kubectl taint nodes tca node-role.kubernetes.io/master:NoSchedule
 
 ### 节点网络优化
 
-可以在需要加速的节点屏蔽目标ip 比如47.79.16.80 让tailscale强制走drep服务器进行加速.
+在 `inventory/group_vars/all.yml` 的 `tailscale_control_plane_relay_peers` 中填写差链路节点的公网 DNS 名称（例如 `tthk.node.dev.cm`），然后执行 `ansible-playbook playbooks/site.yml --tags control-plane-relay`。DNS 必须直接解析到节点公网地址，不能经过 CDN 代理。
 
-```shell
+规则仅部署到 master：拒绝这些对端到 Tailscale UDP 41641 的直连探测，使其与控制平面的连接回落到中继。其他节点之间仍按 Tailscale 正常规则选择直连；粒度为对端节点，不能仅针对解密后的 TCP 6443。不要使用节点级强制 DERP 环境变量。
-iptables -A OUTPUT -p udp --dport 41641 -d 47.79.16.80 -j DROP && \
+
-ip6tables -A OUTPUT -p udp --dport 41641 -d 240b:4001:278:8402:0:3a3d:b8a5:3975 -j DROP && \
+规则由独立 nftables 表和 systemd 服务持久化，不保存或清空 Kubernetes/Tailscale 的动态规则。timer 每 5 分钟重新解析全部 A/AAAA 地址；解析或规则校验失败时保留已有规则。同一公网 NAT 下的节点会共同受该地址规则影响；DNS 未列出的其他公网或内网直连地址不在限制范围内。清空名单并重跑即可取消限制。
-ip6tables -A OUTPUT -p udp --dport 41641 -d 2402:4e00:1420:1700:2c34:8fa6:b6ba::/112 -j DROP
+
-```
+Tailscale 1.102.3 没有受支持的按 peer 强制 DERP 设置：ACL/grants 控制访问权限，`via` 选择出口节点、子网路由器或 app connector；Peer Relay 的授权也只允许在直连不可用时使用中继。`nodeAttrs` 中的 `only-tcp-443` 和调试变量 `TS_DEBUG_ALWAYS_USE_DERP`、`TS_DEBUG_NEVER_DIRECT_UDP` 都作用于整个节点，不能限定到控制平面。参考 [grants 路由说明](https://tailscale.com/docs/reference/syntax/grants#routing-specifications)、[Peer Relay](https://tailscale.com/docs/features/peer-relay)、[节点属性源码](https://github.com/tailscale/tailscale/blob/v1.102.3/tailcfg/tailcfg.go) 和 [调试开关源码](https://github.com/tailscale/tailscale/blob/v1.102.3/wgengine/magicsock/debugknobs.go)。

ansible/ansible.cfg

@@ -7,6 +7,7 @@ stdout_callback = default
 callbacks_enabled = ansible.builtin.default
 interpreter_python = /usr/bin/python3
 deprecation_warnings = False
+forks = 20
 
 [callback_default]
 result_format = yaml

ansible/inventory/group_vars/all.yml

@@ -12,9 +12,13 @@ ha_server_url: "{{ lookup('env', 'HA_SERVER_URL') | default('', true) }}"
 # ============================================
 # K3s Server URL (优先使用 HA_SERVER_URL，否则动态使用 init 节点地址)
 k3s_server_url: "{{ ha_server_url if (ha_server_url | length > 0) else '' }}"
-k3s_version: "v1.35.6+k3s1"
+k3s_version: "v1.35.9+k3s1"
 tailscale_version: "1.102.3"
 
+# 仅这些节点到控制平面的 Tailscale 连接回落到中继；DNS 必须直接指向节点公网地址。
+tailscale_control_plane_relay_peers:
+  - tthk.node.dev.cm
+
 # ETCD 配置
 etcd_snapshot_retention: 1
 etcd_snapshot_schedule_cron: "0 0 * * *"

ansible/playbooks/site.yml

@@ -74,6 +74,7 @@
   hosts: masters
   gather_facts: true
   serial: 1
+  any_errors_fatal: true
   tags: [k3s]
   roles:
     - role: k3s
@@ -109,6 +110,7 @@
       ansible.builtin.slurp:
         src: /var/lib/rancher/k3s/server/node-token
       register: k3s_token_content
+      no_log: true
       delegate_to: "{{ init_node }}"
       vars:
         ansible_port: "{{ hostvars[inventory_hostname].init_node_port }}"
@@ -125,11 +127,13 @@
       delegate_to: "{{ item }}"
       delegate_facts: true
       loop: "{{ ansible_play_hosts }}"
+      no_log: true
 
 - name: Install K3s on other masters
   hosts: masters
   gather_facts: true
   serial: 1
+  any_errors_fatal: true
   tags: [k3s]
   roles:
     - role: k3s
@@ -138,6 +142,7 @@
 - name: Install K3s on agents
   hosts: agents
   gather_facts: true
+  any_errors_fatal: true
   tags: [k3s]
   roles:
     - k3s

ansible/roles/common/files/k3s-control-plane-relay.py

@@ -0,0 +1,30 @@
+#!/usr/bin/python3
+import ipaddress
+import json
+from pathlib import Path
+import socket
+import subprocess
+
+
+def apply_rules(peers, base_rules):
+    addresses = set()
+    for peer in peers:
+        records = socket.getaddrinfo(peer, None, type=socket.SOCK_DGRAM)
+        if not records:
+            raise RuntimeError(f"No addresses for {peer}")
+        addresses.update(ipaddress.ip_address(record[4][0]) for record in records)
+
+    # Resolve every peer before replacing the table, preserving old rules on DNS failure.
+    rules = base_rules + "\n" + "\n".join(
+        f"add element inet k3s_control_plane_relay peers{address.version} {{ {address} }}"
+        for address in sorted(addresses, key=lambda address: (address.version, int(address)))
+    ) + "\n"
+    subprocess.run(["/usr/sbin/nft", "--check", "--file", "-"], input=rules, text=True, check=True)
+    subprocess.run(["/usr/sbin/nft", "--file", "-"], input=rules, text=True, check=True)
+
+
+if __name__ == "__main__":
+    apply_rules(
+        json.loads(Path("/etc/k3s-control-plane-relay.json").read_text()),
+        Path("/etc/k3s-control-plane-relay.nft").read_text(),
+    )

ansible/roles/common/files/k3s-control-plane-relay.service

@@ -0,0 +1,12 @@
+[Unit]
+Description=Selective Tailscale relay for K3s control-plane peers
+Wants=network-online.target
+After=network-online.target tailscaled.service
+Before=k3s.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/sbin/k3s-control-plane-relay
+
+[Install]
+WantedBy=multi-user.target

ansible/roles/common/files/k3s-control-plane-relay.timer

@@ -0,0 +1,9 @@
+[Unit]
+Description=Refresh selective relay peers from DNS
+
+[Timer]
+OnBootSec=30s
+OnUnitActiveSec=5min
+
+[Install]
+WantedBy=timers.target

ansible/roles/common/tasks/control-plane-relay.yml

@@ -0,0 +1,43 @@
+- name: Ensure nftables is available on control-plane nodes
+  ansible.builtin.apt:
+    name: nftables
+    state: present
+
+- name: Validate and deploy the selective relay rules
+  ansible.builtin.template:
+    src: control-plane-relay.nft.j2
+    dest: /etc/k3s-control-plane-relay.nft
+    mode: "0644"
+    validate: /usr/sbin/nft --check --file %s
+  register: control_plane_relay_rules
+
+- name: Configure public DNS names for selective relay
+  ansible.builtin.copy:
+    content: "{{ tailscale_control_plane_relay_peers | to_nice_json }}\n"
+    dest: /etc/k3s-control-plane-relay.json
+    mode: "0644"
+  register: control_plane_relay_peers
+
+- name: Install the selective relay resolver and systemd units
+  ansible.builtin.copy:
+    src: "{{ item.src }}"
+    dest: "{{ item.dest }}"
+    mode: "{{ item.mode }}"
+  loop:
+    - { src: k3s-control-plane-relay.py, dest: /usr/local/sbin/k3s-control-plane-relay, mode: "0755" }
+    - { src: k3s-control-plane-relay.service, dest: /etc/systemd/system/k3s-control-plane-relay.service, mode: "0644" }
+    - { src: k3s-control-plane-relay.timer, dest: /etc/systemd/system/k3s-control-plane-relay.timer, mode: "0644" }
+  register: control_plane_relay_files
+
+- name: Persist and apply selective relay
+  ansible.builtin.systemd:
+    name: k3s-control-plane-relay
+    enabled: true
+    daemon_reload: "{{ control_plane_relay_files.changed }}"
+    state: "{{ 'restarted' if control_plane_relay_rules.changed or control_plane_relay_peers.changed or control_plane_relay_files.changed else omit }}"
+
+- name: Enable periodic DNS refresh
+  ansible.builtin.systemd:
+    name: k3s-control-plane-relay.timer
+    enabled: true
+    state: started

ansible/roles/common/tasks/main.yml

@@ -81,3 +81,8 @@
     name: tailscaled
     enabled: true
     state: started
+
+- name: Configure selective control-plane relay
+  ansible.builtin.import_tasks: control-plane-relay.yml
+  when: inventory_hostname in groups['masters']
+  tags: [control-plane-relay]

ansible/roles/common/templates/control-plane-relay.nft.j2

@@ -0,0 +1,18 @@
+# Only this table is replaced; Kubernetes and Tailscale tables are left intact.
+add table inet k3s_control_plane_relay
+delete table inet k3s_control_plane_relay
+table inet k3s_control_plane_relay {
+    set peers4 {
+        type ipv4_addr
+        flags interval
+    }
+    set peers6 {
+        type ipv6_addr
+        flags interval
+    }
+    chain input {
+        type filter hook input priority -10; policy accept;
+        udp dport 41641 ip saddr @peers4 counter drop
+        udp dport 41641 ip6 saddr @peers6 counter drop
+    }
+}

ansible/roles/k3s/tasks/main.yml

@@ -34,6 +34,7 @@
     mode: "0600"
   when: "'masters' in group_names"
   register: k3s_server_config
+  no_log: true
 
 - name: Deploy K3s agent config
   ansible.builtin.template:
@@ -42,6 +43,7 @@
     mode: "0600"
   when: "'agents' in group_names"
   register: k3s_agent_config
+  no_log: true
 
 - name: Deploy registries.yaml
   ansible.builtin.template:
@@ -70,6 +72,34 @@
 
 
 # 下载安装脚本
+- name: Save an uncompressed etcd snapshot before upgrading
+  ansible.builtin.command:
+    argv:
+      - /usr/local/bin/k3s
+      - etcd-snapshot
+      - save
+      - --etcd-snapshot-name
+      - "pre-upgrade-{{ k3s_version }}"
+      - --etcd-snapshot-compress=false
+  when:
+    - "'masters' in group_names"
+    - k3s_binary.stat.exists
+    - k3s_needs_install
+  changed_when: true
+
+- name: Preserve the server token needed to restore the snapshot
+  ansible.builtin.copy:
+    src: /var/lib/rancher/k3s/server/token
+    dest: "/var/lib/rancher/k3s/server/pre-upgrade-{{ k3s_version }}.token"
+    remote_src: true
+    mode: "0600"
+    force: false
+  when:
+    - "'masters' in group_names"
+    - k3s_binary.stat.exists
+    - k3s_needs_install
+  no_log: true
+
 - name: Download K3s install script
   ansible.builtin.get_url:
     url: "{{ k3s_install_url }}"
@@ -106,13 +136,51 @@
     path: /tmp/k3s-install.sh
     state: absent
 
-# 等待 K3s 就绪 (仅 Server)
+# /readyz includes the local API server's etcd readiness check.
-- name: Wait for K3s server ready
+- name: Wait for the local API server and etcd to be ready
-  ansible.builtin.wait_for:
+  ansible.builtin.command:
-    path: /var/lib/rancher/k3s/server/node-token
+    argv:
-    timeout: 300
+      - /usr/local/bin/k3s
+      - kubectl
+      - --kubeconfig=/etc/rancher/k3s/k3s.yaml
+      - --request-timeout=10s
+      - get
+      - --raw=/readyz
+  register: k3s_api_ready
+  until: k3s_api_ready.rc == 0 and k3s_api_ready.stdout == 'ok'
+  retries: 60
+  delay: 5
+  changed_when: false
   when: "'masters' in group_names"
 
+- name: Wait for the node to report Ready at the requested version
+  ansible.builtin.command:
+    argv:
+      - /usr/local/bin/k3s
+      - kubectl
+      - --kubeconfig=/etc/rancher/k3s/k3s.yaml
+      - --request-timeout=10s
+      - get
+      - node
+      - "{{ node_hostname | default(inventory_hostname) }}"
+      - -o
+      - json
+  delegate_to: "{{ inventory_hostname if inventory_hostname in groups['masters'] else init_node }}"
+  # Worker installs stay parallel; avoid exhausting the shared master's SSH sessions.
+  throttle: 5
+  vars:
+    ansible_port: "{{ hostvars[inventory_hostname].ansible_port if inventory_hostname in groups['masters'] else hostvars[inventory_hostname].init_node_port }}"
+  register: k3s_node_status
+  until: >-
+    k3s_node_status.rc == 0 and
+    (k3s_node_status.stdout | from_json).status.nodeInfo.kubeletVersion == k3s_version and
+    ((k3s_node_status.stdout | from_json).status.conditions |
+      selectattr('type', 'equalto', 'Ready') |
+      selectattr('status', 'equalto', 'True') | list | length) == 1
+  retries: 60
+  delay: 5
+  changed_when: false
+
 # 保存 kubeconfig (仅 cluster-init)
 - name: Fetch kubeconfig
   ansible.builtin.fetch:

ansible/tests/test_control_plane_relay.py

@@ -0,0 +1,56 @@
+import importlib.util
+from pathlib import Path
+import socket
+import subprocess
+import unittest
+from unittest.mock import patch
+
+
+spec = importlib.util.spec_from_file_location(
+    "relay", Path(__file__).parents[1] / "roles/common/files/k3s-control-plane-relay.py"
+)
+relay = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(relay)
+BASE = (Path(__file__).parents[1] / "roles/common/templates/control-plane-relay.nft.j2").read_text()
+
+
+class ControlPlaneRelayTest(unittest.TestCase):
+    @patch.object(relay.subprocess, "run")
+    @patch.object(relay.socket, "getaddrinfo")
+    def test_all_dns_addresses_are_deduplicated_and_checked_before_apply(self, resolve, run):
+        resolve.return_value = [
+            (socket.AF_INET, 0, 0, "", ("192.0.2.1", 0)),
+            (socket.AF_INET6, 0, 0, "", ("2001:db8::1", 0, 0, 0)),
+            (socket.AF_INET, 0, 0, "", ("192.0.2.1", 0)),
+        ]
+        relay.apply_rules(["tthk.node.dev.cm"], BASE)
+        rules = run.call_args_list[0].kwargs["input"]
+        self.assertEqual(rules.count("peers4 { 192.0.2.1 }"), 1)
+        self.assertIn("peers6 { 2001:db8::1 }", rules)
+        self.assertIn("--check", run.call_args_list[0].args[0])
+        self.assertEqual(run.call_args_list[1].kwargs["input"], rules)
+
+    @patch.object(relay.subprocess, "run")
+    @patch.object(relay.socket, "getaddrinfo")
+    def test_any_dns_failure_preserves_existing_rules(self, resolve, run):
+        resolve.side_effect = [
+            [(socket.AF_INET, 0, 0, "", ("192.0.2.1", 0))],
+            socket.gaierror("temporary failure"),
+        ]
+        with self.assertRaises(socket.gaierror):
+            relay.apply_rules(["tthk.node.dev.cm", "other.node.dev.cm"], BASE)
+        run.assert_not_called()
+
+    @patch.object(relay.subprocess, "run", side_effect=subprocess.CalledProcessError(1, "nft"))
+    def test_invalid_rules_are_not_applied(self, run):
+        with self.assertRaises(subprocess.CalledProcessError):
+            relay.apply_rules([], BASE)
+        self.assertEqual(run.call_count, 1)
+
+    @patch.object(relay.subprocess, "run")
+    def test_empty_list_clears_only_the_owned_table(self, run):
+        relay.apply_rules([], BASE)
+        rules = run.call_args.kwargs["input"]
+        self.assertIn("delete table inet k3s_control_plane_relay", rules)
+        self.assertNotIn("add element", rules)
+        self.assertNotIn("flush ruleset", rules)

ansible/tests/test_readiness.py

@@ -0,0 +1,52 @@
+import json
+import unittest
+from pathlib import Path
+
+import yaml
+from jinja2 import Environment, StrictUndefined
+
+
+class ReadinessTest(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        tasks = yaml.safe_load(
+            (Path(__file__).parents[1] / "roles/k3s/tasks/main.yml").read_text()
+        )
+        env = Environment(undefined=StrictUndefined)
+        env.filters["from_json"] = json.loads
+        cls.conditions = {
+            task["register"]: env.compile_expression(task["until"])
+            for task in tasks
+            if "until" in task
+        }
+
+    def test_node_must_be_ready_at_the_requested_version(self):
+        condition = self.conditions["k3s_node_status"]
+        for version, ready, expected in [
+            ("v1.35.9+k3s1", "True", True),
+            ("v1.35.6+k3s1", "True", False),
+            ("v1.35.9+k3s1", "False", False),
+        ]:
+            with self.subTest(version=version, ready=ready):
+                status = {"status": {
+                    "nodeInfo": {"kubeletVersion": version},
+                    "conditions": [{"type": "Ready", "status": ready}],
+                }}
+                self.assertEqual(condition(
+                    k3s_node_status={"rc": 0, "stdout": json.dumps(status)},
+                    k3s_version="v1.35.9+k3s1",
+                ), expected)
+        self.assertFalse(condition(
+            k3s_node_status={"rc": 1, "stdout": ""},
+            k3s_version="v1.35.9+k3s1",
+        ))
+
+    def test_api_requires_a_successful_readyz_response(self):
+        condition = self.conditions["k3s_api_ready"]
+        for rc, stdout, expected in [(0, "ok", True), (1, "ok", False), (0, "", False)]:
+            with self.subTest(rc=rc, stdout=stdout):
+                self.assertEqual(condition(k3s_api_ready={"rc": rc, "stdout": stdout}), expected)
+
+
+if __name__ == "__main__":
+    unittest.main()

flux/README.md

@@ -1,5 +1,21 @@
 # Flux GitOps
 
+## 基础设施维护版本（2026-10-16）
+
+| 应用 | Chart | 应用版本 |
+| --- | --- | --- |
+| cert-manager | v1.21.2 | v1.21.2 |
+| CloudNativePG | 0.29.1 | 1.30.1 |
+| Barman Cloud 插件 | 0.8.1 | v0.15.1 |
+| Loki | 7.3.0 | 3.6.12 |
+| kube-prometheus-stack | 92.9.0 | Operator v0.94.1 / Prometheus v3.15.0 / Grafana 13.2.4 |
+| Velero | 12.2.1 | 1.18.2 |
+| Reflector | 10.0.65 | 10.0.65 |
+
+保留现有 PVC 和 Loki 存储 schema，通过已有 `CreateReplace` 策略更新 Operator CRD。默认后端 requests 按实际用量配置为 CPU 10m、内存 512Mi，使现有 CPU/内存 HPA 能正常计算；内存目标保持 chart 默认的 50%。
+
+Loki 显式使用 3.6.12：chart 7.3.0 的 appVersion 为 3.6.12，但其默认镜像值仍为 3.6.11。业务 PostgreSQL 数据版本独立由 Cluster 的 imageName 管理。
+
 ## 目录结构
 
 ```

flux/infrastructure/infra-data/helmrelease-barman-plugin.yaml

@@ -11,7 +11,7 @@ spec:
   chart:
     spec:
       chart: plugin-barman-cloud
-      version: 0.7.0
+      version: 0.8.1
       sourceRef:
         kind: HelmRepository
         name: cloudnative-pg

flux/infrastructure/infra-data/helmrelease-cloudnative-pg.yaml

@@ -9,7 +9,7 @@ spec:
   chart:
     spec:
       chart: cloudnative-pg
-      version: 0.28.3
+      version: 0.29.1
       sourceRef:
         kind: HelmRepository
         name: cloudnative-pg

flux/infrastructure/infra-devops/helmrelease-cert-manager.yaml

@@ -14,7 +14,7 @@ spec:
   chart:
     spec:
       chart: cert-manager
-      version: v1.20.2
+      version: v1.21.2
       sourceRef:
         kind: HelmRepository
         name: jetstack

flux/infrastructure/infra-devops/helmrelease-reflector.yaml

@@ -9,7 +9,7 @@ spec:
   chart:
     spec:
       chart: reflector
-      version: 10.0.50
+      version: 10.0.65
       sourceRef:
         kind: HelmRepository
         name: emberstack

flux/infrastructure/infra-devops/helmrelease-velero.yaml

@@ -9,7 +9,7 @@ spec:
   chart:
     spec:
       chart: velero
-      version: 12.0.3
+      version: 12.2.1
       sourceRef:
         kind: HelmRepository
         name: vmware-tanzu

flux/infrastructure/infra-monitor/helmrelease-loki.yaml

@@ -9,7 +9,7 @@ spec:
   chart:
     spec:
       chart: loki
-      version: 7.0.0
+      version: 7.3.0
       sourceRef:
         kind: HelmRepository
         name: grafana
@@ -30,6 +30,8 @@ spec:
     chunksCache:
       allocatedMemory: 1024
     loki:
+      image:
+        tag: 3.6.12
       auth_enabled: false
       commonConfig:
         replication_factor: 1

flux/infrastructure/infra-monitor/helmrelease-prometheus.yaml

@@ -9,7 +9,7 @@ spec:
   chart:
     spec:
       chart: kube-prometheus-stack
-      version: 86.3.2
+      version: 92.9.0
       sourceRef:
         kind: HelmRepository
         name: prometheus-community

flux/infrastructure/infra-net/helmrelease-ingress-nginx.yaml

@@ -191,6 +191,10 @@ spec:
           mountPath: /etc/nginx/static
     defaultBackend:
       enabled: true
+      resources:
+        requests:
+          cpu: 10m
+          memory: 512Mi
       image:
         registry: docker.io
         image: devcm/default-backend