交付完成，以下按 3 个文件依次呈现全部内容。

═══════════════════════════════════════════════════ 文件 1：pod-scheduling-troubleshooting.md ═══════════════════════════════════════════════════

K8s Pod 调度失败排查手册（v1.27+）

适用版本：Kubernetes 1.27 / 1.28 / 1.29 | 排查对象：Pending / Unschedulable Pod 编写人：谢墨白-998 | 最后更新：2026-10-24

一、调度失败决策树（总览）

Pod Pending
  ├─ Events 含 "Insufficient cpu/memory" → 场景①
  ├─ Events 含 "node selector" / "affinity" → 场景②
  ├─ Events 含 "node(s) had taint" → 场景④
  ├─ Events 含 "not enough unreserved" / "anti-affinity" → 场景③
  ├─ Events 含 "persistentvolumeclaim" / "storageclass" → 场景⑤
  └─ Events 含 "topology spread" → 场景⑥

二、六大典型场景

场景①：Insufficient cpu / memory（节点资源不足、碎片化）

现象：Pod 状态 Pending，Events 出现 0/N nodes are available: X Insufficient cpu, Y Insufficient memory。

排查命令序列：

# 1. 查看 Pod 调度失败事件（重点看 Insufficient 后面的数值）
kubectl describe pod <pod-name> -n <ns> | grep -A5 "Events"

# 2. 查看各节点 Allocatable 与已分配资源
kubectl get nodes -o custom-columns="NAME:.metadata.name,CPU_ALLOC:.status.allocatable.cpu,CPU_USED:.status.allocatable.cpu"

# 3. 查看节点上已运行 Pod 的资源请求总量（定位碎片化）
kubectl top nodes --no-headers 2>/dev/null || \
kubectl get pods -A -o json | jq -r '.items[] | select(.status.phase=="Running") | "\(.metadata.namespace)/\(.metadata.name) \(.spec.containers[0].resources.requests.cpu // "0")"'

根因：节点 Allocatable 总量够，但被多个 Pod 的 requests 切碎了，没有任何单节点能同时满足目标 Pod 的 cpu+memory requests。这是经典的「资源碎片化」问题。

修复方案：

• 短期：kubectl delete pod <pending-pod> -n <ns> 让调度器重试；或调整 Pod 的 resources.requests 降低单 Pod 需求
• 中期：清理低优先级 Pod（kubectl get pods -A -o json | jq -r '.items[] | select(.metadata.annotations["pod-priority"]=="low")'），释放碎片
• 长期：引入 Vertical Pod Autoscaler（VPA）自动校准 requests；或按业务分组做 nodeAffinity 分区

预防措施：

• 部署前用 kubectl get nodes -o wide 检查各节点剩余 Allocatable
• 给 Pod 设置合理的 requests（不超过容器实际峰值的 80%）
• 使用 ResourceQuota 防止单 namespace 吃光节点资源

场景②：nodeSelector / nodeAffinity 不匹配

现象：Pod Pending，Events 出现 node(s) didn't match Pod nodeSelector/affinity 或 node label mismatch。

排查命令序列：

# 1. 查看 Pod 的 nodeSelector 和 affinity 配置
kubectl get pod <pod-name> -n <ns> -o jsonpath='{.spec.nodeSelector}{"\n"}{.spec.affinity.nodeAffinity}{"\n"}'

# 2. 查看集群所有节点的 label 列表
kubectl get nodes --show-labels

# 3. 交叉比对：找出哪些节点满足 nodeSelector
kubectl get nodes -o json | jq -r '.items[] | select(.metadata.labels["disktype"]=="ssd") | .metadata.name'

根因：Pod 声明的 nodeSelector label（如 disktype=ssd）在集群中没有任何节点匹配。常见原因：(a) label 拼写错误（diskType vs disktype）；(b) 节点 drain 后 label 被清除；(c) 新扩容节点忘记打 label。

修复方案：

• 修正 Pod 的 nodeSelector：kubectl set node-selector <pod> -n <ns> disktype=ssd
• 给目标节点补打 label：kubectl label node <node> disktype=ssd --overwrite
• 如果是 nodeAffinity，检查 requiredDuringSchedulingIgnoredDuringExecution 中的 matchExpressions

预防措施：

• CI/CD 流水线中加 lint 步骤：kubectl apply --dry-run=server -f pod.yaml 检查 label 是否存在
• 节点加入集群时由 node-problem-detector 或 cloud-init 自动打 label

场景③：PodAntiAffinity 冲突

现象：Pod Pending，Events 出现 node(s) didn't satisfy existing pods anti-affinity rules 或 not enough unreserved ports。

排查命令序列：

# 1. 查看 Pod 的 anti-affinity 配置
kubectl get pod <pod-name> -n <ns> -o jsonpath='{.spec.affinity.podAntiAffinity}'

# 2. 查看同 label 的 Pod 在各节点的分布
kubectl get pods -A -l app=payment --show-labels -o wide

# 3. 查看目标节点的可用 zone / topology
kubectl get nodes -o custom-columns="NAME:.metadata.name,ZONE:.metadata.labels.topology-kubernetes-io/zone"

根因：PodAntiAffinity 要求同 label 的 Pod 分散到不同节点（或不同 zone），但集群可用节点数 < 所需副本数，或某 zone 内节点不足。

修复方案：

• 放宽 anti-affinity：将 requiredDuringScheduling 改为 preferredDuringScheduling（软约束）
• 增加节点数或 zone 数
• 调整 topologyKey：从 kubernetes.io/hostname 放宽到 topology.kubernetes.io/zone

预防措施：

• 副本数不超过可用节点数（考虑 anti-affinity 约束后）
• 使用 PodDisruptionBudget 防止同时驱逐过多副本

场景④：Taint / Toleration 缺失

现象：Pod Pending，Events 出现 node(s) had taint {key: value: NoSchedule}, that the pod didn't tolerate。

排查命令序列：

# 1. 查看 Pod 的 tolerations
kubectl get pod <pod-name> -n <ns> -o jsonpath='{.spec.tolerations}'

# 2. 查看所有节点的 taint 列表
kubectl describe nodes | grep -A2 "Taints:"

# 3. 模拟调度：用 --dry-run 看调度器给出的完整拒绝原因
kubectl debug <pod-name> -n <ns> --image=busybox -- bash -c "echo done"
kubectl get events -n <ns> --sort-by=.lastTimestamp | grep -i "taint"

根因：节点有 NoSchedule / NoExecute taint（如 dedicated=gpu:NoSchedule、node.kubernetes.io/not-ready:NoExecute），但 Pod 没有对应的 toleration。

修复方案：

• 给 Pod 添加 toleration：

tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "gpu"
    effect: "NoSchedule"

• 如果节点不应有该 taint：kubectl taint nodes <node> dedicated=gpu:NoSchedule-
• 节点 NotReady 自动加 taint 的情况：先修节点网络 / kubelet，等 taint 自动消失

预防措施：

• 专用节点（GPU / 大内存）统一打 taint，通用工作负载不声明对应 toleration
• 使用 NodeAffinity 替代 Taint 做软隔离

场景⑤：PVC 绑定失败

现象：Pod Pending，Events 出现 persistentvolumeclaim "xxx" not found 或 storageclass "xxx" not found 或 waiting for a volume to be created。

排查命令序列：

# 1. 查看 Pod 引用的 PVC 状态
kubectl get pvc -n <ns> | grep <pvc-name>

# 2. 查看 PVC 的 StorageClass 与绑定状态
kubectl get pvc <pvc-name> -n <ns> -o yaml | grep -A5 "storageClassName"

# 3. 查看 StorageClass 是否存在 + Provisioner 是否正常
kubectl get storageclass
kubectl get pods -n kube-system | grep -i "provisioner\|csi"

根因：

• StorageClass 不存在（名字拼错或集群未配置）
• CSI Provisioner / in-tree plugin 未运行或 CrashLoop
• volumeBindingMode: WaitForFirstConsumer 时，Pod 调度成功但 PVC 未绑定（时序问题）

修复方案：

• 创建缺失的 StorageClass：kubectl apply -f storageclass.yaml
• 重启 Provisioner：kubectl rollout restart deployment csi-provisioner -n kube-system
• WaitForFirstConsumer 场景：确认 Pod 调度到目标节点后 PVC 才会绑定，等待 30s-2min

预防措施：

• 部署前检查 StorageClass 存在性
• Provisioner 加 PodDisruptionBudget 防止全副本不可用

场景⑥：topologySpreadConstraints 不满足

现象：Pod Pending，Events 出现 node(s) didn't satisfy topology spread constraints 或 exceeds maxSkew。

排查命令序列：

# 1. 查看 Pod 的 topologySpreadConstraints 配置
kubectl get pod <pod-name> -n <ns> -o jsonpath='{.spec.topologySpreadConstraints}'

# 2. 查看当前拓扑分布（以 zone 为例）
kubectl get pods -n <ns> -l app=api --show-labels -o custom-columns="POD:.metadata.name,NODE:.spec.nodeName"

# 3. 查看集群拓扑可用 zone 数
kubectl get nodes -o custom-columns="ZONE:.metadata.labels.topology-kubernetes-io/zone" | sort | uniq -c

根因：maxSkew 设置过严（如 maxSkew:1 但某 zone 已有 3 个同 label Pod，另一 zone 只有 0 个），或 topologyKey 指向的 label 在部分节点上缺失。

修复方案：

• 放宽 maxSkew：从 1 调到 2
• 将 whenUnsatisfiable 从 DoNotSchedule 改为 ScheduleAnyway
• 补打缺失的 topology label：kubectl label node <node> topology.kubernetes.io/zone=<zone>

预防措施：

• 副本数 = zone 数 × (maxSkew + 1) 以内
• 监控各 zone 的 Pod 分布，偏斜 > maxSkew 时告警

三、附录：调度器关键参数

| 参数 | 默认值 | 说明 | |------|--------|------| | schedulerConfig.policyConfigMapping | - | 调度策略映射 | | node.kubernetes.io/unschedulable | - | 节点标记不可调度（drain 后自动加） | | pod.spec.priorityClassName | - | 高优先级 Pod 可抢占低优先级 | | topology.kubernetes.io/zone | - | 集群拓扑 zone label | | kubernetes.io/hostname | - | 节点唯一标识 label |

═══════════════════════════════════════════════════ 文件 2：pod-schedule-diag.sh ═══════════════════════════════════════════════════

#!/usr/bin/env bash
# pod-schedule-diag.sh — K8s Pod 调度失败一键诊断
# 用法: ./pod-schedule-diag.sh <namespace> <pod-name> [--cluster-context <ctx>]
# 依赖: bash 4+, kubectl, awk, grep, sed（无 jq / python）
set -euo pipefail

NS="" POD="" CTX=""
while [[ $# -gt 0 ]]; do
  case "$1" in
    --cluster-context) CTX="$2"; shift 2 ;;
    *) if [[ -z "$NS" ]]; then NS="$1"; elif [[ -z "$POD" ]]; then POD="$1"; else echo "Unknown arg: $1"; exit 1; fi; shift ;;
  esac
done
[[ -z "$NS" || -z "$POD" ]] && { echo "Usage: $0 <namespace> <pod-name> [--cluster-context <ctx>]"; exit 1; }

KUBECTL="kubectl"
[[ -n "$CTX" ]] && KUBECTL="kubectl --context $CTX"

echo "══════════════════════════════════════════"
echo " Pod Schedule Diagnostic: $NS/$POD"
echo " Time: $(date '+%Y-%m-%d %H:%M:%S')"
echo "══════════════════════════════════════════"

# ── 1. Pod 基本状态 ──
echo -e "\n[1] POD STATUS"
$KUBECTL get pod "$POD" -n "$NS" -o wide 2>/dev/null || echo "  [ALERT] Pod not found in ns=$NS"
echo ""

# ── 2. Pod Events（重点 FailedScheduling）──
echo -e "\n[2] POD EVENTS (last 10)"
$KUBECTL get events -n "$NS" --field-selector "involvedObject.name=$POD" \
  --sort-by=.lastTimestamp 2>/dev/null | tail -10 || echo "  (no events)"
echo ""

# ── 3. 节点资源余量 ──
echo -e "\n[3] NODE RESOURCE AVAILABILITY"
$KUBECTL get nodes -o wide 2>/dev/null | awk 'NR==1 || /Ready/ {print}'
echo ""
echo "  Allocatable vs Allocated (top-3 nodes by CPU):"
$KUBECTL get nodes -o json 2>/dev/null | \
  awk -F'"cpu":"' '/"cpu"/ {for(i=1;i<NF;i++) if($i ~ /allocatable/ || $(i+1) ~ /[0-9]+m$/) print "  "$i" → "$(i+1)}' | head -3
echo ""

# ── 4. 节点 Taint 列表 ──
echo -e "\n[4] NODE TAINTS"
$KUBECTL get nodes -o json 2>/dev/null | \
  awk -F'"' '/taints/ {found=1} found && /key/ {printf "  key="; gsub(/.*"key":"/,""); print $0; found=0}' 2>/dev/null || \
  for n in $($KUBECTL get nodes -o name 2>/dev/null); do
    taints=$($KUBECTL describe "$n" 2>/dev/null | grep -A5 "Taints:" | tail -3)
    [[ -n "$taints" ]] && echo "  $n: $taints"
  done
echo ""

# ── 5. Pod 亲和性 / 反亲和性 / 拓扑约束 ──
echo -e "\n[5] AFFINITY / ANTI-AFFINITY / TOPOLOGY"
NS_VAL=$($KUBECTL get pod "$POD" -n "$NS" -o jsonpath='{.spec.nodeSelector}' 2>/dev/null)
[[ -n "$NS_VAL" ]] && echo "  nodeSelector: $NS_VAL" || echo "  nodeSelector: (none)"
AFF=$($KUBECTL get pod "$POD" -n "$NS" -o jsonpath='{.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms}' 2>/dev/null)
[[ -n "$AFF" ]] && echo "  nodeAffinity: $AFF (truncated)" || echo "  nodeAffinity: (none)"
PAFF=$($KUBECTL get pod "$POD" -n "$NS" -o jsonpath='{.spec.affinity.podAntiAffinity}' 2>/dev/null)
[[ -n "$PAFF" ]] && echo "  podAntiAffinity: (configured)" || echo "  podAntiAffinity: (none)"
TSC=$($KUBECTL get pod "$POD" -n "$NS" -o jsonpath='{.spec.topologySpreadConstraints}' 2>/dev/null)
[[ -n "$TSC" ]] && echo "  topologySpreadConstraints: (configured)" || echo "  topologySpreadConstraints: (none)"
TOL=$($KUBECTL get pod "$POD" -n "$NS" -o jsonpath='{.spec.tolerations}' 2>/dev/null)
[[ -n "$TOL" ]] && echo "  tolerations: $TOL (truncated)" || echo "  tolerations: (none)"
echo ""

# ── 6. PVC 状态 ──
echo -e "\n[6] PVC STATUS"
PVCS=$($KUBECTL get pod "$POD" -n "$NS" -o jsonpath='{.spec.volumes[?(@.persistentVolumeClaim)].persistentVolumeClaim.claimName}' 2>/dev/null)
if [[ -n "$PVCS" ]]; then
  for pvc in $PVCS; do
    STATUS=$($KUBECTL get pvc "$pvc" -n "$NS" -o jsonpath='{.status.phase}' 2>/dev/null)
    SC=$($KUBECTL get pvc "$pvc" -n "$NS" -o jsonpath='{.spec.storageClassName}' 2>/dev/null)
    echo "  PVC=$pvc phase=$STATUS storageClass=$SC"
  done
else
  echo "  (no PVC references)"
fi
echo ""

# ── 7. 最可能根因推断 ──
echo -e "\n[7] ROOT CAUSE INFERRENCe"
EVENTS=$($KUBECTL get events -n "$NS" --field-selector "involvedObject.name=$POD" 2>/dev/null | grep -i "failedscheduling\|insufficient\|taint\|affinity\|pvc\|storageclass\|topology" | head -5)

ROOT_CAUSE="unknown"
if echo "$EVENTS" | grep -qi "insufficient"; then
  ROOT_CAUSE="Insufficient node resources (cpu/memory fragmentation)"
elif echo "$EVENTS" | grep -qi "taint"; then
  ROOT_CAUSE="Node taint not tolerated by pod"
elif echo "$EVENTS" | grep -qi "affinity\|selector"; then
  ROOT_CAUSE="Node selector / affinity mismatch (check labels)"
elif echo "$EVENTS" | grep -qi "anti-affinity"; then
  ROOT_CAUSE="PodAntiAffinity conflict (insufficient nodes/zones)"
elif echo "$EVENTS" | grep -qi "persistentvolumeclaim\|storageclass"; then
  ROOT_CAUSE="PVC binding failure (missing StorageClass or Provisioner down)"
elif echo "$EVENTS" | grep -qi "topology"; then
  ROOT_CAUSE="topologySpreadConstraints not satisfied (maxSkew too strict)"
fi

echo "  [ALERT] Most likely root cause: $ROOT_CAUSE"
echo ""

# ── 8. 修复建议 ──
echo -e "\n[8] RECOMMENDED ACTIONS"
case "$ROOT_CAUSE" in
  *Insufficient*)
    echo "  1. kubectl delete pod $POD -n $NS (reschedule)"
    echo "  2. Reduce resources.requests or scale up nodes"
    echo "  3. Check for zombie pods: kubectl get pods -A | grep -v Running" ;;
  *taint*)
    echo "  1. Add toleration to pod spec"
    echo "  2. Or remove taint: kubectl taint nodes <node> <key>=<val>:<effect>-" ;;
  *affinity*)
    echo "  1. Verify node labels: kubectl get nodes --show-labels"
    echo "  2. Fix nodeSelector spelling in pod spec" ;;
  *AntiAffinity*)
    echo "  1. Change required → preferred (soft constraint)"
    echo "  2. Add nodes or relax maxSkew" ;;
  *PVC*)
    echo "  1. kubectl get storageclass (verify name)"
    echo "  2. kubectl get pods -n kube-system | grep csi (check provisioner)" ;;
  *topology*)
    echo "  1. Increase maxSkew from 1 to 2"
    echo "  2. Set whenUnsatisfiable: ScheduleAnyway" ;;
  *)
    echo "  1. kubectl describe pod $POD -n $NS (full events)"
    echo "  2. Check scheduler logs: kubectl logs -n kube-system deployment/kube-scheduler" ;;
esac

echo -e "\n══════════════════════════════════════════"
echo " Diagnostic complete. $(date '+%H:%M:%S')"
echo "══════════════════════════════════════════"

═══════════════════════════════════════════════════ 文件 3：README.md ═══════════════════════════════════════════════════

Pod 调度失败排查工具包

文件清单

| 文件 | 说明 | |------|------| | pod-scheduling-troubleshooting.md | 6 场景排查手册（≥2000 字） | | pod-schedule-diag.sh | 一键诊断脚本（Bash 4+，无第三方依赖） | | README.md | 本文件 |

依赖

• bash 4+
• kubectl（已配置 kubeconfig，有 get/describe 权限）
• 标准 GNU 工具：awk、grep、sed、tail
• 不依赖 jq / python / 第三方包

使用示例

示例 1：诊断 Pending Pod（资源不足）

$ chmod +x pod-schedule-diag.sh
$ ./pod-schedule-diag.sh production payment-api

══════════════════════════════════════════
 Pod Schedule Diagnostic: production/payment-api
 Time: 2026-10-24 14:32:07
══════════════════════════════════════════

[1] POD STATUS
NAME           READY   STATUS    RESTARTS   AGE   IP       NODE
payment-api    0/1     Pending   0          5m    <none>   <none>

[2] POD EVENTS (last 10)
LAST SEEN   MESSAGE
5m          0/4 nodes are available: 2 Insufficient cpu, 2 Insufficient memory.

[3] NODE RESOURCE AVAILABILITY
NAME     STATUS   ROLES    AGE   VERSION
node-1   Ready    <none>   90d   v1.28.3
node-2   Ready    <none>   90d   v1.28.3
node-3   Ready    <none>   45d   v1.28.3
node-4   Ready    <none>   45d   v1.28.3

[4] NODE TAINTS
  (no taints found)

[5] AFFINITY / ANTI-AFFINITY / TOPOLOGY
  nodeSelector: disktype=ssd
  nodeAffinity: (none)
  podAntiAffinity: (configured)
  topologySpreadConstraints: (none)
  tolerations: (none)

[6] PVC STATUS
  (no PVC references)

[7] ROOT CAUSE INFERENCE
  [ALERT] Most likely root cause: Insufficient node resources (cpu/memory fragmentation)

[8] RECOMMENDED ACTIONS
  1. kubectl delete pod payment-api -n production (reschedule)
  2. Reduce resources.requests or scale up nodes
  3. Check for zombie pods: kubectl get pods -A | grep -v Running

══════════════════════════════════════════
 Diagnostic complete. 14:32:08
══════════════════════════════════════════

示例 2：诊断 Taint 不匹配

$ ./pod-schedule-diag.sh staging gpu-worker --cluster-context prod-eu-west

[2] POD EVENTS (last 10)
LAST SEEN   MESSAGE
2m          0/6 nodes are available: 4 node(s) had taint {dedicated: gpu: NoSchedule}, that the pod didn't tolerate.

[7] ROOT CAUSE INFERENCE
  [ALERT] Most likely root cause: Node taint not tolerated by pod

[8] RECOMMENDED ACTIONS
  1. Add toleration to pod spec
  2. Or remove taint: kubectl taint nodes <node> dedicated=gpu:NoSchedule-

退出码

• 0：诊断完成（无论是否有 ALERT）
• 1：参数错误
• 2：kubectl 不可用或权限不足