PC's Blog
0%

SeaweedFS 3.97 S3 POST 大文件上传触发 504 超时问题定位

发表于 分类于
本文字数： 5k 阅读时长 ≈ 5 分钟

问题描述

用户通过 S3 POST 预签名 URL 向 SeaweedFS 3.97 上传 1GB 大文件时，APISIX 反向代理返回 504 Gateway Timeout 错误，但文件最终完整写入 SeaweedFS 集群。文件上传请求链路如下:

1
用户 (上传文件) -----> APISIX (TLS 卸载) ------> Seaweedfs S3 ----> Seaweedfs Filer ----> Seaweedfs Volume

定位步骤

1. 复现问题

通过 curl 模拟 1GB 文件上传，结合 tcpdump 抓包分析

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# tcpdump抓包，192.168.42.210:8333为SeaweedFS S3服务地址
tcpdump -i ens33 -nn host 192.168.42.210 and port 8333 -w file_upload.pcap

# curl模拟1GB文件上传（携带预签名参数）
curl -v -X POST \
    -H "Content-MD5: 6c37b5cd511598caa5125bed043ee32b" \
    -F "key=1.0.0/s3testfile_intranet.txt" \
    -F "X-Amz-Algorithm=AWS4-HMAC-SHA256" \
    -F "X-Amz-Credential=7D9C2ZGHK71552W09G8I/20260205/us-east-1/s3/aws4_request" \
    -F "X-Amz-Date=20260205T023814Z" \
    -F "policy=eyJjb25kaXRpb25zIjpbeyJidWNrZXQiOiJmaXJtd2FyZSJ9LFsic3RhcnRzLXdpdGgiLCIka2V5IiwiMS4wLjAvczN0ZXN0ZmlsZV9pbnRyYW5ldC50eHQiXSx7IngtYW16LWFsZ29yaXRobSI6IkFXUzQtSE1BQy1TSEEyNTYifSx7IngtYW16LWNyZWRlbnRpYWwiOiI3RDlDMlpHSEs3MTU1MlcwOUc4SS8yMDI2MDIwNS91cy1lYXN0LTEvczMvYXdzNF9yZXF1ZXN0In0seyJ4LWFtei1kYXRlIjoiMjAyNjAyMDVUMDIzODE0WiJ9XSwiZXhwaXJhdGlvbiI6IjIwMjYtMDItMDVUMDM6Mzg6MTRaIn0=" \
    -F "X-Amz-Signature=127f6e136813102db911108d4e7efcf85395f16996c31aa1b5f721d0c564014b" \
    -F "file=@/home/shr/largefile_1G.bin" \
    "http://192.168.42.210:8333/firmware"

2. Wireshark 抓包分析

  • 连接建立：客户端与 S3 服务完成 TCP 三次握手，客户端发送 100 Continue 报文确认准备上传，链路通信正常
  • 数据传输：客户端通过大量 PSH, ACK 报文完成 1GB 文件数据传输 (PSH,ACK 报文终止仅代表内核态 TCP 接收缓冲区已接收完所有数据，并非 SeaweedFS 用户态处理完成)
  • 关键延迟：从数据传输完成（无新 PSH,ACK 报文）到 SeaweedFS 返回 204 No Content，耗时长达 11 秒
  • 连接关闭：SeaweedFS 返回响应后，客户端与服务端发起 TCP 四次挥手，连接正常关闭

3. 日志分析

查看SeaweedFS filer日志，uploadReaderToChunks函数处理 250 个 4M 分片耗时 11 秒。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# filer组件日志
Feb 05 13:55:19 dev weed[154132]: I0205 13:55:19.326575 filer_server_handlers_write_upload.go:138 request_id:5958aa93-5d65-436a-a7df-cb855e85da2a [PC] uploaded s3testfile_intranet.txt chunk 1 to 4,02feacf06479 [0,4194304)
Feb 05 13:55:19 dev weed[154132]: I0205 13:55:19.452418 filer_server_handlers_write_upload.go:138 request_id:5958aa93-5d65-436a-a7df-cb855e85da2a [PC] uploaded s3testfile_intranet.txt chunk 2 to 4,02ff0696942e [4194304,8388608)

Feb 05 13:55:30 dev weed[154132]: I0205 13:55:30.334513 filer_server_handlers_write_upload.go:138 request_id:5958aa93-5d65-436a-a7df-cb855e85da2a [PC] uploaded s3testfile_intranet.txt chunk 249 to 3,03f6e33e9b5b [1040187392,1044381696)
Feb 05 13:55:30 dev weed[154132]: I0205 13:55:30.378993 filer_server_handlers_write_upload.go:138 request_id:5958aa93-5d65-436a-a7df-cb855e85da2a [PC] uploaded s3testfile_intranet.txt chunk 250 to 3,03f7d6ae9583 [1044381696,1048576000)

Feb 05 13:55:30 dev weed[154132]: I0205 13:55:30.379105 filer_server_handlers_write_autochunk.go:151 request_id:5958aa93-5d65-436a-a7df-cb855e85da2a [PC_DEBUG] doPutAutoChunk Done, begin to save meta data
Feb 05 13:55:30 dev weed[154132]: I0205 13:55:30.398419 filer_server_handlers_write_autochunk.go:290 request_id:5958aa93-5d65-436a-a7df-cb855e85da2a saving /buckets/firmware/1.0.0/s3testfile_intranet.txt
Feb 05 13:55:30 dev weed[154132]: I0205 13:55:30.400166 filer.go:233 request_id:5958aa93-5d65-436a-a7df-cb855e85da2a UpdateEntry /buckets/firmware/1.0.0/s3testfile_intranet.txt: old entry: s3testfile_intranet.txt
Feb 05 13:55:30 dev weed[154132]: I0205 13:55:30.407728 log_read.go:60 localMeta:filer:192.168.42.210:8888@192.168.42.210:27892 ReadFromBuffer at 2026-02-05 05:54:11.522595195 +0000 UTC batch 6. Read bytes 34628 batch 7
Feb 05 13:55:30 dev weed[154132]: I0205 13:55:30.408702 filer.go:244 request_id:5958aa93-5d65-436a-a7df-cb855e85da2a CreateEntry /buckets/firmware/1.0.0/s3testfile_intranet.txt: created
Feb 05 13:55:30 dev weed[154132]: I0205 13:55:30.409047 filer_server_handlers_write_autochunk.go:156 request_id:5958aa93-5d65-436a-a7df-cb855e85da2a [PC_DEBUG] doPutAutoChunk save meta data done

# s3组件日志
Feb 05 13:55:10 dev weed[154316]: I0205 13:55:10.404160 s3api_object_handlers_postpolicy.go:29 PostPolicyBucketHandler firmware
Feb 05 13:55:19 dev weed[154316]: I0205 13:55:19.172726 s3api_object_handlers_put.go:246 putToFiler: no explicit encryption detected, checking for bucket default encryption
Feb 05 13:55:19 dev weed[154316]: I0205 13:55:19.173545 s3api_object_handlers_put.go:323 putToFiler: no SSE-KMS encryption detected
Feb 05 13:55:30 dev weed[154316]: I0205 13:55:30.410215 error_handler.go:121 status 204 :

4. 原因分析

APISIX upstream 读超时配置为 6 秒，而 SeaweedFS 内核态接收完数据后，用户态通过uploadReaderToChunks处理 1GB 文件分片（250 个 4M 分片）并返回响应耗时 11 秒，超出阈值触发 504。因内核态已缓存全量数据，用户态后续完成分片处理，最终文件完整落地。

SeaweedFS 内部调用链

  • S3 层：PostPolicyBucketHandler（s3api_object_handlers_postpolicy.go）→ putToFiler → client.Do (proxyReq)，负责解析请求、校验签名，转发文件流至 Filer；
  • Filer 层：filerHandler → PostHandler → autoChunk → uploadReaderToChunks（核心耗时）→ saveMetaData，其中uploadReaderToChunks将文件按 4M 分片并发上传至 Volume，需等待所有分片上传完成，此过程耗时 11 秒。

解决方法

将 APISIX upstream 的 read_timeout从 6 秒调整为 60 秒，问题解决。