PC's Blog

Life is short, Play more!

0%

VA backend Design

发表于 分类于
本文字数： 13k 阅读时长 ≈ 12 分钟

Overview

1
2
3
4
5
6
7
frontend ------->  backend ---------> MQ ----------->  VA(On-premise)
                       /|\                             |  
                        |------------------------------|
						
peter.backend.com:443  backend
peter.channel.com:5672 MQ
peter.frontend.com     frontend

Design

Auto-upgrade

dispatchAutoUpgradeInOneHourForAll

  • find all running appliances, group by customer_id
  • check if appliance need upgrade
    • check schedule update time is within next 1 hour.
    • check target version valid (expectedStatus = RUNNING and status is not RUNNING)
    • don’t upgrade rollback appliances.
    • don’t upgrade all appliances at the same time. upgrade half appliances first.
  • trigger appliance upgrade
    • get firmware download link
    • update db, add dbentry for IotTask, update appliance status(upgrading, upgradetaskId: taskId)
    • publish iot Task
  • receive response from Appliance
    • write task to db

Heartbeat

appliance断连一小段时间后，连接又恢复正常
appliances长时间断连，如何检测
receive response

1
update heartbeat iot task to db

check heartbeat per 5 minutes

  • find all applinace in ApplianceInfo Table.

Test API

API for frontend

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
curl -H "x-customer-id: 77a1c9561c8d47fb8036c970a4f2ee73" "https://peter.backend.com/ui/appliances/?start=1736680946&applianceId=974bf535-7930-474e-8da6-780cafff284d"
curl -H "x-customer-id: 77a1c9561c8d47fb8036c970a4f2ee73" "https://peter.backend.com/ui/appliances/?start=1736680946"
curl https://peter.backend.com/ui/appliances/image/
curl -X DELETE https://peter.backend.com/ui/appliances/974bf535-7930-474e-8da6-780cafff284d/
curl -X POST https://peter.backend.com/ui/appliances/974bf535-7930-474e-8da6-780cafff284d/upgrade/
curl https://peter.backend.com/ui/appliances/974bf535-7930-474e-8da6-780cafff284d/settings/
curl -X POST https://peter.backend.com/ui/appliances/974bf535-7930-474e-8da6-780cafff284d/settings/
curl https://peter.backend.com/ui/appliances/974bf535-7930-474e-8da6-780cafff284d/storage/
curl -X POST https://peter.backend.com/ui/appliances/974bf535-7930-474e-8da6-780cafff284d/storage/
curl -X POST https://peter.backend.com/ui/appliances/974bf535-7930-474e-8da6-780cafff284d/log/
curl https://peter.backend.com/ui/appliances/974bf535-7930-474e-8da6-780cafff284d/services/

curl -X POST https://peter.backend.com/ui/services/squid/install/
curl -X POST https://peter.backend.com/ui/services/squid/974bf535-7930-474e-8da6-780cafff284d/toggle/
curl https://peter.backend.com/ui/services/squid/974bf535-7930-474e-8da6-780cafff284d/settings/
curl -X POST https://peter.backend.com/ui/services/squid/974bf535-7930-474e-8da6-780cafff284d/settings/

curl https://peter.backend.com/ui/tasks/12345/

API for VA

1
2
3
4
5
6
curl -X POST -H "Authorization: Bearer $token" -i https://peter.backend.com/va/register/
curl -X POST https://peter.backend.com/va/register/
curl -X POST https://peter.backend.com/va/974bf535-7930-474e-8da6-780cafff284d/tasks/
curl https://peter.backend.com/va/974bf535-7930-474e-8da6-780cafff284d/healthz/
curl -X POST https://peter.backend.com/va/974bf535-7930-474e-8da6-780cafff284d/log/
curl -X POST https://peter.backend.com/va/974bf535-7930-474e-8da6-780cafff284d/squid/metrics/

backend Design

Database Design

Customer表

1
2
3
4
5
6
7
8
9
desc app_customer;
+----------------+-------------+------+-----+---------+-------+
| Field          | Type        | Null | Key | Default | Extra |
+----------------+-------------+------+-----+---------+-------+
| id             | char(32)    | NO   | PRI | NULL    |       |
| register_token | longtext    | NO   |     | NULL    |       |
| created_at     | datetime(6) | NO   |     | NULL    |       |
| updated_at     | datetime(6) | NO   |     | NULL    |       |
+----------------+-------------+------+-----+---------+-------+

ApplianceInfo表

1
2
3
4
5
6
7
8
9
10
11
12
13
desc app_applianceinfo;
+--------------------+-------------+------+-----+----------------+-------------------+
| Field              | Type        | Null | Key | Default        | Extra             |
+--------------------+-------------+------+-----+----------------+-------------------+
| id                 | char(32)    | NO   | PRI | NULL           |                   |
| customer_id        | char(32)    | NO   | MUL | NULL           |                   |
| created_at         | datetime(6) | NO   |     | NULL           |                   |
| updated_at         | datetime(6) | NO   |     | NULL           |                   |
| version            | varchar(16) | NO   |     | NULL           |                   |
| status             | varchar(16) | NO   |     | NULL           |                   |
| appliance_settings | json        | NO   |     | NULL           |                   |
| capacity           | json        | NO   |     | _utf8mb3\'{}\' | DEFAULT_GENERATED |
+--------------------+-------------+------+-----+----------------+-------------------+

IotTask表

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
> desc app_iottask;
+---------------+-------------+------+-----+---------+-------+
| Field         | Type        | Null | Key | Default | Extra |
+---------------+-------------+------+-----+---------+-------+
| id            | char(32)    | NO   | PRI | NULL    |       |
| created_at    | datetime(6) | NO   | MUL | NULL    |       |
| updated_at    | datetime(6) | NO   |     | NULL    |       |
| task_type     | varchar(50) | NO   | MUL | NULL    |       |
| appliance_id  | char(32)    | NO   | MUL | NULL    |       |
| customer_id   | char(32)    | NO   |     | NULL    |       |
| message       | json        | NO   |     | NULL    |       |
| status        | varchar(16) | NO   | MUL | NULL    |       |
| retry_count   | int         | NO   |     | NULL    |       |
| result        | json        | NO   |     | NULL    |       |
| error_message | longtext    | NO   |     | NULL    |       |
+---------------+-------------+------+-----+---------+-------+

ApplianceVersion表

1
2
3
4
5
6
7
8
9
10
11
12
13
14
mysql> desc app_applianceversion;
+------------------+-------------+------+-----+---------+-------+
| Field            | Type        | Null | Key | Default | Extra |
+------------------+-------------+------+-----+---------+-------+
| version          | varchar(32) | NO   | PRI | NULL    |       |
| created_at       | datetime(6) | NO   |     | NULL    |       |
| updated_at       | datetime(6) | NO   |     | NULL    |       |
| version_major    | int         | NO   |     | NULL    |       |
| version_minor    | int         | NO   |     | NULL    |       |
| version_revision | int         | NO   |     | NULL    |       |
| version_build    | int         | NO   |     | NULL    |       |
| display_version  | varchar(32) | NO   |     | NULL    |       |
| firmware_info    | json        | NO   |     | NULL    |       |
+------------------+-------------+------+-----+---------+-------+

HeartbeatIotTask表

1
2
3
4
5
6
7
8
9
10
11
12
13
14
desc app_heartbeatiottask;
+---------------+-------------+------+-----+---------+----------------+
| Field         | Type        | Null | Key | Default | Extra          |
+---------------+-------------+------+-----+---------+----------------+
| id            | bigint      | NO   | PRI | NULL    | auto_increment |
| created_at    | datetime(6) | NO   | MUL | NULL    |                |
| updated_at    | datetime(6) | NO   |     | NULL    |                |
| appliance_id  | char(32)    | NO   | MUL | NULL    |                |
| customer_id   | char(32)    | NO   |     | NULL    |                |
| message       | json        | NO   |     | NULL    |                |
| status        | varchar(16) | NO   | MUL | NULL    |                |
| result        | json        | NO   |     | NULL    |                |
| error_message | longtext    | NO   |     | NULL    |                |
+---------------+-------------+------+-----+---------+----------------+

IotTask Design

  • install/uninstall/enable/disable/configure services
  • upgrade firmware/services
  • collect metrics about va and services
    • va’s cpu,memory, storage, network usage
    • maintain service status(running, disabled, disconnected)

  • heartbeart
  • collect debug logs
  • collect metrics
  • extend storage
  • change api key and server ssl certificates
  • remote shell

HeartBeat

use applianceId as taskId to avoid database overwhelm
Heartbeat Body 

1
2
3
4
5
{
	"taskId": applianceId,
	"taskType": 
	"serverTime": 1736588457
}

Heartbeat Result

1

Upgrade Appliance

1
2
3
4
5
6
7
{
	"taskId": "uuid",
	"taskType": "upgradeAppliance",
	"targetVersion": "1.0.0.100",
	"firmwarePath": "https://XXX/",
	"firmwareSha256": "XXX"
}

Install services

1
2
3
{
	
}

Collect log

1
2
3
4
5
{
	"start": "2021-01-01",
	"end": "2021-02-02",
	"uploadPath": ""
}

Collect Appliance Metrics

backend API design

API for VA

VA register

POST /va/register

  • VA发起注册请求, Header中携带JWT
  • 校验请求头的JWT
    • 解析JWT中的customer_id, 查数据库是否存在该customer
    • 检查JWT是否过期，如果过期就为客户重新生成一个新的JWT，继续校验
  • 从请求体中读取VA信息，将VA信息写入数据库
  • 响应VA，返回(iotHost, iotCert, applianceId)给VA。

Response Body

1
2
3
4
5
6
7
{
	"applianceId": "uuid",
	"applianceToken": "",
	"iotHost": "peter.channel.com"
	"iotPort": 5672
	"message": "error message..." 
}

Update IOT Task Results

POST /va/{appliance_id}/tasks

Request Body

1
2
3
4
5
6
7
8
{
	"taskId": "uuid",
	"taskStatus": "success | failed | ignored",
	"errorMessage": "",
	"taskResult": {
		"field1": ...
	}
}

IOT Health Check

GET /va/{appliance_id}/healthz

Upload Service Metrics

POST /va/{appliance_id}/{service_code>/metrics

Upload Audit Log

POST /var/{appliance_id}/log

API for frontend

Appliance Management

Get Appliance Image Info

GET /ui/appliances/image

List Appliance

GET /ui/appliances

  • get customer_id from request header ‘x-customer-id’
  • check if customer exists in db
  • get applianceId from request params
  • return appliances which expectedStatus are not unregistered.

Request Params

param optional example description
start True 1700020200 filter register time of appliances
applianceId True uuid

Response

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
{
	"data": [
		{
			"applianceId": uuid,
			"hostname": "localhost",
			"ipv4Address": "1.2.3.4",
			"status": "preparing | running | disconnected | upgrading",
			"upgradeStatus": "downloading | upgrading",		// optional
			"logCollectStatus": "packaging | uploading",
			"version": "2.0.0",
			"installedServiceList": [
				{
					"serviceCode": "squid",
					"version": "1.1.1",
					"healthStatus": "healthy | unhealthy",
					"fullName": "Squid",
					"description": "Forward Proxy",
				},
			],
			"lastConnectedTime": 1700020200,
			"upgradeStartedTime": 1700020200, // if upgradeStartedTime exists, appliance is upgrading
			"totalStorage": 1024576,
			"usedStorage": 10240,
			"totalCpu": 8,
			"totalMemory": 16384
		}
	]
}

Delete Appliance

DELETE /ui/appliances/

Upgrade Appliance

POST /ui/appliances//upgrade

Get Appliance Settings

GET /ui/appliances//settings

Modify Appliance Settings

POST /ui/appliances//settings

Get Storage Details

GET /ui/appliances//storage

Get Appliance Metrics

POST /ui/appliances//metrics

Collect Appliance Log

POST /ui/appliances//log

Service Management

List Service

GET /ui/appliances//services

Install/Uninstall/Upgrade Service

POST /ui/services//install
Install Service Request

1
2
3
4
{
	"applianceId": "",
	"action": "install | uninstall | upgrade"
}

Enable/Disable Service

POST /ui/services///toggle

Get Service Settings

GET /ui/services///settings

Configure Service Settings

POST /ui/services///settings

Query Task result

GET /ui/tasks/

IOT(RabbitMQ)

RabbitMQ命令行操作

查看队列

1
2
3
4
5
6
# rabbitmqadmin --username=admin --password=V2SG@xdr list queues name messages
+---------------+----------+----------+
|     name      | messages | va_12345 |
+---------------+----------+----------+
| va_task_12345 | 2        |          |
+---------------+----------+----------+

删除队列

1
rabbitmqadmin --username=admin --password=V2SG@xdr delete queue name=va_task_12345

删除交换机

1
rabbitmqadmin --username=admin --password=V2SG@xdr delete exchange name=va_task --vhost=/