[2026-04-25 15:37:11.188912] [fail] - 用例test_adu_trigger_event失败, 失败原因：record_auto进程未运行

[2026-04-25 15:37:11.915934] [fail] - 用例test_aduout_check_services失败, 失败原因：有服务未在运行: ['cowa.service']

[2026-04-25 15:37:12.901864] [fail] - 用例test_aduout_check_process失败, 失败原因：有进程未在运行: ['camera_qrcode', 'cowa.sh', 'node_server', 'record_auto', 'system_monitor', 'foxglove', 'crbag record -a', 'adu_system', 'runuser -u']

[2026-04-25 15:37:15.838181] [fail] - 用例test_adu_crload失败, 失败原因：adu crload及yaml有进程未在运行: ['fakeca.*drive.yaml', 'hmi.*drive.yaml', 'hub5.*drive.yaml', 'lidar.*drive.yaml', 'lidar-.*drive.yaml', 'svbox.*drive.yaml', 'zmq.*drive.yaml', 'channel_monitor.*health.yaml', 'crload.*VoyanceService_1.yaml', 'crload.*SafetyService.yaml', 'crload.*CameraConvert.yaml', 'loc.*location.yaml', 'odom-l.*location.yaml', 'slam-l.*location.yaml', 'bev.*perception.yaml', 'freesp.*perception.yaml', 'lidar.*perception.yaml', 'mosaic.*perception.yaml', 'predic.*perception.yaml', 'svedge.*perception.yaml', 'tldet.*perception.yaml', 'track.*perception.yaml', 'unifor.*perception.yaml', 'work.*perception.yaml', 'camabn.*preception_aux.yaml', 'aeb.*pnc.yaml', 'contro.*pnc.yaml', 'crload.*pnc.yaml', 'operat.*pnc.yaml', 'planni.*pnc.yaml', 'crload.*car_recorder.yaml']

[2026-04-25 15:37:37.778357] [fail] - 用例test_adu_check_exists_modelbin_modelcfg失败, 失败原因：['/opt/cowa/crpilot/share/model/sv_charge_keypoint文件夹内model.cfg缺失', '/opt/cowa/crpilot/share/model/camera_digit_rec文件夹内model.cfg缺失', '/opt/cowa/crpilot/share/model/camera_brush_seg文件夹内model.cfg缺失', '/opt/cowa/crpilot/share/model/sv_charge_bbox文件夹内model.cfg缺失', '/opt/cowa/crpilot/share/model/camera_dynamic_1280文件夹内model.cfg缺失', '/opt/cowa/crpilot/share/model/calibration_quality文件夹内model.cfg缺失', '/opt/cowa/crpilot/share/model/feature_extract文件夹内model.cfg缺失', '/opt/cowa/crpilot/share/model/feature_match文件夹内model.cfg缺失', '/opt/cowa/crpilot/share/model/face_detect文件夹内model.cfg缺失', '/opt/cowa/crpilot/share/model/camera_roadedge文件夹内model.cfg缺失', '/opt/cowa/crpilot/share/model/plate_detect文件夹内model.cfg缺失', '/opt/cowa/crpilot/share/model/camera_dynamic文件夹内model.cfg缺失', '/opt/cowa/crpilot/share/model/camera_tl_post文件夹内model.cfg缺失', '/opt/cowa/crpilot/share/model/camera_object文件夹内model.cfg缺失', '/opt/cowa/crpilot/share/model/tiny_panoramic_seg文件夹内model.cfg缺失', '/opt/cowa/crpilot/share/model/face_arcface文件夹内model.cfg缺失']

[2026-04-25 15:37:48.530536] [fail] - 用例test_adu_check_model_info_result失败, 失败原因：model.cfg比对结果不一致\n\n['【sv_edge_continuous/model.cfg内容】\\nurl: http://crpilot.sandbox.cowarobot.com/inference/sv_edge_continuous.v2.tar.gz\nsize: [[[1, 320, 800, 3], [1, 4, 64, 80, 200], [1, 4, 40, 100]]]\nsize_min: [[[1, 320, 800, 3], [1, 4, 64, 80, 200], [1, 4, 40, 100]]]\nsize_max: [[[1, 320, 800, 3], [1, 4, 64, 80, 200], [1, 4, 40, 100]]]\nplugin_libs: [libimplugins.so]\nmodel: share/model/sv_edge_continuous/model.bin\\n', '【sv_edge_continuous/model_info输出内容】\\n\\n', '【camera_tl_e2e/model.cfg内容】\\nurl: http://crpilot.sandbox.cowarobot.com/inference/camera_tl_e2e.v8.tar.gz\nsize: [[[16, 128, 128, 3], [1, 16, 4], [1, 16, 11], [24, 1, 512]]]\nsize_min: [[[16, 128, 128, 3], [1, 16, 4], [1, 16, 11], [24, 1, 512]]]\nsize_max: [[[16, 128, 128, 3], [1, 16, 4], [1, 16, 11], [24, 1, 512]]]\nplugin_libs: [libimplugins.so]\nforce_fp32: true\nmodel: share/model/camera_tl_e2e/model.bin\\n', '【camera_tl_e2e/model_info输出内容】\\n\\n']

[2026-04-25 15:37:51.493287] [fail] - 用例test_tbox_network_out失败, 失败原因：不能 ping 通千寻服务器：203.107.45.154

[2026-04-25 15:38:09.837818] [fail] - 用例test_adu_hmi_port失败, 失败原因：20秒内未检测到 hmi 8888端口占用

[2026-04-25 15:38:11.183410] [fail] - 用例test_adu_dmesg_oom失败, 失败原因：dmesg日志有内存溢出日志: [六 4月 25 15:36:55 2026] Out of memory: Killed process 7566 (crload) total-vm:63177180kB, anon-rss:52436516kB, file-rss:7680kB, shmem-rss:0kB, UID:0 pgtables:104820kB oom_score_adj:0

[2026-04-25 15:38:12.822784] [fail] - 用例test_adu_log_oom失败, 失败原因：['/var/log/kern.log', '/var/log/syslog']文件有内存溢出日志

[2026-04-25 15:38:19.014677] [fail] - 用例test_adu_log_pole失败, 失败原因：以下桩检查失败：['ADUAlive桩未触发', 'ADUOn桩未触发']

[2026-04-25 15:40:17.578798] [fail] - 用例test_tbox_log_oom失败, 失败原因：['/var/log/crpilot/ipmi.log']文件有内存溢出日志

[2026-04-25 15:40:29.126455] [fail] - 用例test_adu_check_coredump失败, 失败原因：coredump设置不正确或者新版本已做修改

[2026-04-25 15:44:30.570439] [fail] - 用例test_adu_check_event失败, 失败原因：/data/20260425-15.35.10-X3S-910096/events下未找到voyance_tag;takeover_autodrive事件触发生成的json
/data/20260425-15.35.10-X3S-910096/events下未找到Intervention;高风险事件触发生成的json
/data/20260425-15.35.10-X3S-910096/events下未找到Empty;Intervention事件触发生成的json
/data/20260425-15.35.10-X3S-910096/events下未找到voyance_tag:monitor:高风险事件触发生成的json


[2026-04-25 15:45:21.337945] [fail] - 用例test_adu_check_fail_log失败, 失败原因：/var/log/crpilot/pcie.log.INFO.26.04.25-15.35.06.822370527.6649.log: 
E260425 15:35:21:867259	pcie2camera.cc:418 RPC not found.
E260425 15:35:41:880562	pcie2camera.cc:418 RPC not found.
E260425 15:36:01:894113	pcie2camera.cc:418 RPC not found.
E260425 15:36:21:907724	pcie2camera.cc:418 RPC not found.
E260425 15:36:41:917252	pcie2camera.cc:418 RPC not found.
/var/log/crpilot/crbag.log.INFO.26.04.25-15.45.07.330006761.45604.log fatal: 
F260425 15:45:16:613812	zmq_service.cc:349 AddProcess process_name: "crbag_45604" process_id: 45604 listen_port: 35389 address: "127.0.0.1" failed. Please check node_server is running?
/var/log/crpilot/img2jpeg.log.INFO.26.04.25-15.38.33.554045537.42483.log fatal: 
F260425 15:38:42:634850	zmq_service.cc:349 AddProcess process_name: "img2jpeg_42483" process_id: 42483 listen_port: 33493 address: "127.0.0.1" failed. Please check node_server is running?
/var/log/crpilot/base_info.log.INFO.26.04.25-15.38.19.831346052.42252.log fatal: 
F260425 15:38:29:645818	zmq_service.cc:349 AddProcess process_name: "python3.10_42252" process_id: 42252 listen_port: 36079 address: "127.0.0.1" failed. Please check node_server is running?
/var/log/crpilot/crbag.log.INFO.26.04.25-15.37.09.174999933.34092.log fatal: 
F260425 15:37:24:188613	zmq_service.cc:349 AddProcess process_name: "crbag_34092" process_id: 34092 listen_port: 34875 address: "127.0.0.1" failed. Please check node_server is running?