Compare commits
merge into: 10235501447:master
10235501447:finalv1
10235501447:finalv2
10235501447:finalv3
10235501447:game_control
10235501447:gesture
10235501447:gesture-game
10235501447:gesture_for_chrome
10235501447:master
10235501447:screenshot
10235501447:vosk_inc
10235501447:wavesign
10235501447:web
pull from: 10235501447:web
10235501447:finalv1
10235501447:finalv2
10235501447:finalv3
10235501447:game_control
10235501447:gesture
10235501447:gesture-game
10235501447:gesture_for_chrome
10235501447:master
10235501447:screenshot
10235501447:vosk_inc
10235501447:wavesign
10235501447:web
80 changed files with 7115 additions and 809 deletions
Unified View
Diff Options
-
+0 -3.gitignore
-
+0 -149README.md
-
BINimg/主平台自动化测试结果.png
-
+14 -0index.html
-
+2907 -0package-lock.json
-
+43 -0package.json
-
BINpublic/mediapipe/gesture_recognizer.task
-
+20 -0public/mediapipe/wasm/vision_wasm_internal.js
-
BINpublic/mediapipe/wasm/vision_wasm_internal.wasm
-
+20 -0public/mediapipe/wasm/vision_wasm_nosimd_internal.js
-
BINpublic/mediapipe/wasm/vision_wasm_nosimd_internal.wasm
-
BINpublic/wavecontrol1.jpg
-
BINpublic/wavecontrol2.jpg
-
BINpublic/wavecontrol3.jpg
-
BINpublic/wavecontrol4.jpg
-
BINpublic/wavecontrol5.jpg
-
BINpublic/wavecontrol6.jpg
-
BINpublic/wavecontrol7.jpg
-
BINpublic/wavecontrol8.jpg
-
BINpublic/wavecontrol9.jpg
-
+9 -0requirements.txt
-
+0 -75sprint1/sprint1-planning.md
-
+0 -10sprint1/迭代计划.md
-
BINsprint2/WaveControl-sprint2.pptx
-
+0 -120sprint2/sprint2.md
-
BINsprint3/Sprint 3 技术文档：手势识别与游戏控制.pptx
-
+0 -146sprint3/sprint3.md
-
BINsprint4/WaveControl-sprint4.pptx
-
BINsrc-py/.DS_Store
-
+84 -0src-py/VoiceController.py
-
BINsrc-py/__pycache__/VoiceController.cpython-312.pyc
-
+45 -0src-py/main.py
-
+0 -0src-py/router/__init__.py
-
BINsrc-py/router/__pycache__/__init__.cpython-312.pyc
-
BINsrc-py/router/__pycache__/ws.cpython-312.pyc
-
+260 -0src-py/router/ws.py
-
BINsrc-tauri/.DS_Store
-
+13 -0src-tauri/.gitignore
-
+32 -0src-tauri/Cargo.toml
-
+8 -0src-tauri/Info.plist
-
+3 -0src-tauri/build.rs
-
+41 -0src-tauri/capabilities/default.json
-
+17 -0src-tauri/capabilities/desktop.json
-
+58 -0src-tauri/src/lib.rs
-
+6 -0src-tauri/src/main.rs
-
+61 -0src-tauri/tauri.conf.json
-
+13 -0src-tauri/tauri.macos.conf.json
-
BINsrc/.DS_Store
-
+35 -0src/App.vue
-
+34 -0src/AppMediaPipe.vue
-
+31 -0src/components/AutoStart.vue
-
+77 -0src/components/CircleProgress.vue
-
+81 -0src/components/DevTool.vue
-
+82 -0src/components/GestureCard.vue
-
+23 -0src/components/GestureIcon.vue
-
+41 -0src/components/Menu.vue
-
+259 -0src/hand_landmark/VideoDetector.vue
-
+291 -0src/hand_landmark/detector.ts
-
+507 -0src/hand_landmark/gesture_handler.ts
-
+49 -0src/locales/en.ts
-
+18 -0src/locales/i18n.ts
-
+47 -0src/locales/zh.ts
-
+85 -0src/main.ts
-
+30 -0src/py_api.ts
-
+38 -0src/router/index.ts
-
+87 -0src/store/app.ts
-
+46 -0src/utils/subWindow.ts
-
BINsrc/view/.DS_Store
-
+348 -0src/view/mainWindow/Guide.vue
-
+367 -0src/view/mainWindow/Home.vue
-
+607 -0src/view/mainWindow/MainWindow.vue
-
+95 -0src/view/mainWindow/Update.vue
-
+93 -0src/view/subWindow/SubWindow.vue
-
+7 -0src/vite-env.d.ts
-
+30 -0tsconfig.json
-
+10 -0tsconfig.node.json
-
+43 -0vite.config.ts
-
+0 -306项目测试文档.md
-
BIN项目测试结果.xlsx
+ 0
- 3
.gitignore
View File
| @ -1,3 +0,0 @@ | |||||
| node_modules/ | |||||
| src-tauri/ | |||||
| ~$项目测试结果.xlsx | |||||
+ 0
- 149
README.md
View File
| @ -1,149 +0,0 @@ | |||||
| # 📡 WaveControl 隔空手势控制系统 | |||||
| > 一套融合**手势识别**与**语音控制**的非接触式人机交互系统，包含主控制平台、手语学习平台 WaveSign、赛车游戏控制模块三大子系统，致力于打造自然、高效、多场景适配的“举手即控”体验。 | |||||
| ## 项目简介 | |||||
| 在厨房、医疗、演讲等“无法触控”或“不便触控”的环境中，传统鼠标/键盘交互模式效率低、操作受限。**WaveControl**以此为切入点，构建了一个基于摄像头识别的隔空控制系统，融合手势识别与语音识别，实现对系统级输入（键盘/鼠标）、手语教学以及游戏控制等功能。 | |||||
| 项目采用模块化架构设计，包含三大子系统： | |||||
| 1. **主控制平台**：支持窗口操作、媒体控制、鼠标替代、游戏映射等功能 | |||||
| 2. **赛车游戏交互系统**：实现与《Rush Rally Origins》等赛车游戏的隔空手柄交互 | |||||
| 3. **WaveSign 手语通**：面向听障人群的手语学习与社区互动平台 | |||||
| ## 测试信息 | |||||
| 本项目配套了完整的[测试文档](./项目测试文档.md)，涵盖以下子模块： | |||||
| - ✅ **主控制系统测试程序**：手势识别 → 键盘鼠标映射 → 交互反馈 | |||||
| - ✅ **游戏控制测试程序**：手势实时识别 → 虚拟手柄信号发送 → 赛车游戏响应验证 | |||||
| - ✅ **手语识别与打分测试程序**：摄像头实时捕捉动作 → MediaPipe评分 → UI动画与文本反馈 | |||||
| - ✅ **用户交互测试**：社区发帖、点赞评论、任务管理、日程提醒等核心功能均有覆盖 | |||||
| 测试方式支持浏览器调试 + 控制台运行日志追踪，若遇白屏可通过 Chrome DevTools 检查模块加载或路径引用情况。 | |||||
| ## Git 分支说明 | |||||
| | 分支名 | 描述 | 跳转链接 | | |||||
| | -------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | | |||||
| | `master` | 主分支，已完成整合，适用于演示与部署 | [🔗 master 分支](https://gitee.com/wydhhh/software-engineering/tree/master/) | | |||||
| | `finalv1` | 前后端初步融合尝试，手势 → 页面响应逻辑测试阶段 | [🔗 finalv1](https://gitee.com/wydhhh/software-engineering/tree/finalv1/) | | |||||
| | `finalv2` | 完成手势识别与前端事件联动，页面按钮联动测试 | [🔗 finalv2](https://gitee.com/wydhhh/software-engineering/tree/finalv2/) | | |||||
| | `finalv3` | 增加语音识别、手势控制切换、音乐控制、手语平台接入、各子系统联调优化 | [🔗 finalv3](https://gitee.com/wydhhh/software-engineering/tree/finalv3/) | | |||||
| | `gesture` | 手势识别逻辑独立开发模块 | [🔗 gesture](https://gitee.com/wydhhh/software-engineering/tree/gesture/) | | |||||
| | `gesture_for_chrome` | 针对 Chrome 插件开发的手势控制方案（PPT翻页） | [🔗 gesture_for_chrome](https://gitee.com/wydhhh/software-engineering/tree/gesture_for_chrome/) | | |||||
| | `gesture-game` | 初步游戏控制实验，控制小球移动 | [🔗 gesture-game](https://gitee.com/wydhhh/software-engineering/tree/gesture-game/) | | |||||
| | `game_control` | 控制《Rush Rally Origins》赛车游戏，已支持加速转向等 | [🔗 game_control](https://gitee.com/wydhhh/software-engineering/tree/game_control/) | | |||||
| | `wavesign` | 手语通子系统开发主线，包括教学评分、社区、日程等 | [🔗 wavesign](https://gitee.com/wydhhh/software-engineering/tree/wavesign/) | | |||||
| | `web` | 最初的网页原型设计，UI 静态草稿 | [🔗 web](https://gitee.com/wydhhh/software-engineering/tree/web/) | | |||||
| | `screenshot` | 手势控制截屏模块，用于快速抓取操作界面 | [🔗 screenshot](https://gitee.com/wydhhh/software-engineering/tree/screenshot/) | | |||||
| | `vosk_inc` | 接入 VOSK 实现语音识别与实时字幕展示 | [🔗 vosk_inc](https://gitee.com/wydhhh/software-engineering/tree/vosk_inc/) | | |||||
| ## 技术架构 | |||||
| | 层级 | 技术方案 | | |||||
| | ------------ | ------------------------------------------------------------ | | |||||
| | 前端 | vue3 + TypeScript +HTML + CSS + Tailwind CSS + JavaScript + PySide2（Qt GUI） | | |||||
| | 后端 | Django 4.x（主平台 + 手语通） + Python 脚本逻辑（游戏控制） | | |||||
| | 手势识别 | MediaPipe Hand Landmarker | | |||||
| | 虚拟设备控制 | 键盘鼠标模拟、vgamepad 虚拟手柄（XInput） | | |||||
| | 数据处理 | Kalman Filter（手势抖动滤波）、SQLite3 数据库 | | |||||
| ## 模块介绍 | |||||
| ### 1️⃣ 主控制平台（WaveControl） | |||||
| ##### ✌️ **功能特色** | |||||
| - 多种预设手势操作（点击、滚动、后退等） | |||||
| - 手势范围调节与自定义手势库 | |||||
| - 支持语音识别辅助控制 | |||||
| - 界面直观，状态反馈实时 | |||||
| ##### 🎯 应用场景 | |||||
| - 📺 沙发上追剧时，不再找遥控器，用手势暂停/快进 | |||||
| - 👨🏫 教学/演讲中，用手势控制 PPT 流畅翻页 | |||||
| - 🧑🍳 厨房做饭时，隔空查食谱不怕弄脏设备 | |||||
| - 🏥 医疗/无菌操作室中，非接触式操作电脑界面 | |||||
| - 🕹️ 游戏中挥手即控，沉浸感倍增 | |||||
| 🖼️ **UI 示例** | |||||
| - 控制主面板（准确率/响应时间/识别窗口） | |||||
| - 手势管理面板（快捷操作映射设置） | |||||
| ### 2️⃣ 游戏控制模块 | |||||
| 🔗 项目演示：[游戏控制项目演示视频](https://www.bilibili.com/video/BV1H5gLzsEn8?vd_source=46a0e2ec60dfb8a247c96905ee47d378) | |||||
| 🎮 目标：**无需实体手柄，通过摄像头即可玩赛车游戏！** | |||||
| 适配游戏：Steam平台《Rush Rally Origins》及支持 Xbox手柄的其他游戏 | |||||
| **实现要点：** | |||||
| | 功能 | 技术说明 | | |||||
| | ------------ | ----------------------------------------------------------- | | |||||
| | 摄像头识别 | OpenCV + MediaPipe | | |||||
| | 手势控制映射 | 👍右手拇指上扬 = 加速 👍左手 = 刹车 ✋左倾 = 左转，右倾 = 右转 | | |||||
| | 虚拟手柄接口 | vgamepad + XInput | | |||||
| | 抖动滤除 | Kalman 滤波器平滑动作 | | |||||
| | UI反馈 | PySide2 构建调试窗口 | | |||||
| ### 3️⃣ 手语通子项目：**WaveSign** | |||||
| 🔗 项目演示：[手语通项目演示视频](https://www.bilibili.com/video/BV1Fig5zHEhq?vd_source=46a0e2ec60dfb8a247c96905ee47d378) | |||||
| 项目定位：帮助听障人群及其家人朋友学习、练习、交流手语的综合平台 | |||||
| **功能模块：** | |||||
| - ✅ **手语教学与评分系统**：上传视频或用摄像头练习手语动作，系统打分反馈 | |||||
| - 🗺️ **课程地图与互动练习**：任务式学习，配合卡片式巩固练习 | |||||
| - 👥 **社区交流**：发帖、评论、点赞、关注等功能 | |||||
| - 📅 **日程管理**：内置待办事项与日历，辅助学习安排 | |||||
| - 🧭 **生活服务**： | |||||
| - 出行导航（结合地图与实时提醒） | |||||
| - 辅助器具推荐 | |||||
| - 就业信息推送 | |||||
| - 无障碍亲子/技能活动预告 | |||||
| **技术实现：** | |||||
| - Django + SQLite 构建用户系统与服务逻辑 | |||||
| - 使用 MediaPipe 实时评分用户手语表现 | |||||
| - 前端页面响应式 + 卡片式交互体验 | |||||
| ## 项目成员 | |||||
| 王云岱 朱子玥 杨嘉莉 | |||||
| ## 📌 项目进度 | |||||
| - ✅ 第一轮：系统原型 + UI设计 + 手势识别框架 | |||||
| - ✅ 第二轮：主平台功能实现 + 手势控制实现 | |||||
| - ✅ 第三轮：打通语音识别 + 游戏交互控制 + 手语通平台初步实现 | |||||
| - ✅ 第四轮：赛车游戏交互实现 + 手语通平台完整搭建 | |||||
| - 🧪 第五轮：综合测试 + 用户体验调优 + 结项演示 | |||||
BIN
img/主平台自动化测试结果.png
View File
| Before | After |
|---|---|
|
|
| Width: 2036 | Height: 681 | Size: 93 KiB |
+ 14
- 0
index.html
View File
| @ -0,0 +1,14 @@ | |||||
| <!doctype html> | |||||
| <html lang="en" style="height: 100%;width: 100%;"> | |||||
| <head> | |||||
| <meta charset="UTF-8" /> | |||||
| <link rel="icon" type="image/svg+xml" href="/vite.svg" /> | |||||
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> | |||||
| <title>Tauri + Vue + Typescript App</title> | |||||
| </head> | |||||
| <body style="height: 100%;width: 100%;overflow: hidden;"> | |||||
| <div id="app" style="height: 100%;width: 100%"></div> | |||||
| <script type="module" src="/src/main.ts"></script> | |||||
| </body> | |||||
| </html> | |||||
+ 2907
- 0
package-lock.json
File diff suppressed because it is too large
View File
+ 43
- 0
package.json
View File
| @ -0,0 +1,43 @@ | |||||
| { | |||||
| "name": "tauri", | |||||
| "private": true, | |||||
| "version": "0.1.0", | |||||
| "type": "module", | |||||
| "scripts": { | |||||
| "install-reqs": "npm install && pip install -r requirements.txt", | |||||
| "dev": "vite", | |||||
| "build:py": "pyinstaller --noconfirm --distpath src-tauri/bin/ main_win.spec", | |||||
| "build:py-mac": "pyinstaller --noconfirm --distpath src-tauri/bin/ main_mac.spec", | |||||
| "build:icons": "npx tauri icon public/lazyeat.png", | |||||
| "build:icons-mac": "npx tauri icon public/lazyeat.png --output src-tauri/icons/mac", | |||||
| "build": "vite build", | |||||
| "preview": "vite preview", | |||||
| "tauri": "npx tauri", | |||||
| "tauri dev": "npx tauri" | |||||
| }, | |||||
| "dependencies": { | |||||
| "@icon-park/vue-next": "^1.4.2", | |||||
| "@mediapipe/tasks-vision": "^0.10.22-rc.20250304", | |||||
| "@tauri-apps/api": "^2", | |||||
| "@tauri-apps/plugin-autostart": "^2.2.0", | |||||
| "@tauri-apps/plugin-notification": "^2.2.2", | |||||
| "@tauri-apps/plugin-opener": "^2.2.6", | |||||
| "@tauri-apps/plugin-store": "^2.2.0", | |||||
| "@tauri-apps/plugin-window-state": "^2.2.1", | |||||
| "element-plus": "^2.9.5", | |||||
| "pinia": "^3.0.1", | |||||
| "pinia-shared-state": "^1.0.1", | |||||
| "vue": "^3.5.13", | |||||
| "vue-i18n": "^9.14.4", | |||||
| "vue-router": "^4.5.0" | |||||
| }, | |||||
| "devDependencies": { | |||||
| "@tauri-apps/cli": "^2", | |||||
| "@vitejs/plugin-vue": "^5.2.1", | |||||
| "naive-ui": "^2.41.0", | |||||
| "sass-embedded": "^1.85.1", | |||||
| "typescript": "~5.6.2", | |||||
| "vite": "^6.0.3", | |||||
| "vue-tsc": "^2.1.10" | |||||
| } | |||||
| } | |||||
BIN
sprint1/隔空手势识别系统Sprint1.pptx → public/mediapipe/gesture_recognizer.task
View File
+ 20
- 0
public/mediapipe/wasm/vision_wasm_internal.js
File diff suppressed because it is too large
View File
BIN
public/mediapipe/wasm/vision_wasm_internal.wasm
View File
+ 20
- 0
public/mediapipe/wasm/vision_wasm_nosimd_internal.js
File diff suppressed because it is too large
View File
BIN
public/mediapipe/wasm/vision_wasm_nosimd_internal.wasm
View File
BIN
public/wavecontrol1.jpg
View File
| Before | After |
|---|---|
|
|
| Width: 476 | Height: 476 | Size: 18 KiB |
BIN
public/wavecontrol2.jpg
View File
| Before | After |
|---|---|
|
|
| Width: 509 | Height: 509 | Size: 21 KiB |
BIN
public/wavecontrol3.jpg
View File
| Before | After |
|---|---|
|
|
| Width: 498 | Height: 498 | Size: 22 KiB |
BIN
public/wavecontrol4.jpg
View File
| Before | After |
|---|---|
|
|
| Width: 532 | Height: 532 | Size: 24 KiB |
BIN
public/wavecontrol5.jpg
View File
| Before | After |
|---|---|
|
|
| Width: 614 | Height: 614 | Size: 25 KiB |
BIN
public/wavecontrol6.jpg
View File
| Before | After |
|---|---|
|
|
| Width: 475 | Height: 475 | Size: 22 KiB |
BIN
public/wavecontrol7.jpg
View File
| Before | After |
|---|---|
|
|
| Width: 548 | Height: 548 | Size: 21 KiB |
BIN
public/wavecontrol8.jpg
View File
| Before | After |
|---|---|
|
|
| Width: 522 | Height: 522 | Size: 20 KiB |
BIN
public/wavecontrol9.jpg
View File
| Before | After |
|---|---|
|
|
| Width: 484 | Height: 484 | Size: 18 KiB |
+ 9
- 0
requirements.txt
View File
| @ -0,0 +1,9 @@ | |||||
| pynput | |||||
| fastapi | |||||
| opencv-python | |||||
| numpy | |||||
| uvicorn | |||||
| vosk | |||||
| pygrabber | |||||
| pyinstaller | |||||
| sounddevice | |||||
+ 0
- 75
sprint1/sprint1-planning.md
View File
| @ -1,75 +0,0 @@ | |||||
| ### PPT1 标题 时间 组员 | |||||
| - 标题：隔空手势识别系统Sprint1 | |||||
| - 时间：[具体时间] | |||||
| - 组员：[组员姓名] | |||||
| ### PPT2 motivation | |||||
| 在当今数字化时代，传统的鼠标键盘操作方式存在一定的局限性，如在某些场景下操作不够便捷、卫生等问题。隔空手势识别系统的出现，能够为用户提供更加自然、便捷的交互方式，填补了市场在非接触式交互领域的空白。它可以广泛应用于智能家居、智能办公、教育演示等多个场景，为用户带来全新的体验。 | |||||
| ### PPT3 影响地图 | |||||
| - why? | |||||
| - 解决传统交互方式在特殊场景下的不便，如疫情期间减少接触、医疗场景避免交叉感染等。 | |||||
| - 满足用户对更加自然、便捷交互方式的需求，提升用户体验。 | |||||
| - 助力企业提升产品的科技感和竞争力，开拓新的市场领域。 | |||||
| - who? | |||||
| - 普通消费者：用于家庭娱乐、智能家居控制等。 | |||||
| - 办公人群：在会议演示、办公操作中提高效率。 | |||||
| - 教育工作者：在教学过程中进行更加生动的演示。 | |||||
| - 医疗人员：在医疗操作中避免交叉感染。 | |||||
| - how? | |||||
| - 核心部分分工：算法团队负责手势识别和语音识别算法的开发；软件团队负责系统的整体架构和交互界面的设计；硬件团队负责传感器等硬件设备的选型和开发。 | |||||
| - 采用的技术：计算机视觉技术用于手势识别，语音识别技术用于语音交互，深度学习算法提高识别的准确性和稳定性。 | |||||
| - what? | |||||
| - 核心功能：远程操控鼠标指针、点击操作、复杂手势识别与响应、语音识别与指令执行。 | |||||
| ### PPT4 头脑风暴 | |||||
| - 点子1：在游戏中使用手势进行角色控制，增强游戏的沉浸感。 | |||||
| - 点子2：在商场的自助购物终端上使用手势操作，提高购物效率。 | |||||
| - 点子3：在汽车驾驶中，通过手势控制车内娱乐系统和导航系统，提高驾驶安全性。 | |||||
| - 点子4：在智能健身房中，使用手势控制健身设备和课程选择。 | |||||
| - 点子5：在虚拟现实和增强现实场景中，使用手势进行更加自然的交互。 | |||||
| ### PPT5 basic design | |||||
| 系统采用分层架构设计，包括硬件层、驱动层、算法层和应用层。硬件层负责数据的采集，驱动层负责硬件设备的驱动和数据传输，算法层负责手势和语音的识别，应用层负责与用户的交互和业务逻辑的处理。 | |||||
| ### PPT6 用户故事 | |||||
| - card1 | |||||
| - conversation1：用户在客厅看电视，想要切换频道，但不想起身找遥控器。 | |||||
| - confirmation1：用户通过隔空手势轻松切换频道，无需使用遥控器。 | |||||
| - card2 | |||||
| - conversation2：办公人员在会议中需要展示文档，想要翻页但不想触碰鼠标。 | |||||
| - confirmation2：办公人员通过手势实现文档的翻页操作，提高会议效率。 | |||||
| - card3 | |||||
| - conversation3：教育工作者在教学过程中，想要展示图片但不想走到电脑前操作。 | |||||
| - confirmation3：教育工作者通过手势控制图片的切换和缩放，使教学更加生动。 | |||||
| ### PPT7 初步开发构想 | |||||
| - v1（本周四）：完成硬件设备的选型和采购，搭建开发环境，实现基本的手势识别算法。 | |||||
| - v2（下周一）：完成系统的整体架构设计，实现手势识别与鼠标指针的初步关联。 | |||||
| - v3（下周四）：完善手势识别的准确性和稳定性，实现语音识别功能，并进行初步的测试。 | |||||
| - v4（最终）：完成系统的所有功能开发，进行全面的测试和优化，准备上线发布。 | |||||
| ### PPT8 初步模块设想 | |||||
| - 子模块：硬件模块、手势识别模块、语音识别模块、交互界面模块、业务逻辑模块。 | |||||
| - 工作量对比：硬件模块和算法模块的工作量相对较大，交互界面模块和业务逻辑模块的工作量相对较小。可以用简单的数字比例表示，如硬件模块：30%，手势识别模块：25%，语音识别模块：20%，交互界面模块：15%，业务逻辑模块：10%。 | |||||
| - 优先级：手势识别模块和硬件模块的优先级较高，需要优先开发；语音识别模块和交互界面模块的优先级次之；业务逻辑模块的优先级相对较低。 | |||||
| ### PPT9 future view | |||||
| 未来，我们将不断优化系统的性能和功能，拓展更多的应用场景。例如，与更多的智能家居设备进行集成，实现更加智能化的家居控制；在工业领域应用，提高生产效率和安全性等。 | |||||
| ### PPT10 用户旅程 | |||||
| - 优化点1：手势识别的准确性和稳定性需要进一步提高，减少误识别的情况。 | |||||
| - 优化点2：语音识别的灵敏度和准确性需要优化，确保在不同环境下都能准确识别用户的语音指令。 | |||||
| - 优化点3：交互界面的设计需要更加简洁直观，提高用户的操作体验。 | |||||
| - 优化点4：系统的响应速度需要加快，减少用户操作的等待时间。 | |||||
| - 优化点5：增加更多的手势和语音指令，满足用户多样化的需求。 | |||||
| - 优化点6：提高系统的兼容性，支持更多的操作系统和硬件设备。 | |||||
| ### PPT11 在迭代中开发 | |||||
| - sprint planning：在每个冲刺阶段开始前，制定详细的计划，明确目标和任务。 | |||||
| - daily scrum：每天进行短会，沟通工作进展和遇到的问题。 | |||||
| - sprint review：在每个冲刺阶段结束后，进行评审，展示成果并收集反馈。 | |||||
| - sprint retrospective：对每个冲刺阶段进行回顾和总结，分析问题并提出改进措施。 | |||||
| ### PPT12 thanks | |||||
| 感谢大家的聆听！ | |||||
+ 0
- 10
sprint1/迭代计划.md
View File
| @ -1,10 +0,0 @@ | |||||
| # 📈 迭代计划（Sprint Plan） | |||||
| ## 🗓️ 项目时间线 | |||||
| | 时间 | 内容说明 | | |||||
| | -------- | ------------------------------------------------- | | |||||
| | 本周四 | 🧩 第一次迭代演示（系统架构设计 + 原型 + UI 草稿） | | |||||
| | 下周一 | 🔁 第二次迭代演示（打通手势识别 → 系统控制的闭环） | | |||||
| | 下周四 | 🎙 第三次迭代演示（加入语音控制，初步功能联动） | | |||||
| | 下下周三 | 🚀 第四次迭代汇报（最终系统完整交付 + 场景演示） | | |||||
BIN
sprint2/WaveControl-sprint2.pptx
View File
+ 0
- 120
sprint2/sprint2.md
View File
| @ -1,120 +0,0 @@ | |||||
| ## 一、本轮迭代目标 | |||||
| 本 Sprint 主要目标是完成 **基于浏览器端的手势控制系统雏形**，实现如下基础功能： | |||||
| - 搭建前端 Web 应用结构与 UI 页面 | |||||
| - 集成 MediaPipe 手势识别模型，完成实时手势检测 | |||||
| - 映射部分手势为系统控制行为（鼠标移动、点击等） | |||||
| - 搭建基础手势处理逻辑与组件封装结构 | |||||
| - 支持语音识别指令发送流程（准备后端对接） | |||||
| ------ | |||||
| ## 二、本轮主要完成内容 | |||||
| | 类别 | 工作内容 | | |||||
| | -------- | ------------------------------------------------------------ | | |||||
| | 前端搭建 | 基于 Vite + Vue 3 + TypeScript 完成项目结构、模块划分、页面初始化 | | |||||
| | 手势识别 | 集成 Google MediaPipe WASM 模型，完成实时检测与渲染 | | |||||
| | 控制逻辑 | 封装 `GestureHandler`、`TriggerAction`，实现手势到系统操作的映射 | | |||||
| | UI设计 | 完成仪表盘式主界面设计，支持响应式、自定义手势预览等 | | |||||
| | 模型结构 | 初步梳理 `Detector` 识别流程，统一封装初始化与推理逻辑 | | |||||
| | 后端准备 | 初始化 Python 接口 WebSocket 构建，支持语音识别交互 | | |||||
| ------ | |||||
| ## 三、系统结构设计 | |||||
| ``` | |||||
| [用户摄像头] | |||||
| ↓ | |||||
| [VideoDetector.vue] → 捕捉视频帧，传入检测器 | |||||
| ↓ | |||||
| [Detector.ts] → 调用 MediaPipe 识别手势 | |||||
| ↓ | |||||
| [GestureHandler.ts] → 映射行为（鼠标/键盘/语音） | |||||
| ↓ | |||||
| [TriggerAction.ts] → 向后端/系统发出控制指令 | |||||
| ↓ | |||||
| [Python 后端]（语音识别/扩展指令） ← WebSocket 接入 | |||||
| ``` | |||||
| ------ | |||||
| ## 四、核心技术栈与理由 | |||||
| | 技术 | 用途 | 原因 | | |||||
| | ---------------- | -------------------- | ------------------------------------------ | | |||||
| | Vue 3 + Vite | 前端框架 + 构建工具 | 快速开发，热更新快，Composition API 更灵活 | | |||||
| | TypeScript | 增强类型约束 | 减少运行时错误，提升可维护性 | | |||||
| | MediaPipe Tasks | 手势识别模型 | 体积小，支持浏览器部署，准确率高 | | |||||
| | WebSocket | 前后端实时通信 | 保持实时交互流畅 | | |||||
| | Python + FastAPI | 后端扩展接口（语音） | 快速搭建接口，后续支持 Py 模型运行 | | |||||
| ------ | |||||
| ## 五、主要功能点与说明 | |||||
| ### 1. Detector 类封装 | |||||
| - 初始化 WASM 模型与识别器 | |||||
| - 封装 `detect()` 每帧调用逻辑 | |||||
| - 支持获取 `landmarks`, `gestures`, `handedness` | |||||
| ### 2. 手势识别与映射 | |||||
| - 仅识别食指 → 鼠标移动 | |||||
| - 食指 + 中指 → 鼠标点击 | |||||
| - 食指 + 拇指捏合 → 滚动 | |||||
| - 四指竖起 → 发送快捷键 | |||||
| - 小指+拇指 → 启动语音识别（前端 WebSocket） | |||||
| ### 3. UI 界面模块 | |||||
| - 实时视频预览窗口 | |||||
| - 子窗口：手势反馈 / 执行动作提示 | |||||
| - 配置面板：开关检测 / 配置行为映射 | |||||
| ------ | |||||
| ## 六、识别代码示例（关键手势） | |||||
| ```ts | |||||
| if (gesture === HandGesture.ONLY_INDEX_UP) { | |||||
| this.triggerAction.moveMouse(x, y) | |||||
| } | |||||
| if (gesture === HandGesture.INDEX_AND_THUMB_UP) { | |||||
| this.triggerAction.scrollUp() | |||||
| } | |||||
| ``` | |||||
| - 每种手势匹配后，调用封装的 WebSocket + 系统接口发送动作 | |||||
| - `TriggerAction` 支持复用 | |||||
| ------ | |||||
| ## 七、问题与解决方案 | |||||
| | 问题描述 | 解决方法 | | |||||
| | --------------------------------- | ---------------------------------------------------- | | |||||
| | WebSocket 连接后反复断开 | 增加连接状态判断 `readyState !== OPEN`，避免提前发送 | | |||||
| | Tauri 插件调用 `invoke undefined` | 使用 `npm run tauri dev` 启动，而非普通浏览器启动 | | |||||
| | 页面刷新后路由丢失（404） | Vue Router 改为 `createWebHashHistory()` 模式 | | |||||
| | 模型未加载或报错提示 | 增加 loading 控制，捕捉初始化异常 | | |||||
| ------ | |||||
| ## 八、未来规划（Sprint 2 预告） | |||||
| | 方向 | 说明 | | |||||
| | ----------------- | --------------------------------------------------- | | |||||
| | 多手势拓展 | 三指滚动、多指触发多功能、多手联合判断 | | |||||
| | 自定义 .task 模型 | 支持用户采集数据，自定义模型替换现有 MediaPipe 模型 | | |||||
| | 后端语音指令集成 | Vosk 模型识别语音，触发系统控制 | | |||||
| | 设置项管理 | 实现用户自定义快捷键、手势配置界面 | | |||||
BIN
sprint3/Sprint 3 技术文档：手势识别与游戏控制.pptx
View File
+ 0
- 146
sprint3/sprint3.md
View File
| @ -1,146 +0,0 @@ | |||||
| # Sprint 3 技术文档 | |||||
| **识别到控制闭环打通 + 支持游戏操作** | |||||
| ------ | |||||
| ## 一、概述 | |||||
| 本阶段实现了一个运行于浏览器端的手势识别系统，结合 MediaPipe 和 WebSocket，打通从摄像头手势识别 → 系统控制（鼠标、滚动、键盘） → 控制小游戏角色的全流程。 | |||||
| 新增支持多种系统控制动作和游戏动作，包括：左右移动、跳跃、组合跳跃等。 | |||||
| ------ | |||||
| ## 二、模块结构概览 | |||||
| 模块分工如下： | |||||
| - **Detector.ts** | |||||
| 加载并初始化 MediaPipe 手势模型，检测手部关键点与手势状态，并调用 `GestureHandler`。 | |||||
| - **GestureHandler.ts** | |||||
| 将识别出的手势映射为系统操作（鼠标、滚动、键盘指令等），支持连续确认、平滑处理、动作节流。 | |||||
| - **TriggerAction.ts** | |||||
| 封装 WebSocket 通信，向后端发送 JSON 格式的控制命令。 | |||||
| ``` | |||||
| [摄像头视频流] | |||||
| ↓ | |||||
| [VideoDetector.vue] | |||||
| - 捕捉视频帧 | |||||
| - 显示实时画面 | |||||
| ↓ | |||||
| [Detector.ts] | |||||
| - 使用 MediaPipe Tasks WASM 模型进行手势识别 | |||||
| - 返回关键点、手势类型 | |||||
| ↓ | |||||
| [GestureHandler.ts] | |||||
| - 将识别结果转为行为 | |||||
| - 识别特定组合，如食指 + 拇指 → 滚动 | |||||
| ↓ | |||||
| [TriggerAction.ts] | |||||
| - 向 WebSocket 发送控制指令 | |||||
| - 控制鼠标/键盘/后端动作 | |||||
| ↓ | |||||
| [后端 Python] | |||||
| - WebSocket 接口接收指令 | |||||
| - 准备语音接口/控制中枢 | |||||
| ``` | |||||
| ------ | |||||
| ## 三、识别手势说明（扩展后） | |||||
| 每帧识别手指竖起状态（拇指到小指，0 或 1），组合为状态串，查表识别对应手势： | |||||
| | 状态串 | 手势名 | 动作描述 | | |||||
| | ----------- | ------------------- | ------------------------- | | |||||
| | `0,1,0,0,0` | only_index_up | 鼠标移动 | | |||||
| | `1,1,0,0,0` | index_and_thumb_up | 鼠标点击 | | |||||
| | `0,0,1,1,1` | scroll_gesture_2 | 页面滚动 | | |||||
| | `1,0,1,1,1` | scroll_gesture_2 | 页面滚动（兼容变体） | | |||||
| | `0,1,1,1,1` | four_fingers_up | 发送快捷键（如播放/全屏） | | |||||
| | `1,1,1,1,1` | stop_gesture | 暂停/开始识别 | | |||||
| | `1,0,0,0,1` | voice_gesture_start | 启动语音识别 | | |||||
| | `0,0,0,0,0` | voice_gesture_stop | 停止语音识别 | | |||||
| | `0,1,1,0,0` | jump | 小人跳跃（上键） | | |||||
| | `1,1,1,0,0` | rightjump | 小人右跳（右+上组合） | | |||||
| | 自定义判断 | direction_right | 小人右移（长按右键） | | |||||
| | 自定义判断 | delete_gesture | 小人左移（长按左键） | | |||||
| ------ | |||||
| ## 四、控制行为逻辑说明 | |||||
| ### 鼠标控制 | |||||
| - **移动**：基于食指指尖位置映射至屏幕坐标，支持平滑处理，减少抖动。 | |||||
| - **点击**：食指与拇指同时上举，节流处理防止重复点击。 | |||||
| - **滚动**：捏合（拇指+食指）后上下移动控制滚动方向，带阈值判断。 | |||||
| ### 快捷键/系统操作 | |||||
| - 四指上举 → 发送自定义按键（如全屏、暂停、下一集等）。 | |||||
| - 五指上举 → 暂停/开始识别。支持进度提示。 | |||||
| - 删除手势（右手拇指伸出，其余收起）→ 连续发送 Backspace/方向键。 | |||||
| - 自定义 holdKey 方法支持模拟长按。 | |||||
| ### 游戏控制扩展（键盘模拟） | |||||
| | 手势名 | 动作 | | |||||
| | --------------- | -------------------- | | |||||
| | delete_gesture | 向左移动（左键长按） | | |||||
| | direction_right | 向右移动（右键长按） | | |||||
| | jump | 向上跳跃（上键） | | |||||
| | rightjump | 右跳（右后延迟上） | | |||||
| ------ | |||||
| ## 五、性能与稳定性处理 | |||||
| | 问题 | 解决方案 | | |||||
| | ------------------ | ------------------------------------------------------ | | |||||
| | 手势误判 | 连续帧确认（minGestureCount ≥ 5） | | |||||
| | 鼠标抖动 | 屏幕坐标引入平滑系数 `smoothening` | | |||||
| | 动作重复触发 | 节流控制点击/滚动/快捷键/方向键（如 `CLICK_INTERVAL`） | | |||||
| | WebSocket 中断重连 | 异常断线后自动重连，带重试时间间隔 | | |||||
| | 滚动误触 | 拇指与食指距离阈值控制是否处于“捏合”状态 | | |||||
| | UI 状态提示不同步 | 使用全局 store（如 `app_store.sub_window_info`）更新 | | |||||
| ------ | |||||
| ## 六、技术特性总结 | |||||
| - 支持 MediaPipe WASM 模型在前端本地运行，低延迟识别 | |||||
| - 所有动作模块化封装，便于未来扩展新手势、新控制指令 | |||||
| - 手势控制与后端通过 WebSocket 实时交互，接口稳定 | |||||
| - 支持游戏场景（虚拟角色）控制，具备实际互动展示能力 | |||||
| - 控制逻辑可配置：如识别区域、快捷键内容、手势灵敏度 | |||||
| ------ | |||||
| ## 七、后续优化建议（下一阶段） | |||||
| | 方向 | 目标 | | |||||
| | -------------------- | ----------------------------------------------- | | |||||
| | 增加左手手势组合识别 | 允许双手协同控制，比如捏合+四指等 | | |||||
| | 提升模型识别精度 | 支持定制 MediaPipe .task 文件、自采样训练 | | |||||
| | UI 设置面板 | 提供手势→动作自定义、阈值调节、快捷键修改 | | |||||
| | 引入语音控制 | Whisper/Vosk 实现语音命令解析，结合手势联动使用 | | |||||
| | 多模式切换 | 视频控制、PPT 控制、游戏控制等可切换交互模式 | | |||||
| | 场景演示/视频录制 | 准备真实交互展示视频，用于演示或上线宣传 | | |||||
| ------ | |||||
| ## 八、结语 | |||||
| 本轮迭代成功实现了从手势识别到控制动作的完整闭环，扩展支持了游戏角色控制（多方向移动与跳跃），并在操作流畅性、准确性与系统解耦方面都取得实质进展。后续可以围绕用户可配置性、语音识别联动与多场景适配进一步扩展系统能力。 | |||||
BIN
sprint4/WaveControl-sprint4.pptx
View File
BIN
src-py/.DS_Store
View File
+ 84
- 0
src-py/VoiceController.py
View File
| @ -0,0 +1,84 @@ | |||||
| import json | |||||
| import threading | |||||
| import sounddevice as sd | |||||
| import numpy as np | |||||
| from vosk import Model, KaldiRecognizer | |||||
| big_model_path = "big-model" | |||||
| small_model_path = "model" | |||||
| class VoiceController: | |||||
| def __init__(self, model_type='small'): | |||||
| if model_type == 'small': | |||||
| self.model = Model(small_model_path) | |||||
| else: | |||||
| self.model = Model(big_model_path) | |||||
| self.zh_text = None | |||||
| self.is_recording = False | |||||
| self.recognizer = KaldiRecognizer(self.model, 16000) | |||||
| # 初始化输入流 | |||||
| self.stream = sd.InputStream( | |||||
| samplerate=16000, | |||||
| channels=1, | |||||
| blocksize=4096, | |||||
| dtype='int16' # 对应原来 PyAudio 的 paInt16 | |||||
| ) | |||||
| self.stream.start() | |||||
| def record_audio(self): | |||||
| self.frames = [] | |||||
| print("录音开始...") | |||||
| # 持续录音直到标志改变 | |||||
| while self.is_recording: | |||||
| data, _ = self.stream.read(4096) | |||||
| self.frames.append(data.tobytes()) # 转成 bytes，保持跟原先 pyaudio 一致 | |||||
| def start_record_thread(self): | |||||
| self.is_recording = True | |||||
| threading.Thread(target=self.record_audio, daemon=True).start() | |||||
| def stop_record(self): | |||||
| self.is_recording = False | |||||
| def transcribe_audio(self) -> str: | |||||
| self.recognizer.Reset() | |||||
| # 分块处理音频数据 | |||||
| for chunk in self.frames: | |||||
| self.recognizer.AcceptWaveform(chunk) | |||||
| result = json.loads(self.recognizer.FinalResult()) | |||||
| text = result.get('text', '') | |||||
| text = text.replace(' ', '') | |||||
| print(f"识别结果: {text}") | |||||
| return text | |||||
| if __name__ == '__main__': | |||||
| pass | |||||
| # from PyQt5.QtWidgets import QApplication, QPushButton | |||||
| # | |||||
| # app = QApplication([]) | |||||
| # | |||||
| # # 点击按钮开始录音 | |||||
| # voice_controller = VoiceController() | |||||
| # | |||||
| # | |||||
| # def btn_clicked(): | |||||
| # if voice_controller.is_recording: | |||||
| # voice_controller.stop_record() | |||||
| # text = voice_controller.transcribe_audio() | |||||
| # print(text) | |||||
| # else: | |||||
| # voice_controller.start_record_thread() | |||||
| # | |||||
| # | |||||
| # btn = QPushButton('开始录音') | |||||
| # btn.clicked.connect(btn_clicked) | |||||
| # btn.show() | |||||
| # | |||||
| # app.exec_() | |||||
BIN
src-py/__pycache__/VoiceController.cpython-312.pyc
View File
+ 45
- 0
src-py/main.py
View File
| @ -0,0 +1,45 @@ | |||||
| import os | |||||
| import sys | |||||
| import uvicorn | |||||
| from fastapi import FastAPI | |||||
| from fastapi.middleware.cors import CORSMiddleware | |||||
| from router.ws import router as ws_router | |||||
| if hasattr(sys, 'frozen'): | |||||
| # pyinstaller打包成exe时，sys.argv[0]的值是exe的路径 | |||||
| # os.path.dirname(sys.argv[0])可以获取exe的所在目录 | |||||
| # os.chdir()可以将工作目录更改为exe的所在目录 | |||||
| os.chdir(os.path.dirname(sys.argv[0])) | |||||
| app = FastAPI() | |||||
| app.add_middleware( | |||||
| CORSMiddleware, | |||||
| allow_origins=["*"], | |||||
| allow_credentials=True, | |||||
| allow_methods=["*"], | |||||
| allow_headers=["*"], | |||||
| ) | |||||
| # 添加 WebSocket 路由 | |||||
| app.include_router(ws_router) | |||||
| @app.get("/") | |||||
| def read_root(): | |||||
| return "ready" | |||||
| @app.get("/shutdown") | |||||
| def shutdown(): | |||||
| import signal | |||||
| import os | |||||
| os.kill(os.getpid(), signal.SIGINT) | |||||
| if __name__ == '__main__': | |||||
| port = 62334 | |||||
| print(f"Starting server at http://127.0.0.1:{port}/docs") | |||||
| uvicorn.run(app, host="127.0.0.1", port=port) | |||||
+ 0
- 0
src-py/router/__init__.py
View File
BIN
src-py/router/__pycache__/__init__.cpython-312.pyc
View File
BIN
src-py/router/__pycache__/ws.cpython-312.pyc
View File
+ 260
- 0
src-py/router/ws.py
View File
| @ -0,0 +1,260 @@ | |||||
| import json | |||||
| from dataclasses import dataclass | |||||
| from typing import Dict, List, Optional, Union | |||||
| from fastapi import APIRouter, WebSocket, WebSocketDisconnect | |||||
| from pynput.keyboard import Controller as KeyboardController | |||||
| from pynput.keyboard import Key | |||||
| from pynput.mouse import Button, Controller | |||||
| router = APIRouter() | |||||
| # 存储活跃的WebSocket连接 | |||||
| active_connection: Optional[WebSocket] = None | |||||
| @dataclass | |||||
| class WebSocketMessage: | |||||
| """WebSocket消息数据类""" | |||||
| type: str | |||||
| msg: str = "" | |||||
| title: str = "提示" | |||||
| duration: int = 1 | |||||
| data: Dict = None | |||||
| def __post_init__(self): | |||||
| if self.data is None: | |||||
| self.data = {"x": 0, "y": 0, "key_str": ""} | |||||
| def to_dict(self) -> Dict: | |||||
| """将消息转换为字典格式""" | |||||
| return { | |||||
| "type": self.type, | |||||
| "msg": self.msg, | |||||
| "title": self.title, | |||||
| "duration": self.duration, | |||||
| "data": self.data, | |||||
| } | |||||
| class WebSocketMessageType: | |||||
| """WebSocket消息类型常量""" | |||||
| # 系统消息类型 | |||||
| INFO = "info" | |||||
| SUCCESS = "success" | |||||
| WARNING = "warning" | |||||
| ERROR = "error" | |||||
| # 鼠标操作类型 | |||||
| MOUSE_MOVE = "mouse_move" | |||||
| MOUSE_CLICK = "mouse_click" | |||||
| MOUSE_SCROLL_UP = "mouse_scroll_up" | |||||
| MOUSE_SCROLL_DOWN = "mouse_scroll_down" | |||||
| # 键盘操作类型 | |||||
| SEND_KEYS = "send_keys" | |||||
| # 语音操作类型 | |||||
| VOICE_RECORD = "voice_record" | |||||
| VOICE_STOP = "voice_stop" | |||||
| class MessageSender: | |||||
| """消息发送器,发送给前端""" | |||||
| @staticmethod | |||||
| async def send_message( | |||||
| ws_data_type: str, msg: str, title: str = "提示", duration: int = 1 | |||||
| ) -> None: | |||||
| """ | |||||
| 发送消息到WebSocket客户端 | |||||
| Args: | |||||
| ws_data_type: 消息类型 | |||||
| msg: 消息内容 | |||||
| title: 消息标题 | |||||
| duration: 显示持续时间 | |||||
| """ | |||||
| if not active_connection: | |||||
| return | |||||
| try: | |||||
| message = WebSocketMessage( | |||||
| type=ws_data_type, msg=msg, title=title, duration=duration | |||||
| ) | |||||
| await active_connection.send_json(message.to_dict()) | |||||
| except Exception as e: | |||||
| print(f"发送消息失败: {e}") | |||||
| class GestureHandler: | |||||
| """手势控制器""" | |||||
| def __init__(self): | |||||
| self.keyboard = KeyboardController() | |||||
| self.mouse = Controller() | |||||
| def move_mouse(self, x: int, y: int) -> None: | |||||
| """移动鼠标到指定位置""" | |||||
| self.mouse.position = (x, y) | |||||
| def click_mouse(self) -> None: | |||||
| """点击鼠标左键""" | |||||
| self.mouse.click(Button.left) | |||||
| def scroll_up(self) -> None: | |||||
| """向上滚动""" | |||||
| self.mouse.scroll(0, 1) | |||||
| def scroll_down(self) -> None: | |||||
| """向下滚动""" | |||||
| self.mouse.scroll(0, -1) | |||||
| def send_keys(self, key_str: str) -> None: | |||||
| """ | |||||
| 发送按键事件（支持组合键） | |||||
| Args: | |||||
| key_str: 按键字符串（如 'ctrl+r' 或 'F11'） | |||||
| """ | |||||
| try: | |||||
| keys = [self.__parse_key(key) for key in key_str.split("+")] | |||||
| self.__execute_keys(keys) | |||||
| except Exception as e: | |||||
| print(f"发送按键失败: {e}") | |||||
| def __parse_key(self, key_str: str) -> Union[str, Key]: | |||||
| """ | |||||
| 解析单个按键 | |||||
| Args: | |||||
| key_str: 按键字符串 | |||||
| Returns: | |||||
| 解析后的按键对象 | |||||
| """ | |||||
| key_str = key_str.strip().lower() | |||||
| if hasattr(Key, key_str): | |||||
| return getattr(Key, key_str) | |||||
| elif len(key_str) == 1: | |||||
| return key_str | |||||
| elif key_str.startswith("f"): | |||||
| try: | |||||
| return getattr(Key, key_str) | |||||
| except AttributeError: | |||||
| raise ValueError(f"无效的功能键: {key_str}") | |||||
| else: | |||||
| raise ValueError(f"无效的按键: {key_str}") | |||||
| def __execute_keys(self, keys: List[Union[str, Key]]) -> None: | |||||
| """ | |||||
| 执行按键序列 | |||||
| Args: | |||||
| keys: 按键列表 | |||||
| """ | |||||
| pressed_keys = [] | |||||
| try: | |||||
| # 按下所有键 | |||||
| for key in keys: | |||||
| self.keyboard.press(key) | |||||
| pressed_keys.append(key) | |||||
| # 释放所有键（按相反顺序） | |||||
| for key in reversed(pressed_keys): | |||||
| self.keyboard.release(key) | |||||
| except Exception as e: | |||||
| print(f"执行按键失败: {e}") | |||||
| class VoiceHandler: | |||||
| """语音处理控制器""" | |||||
| def __init__(self): | |||||
| from VoiceController import VoiceController | |||||
| self.controller: Optional[VoiceController] = None | |||||
| try: | |||||
| self.controller = VoiceController() | |||||
| except Exception as e: | |||||
| print(f"语音控制器初始化失败: {e}") | |||||
| async def start_recording(self, websocket: WebSocket) -> None: | |||||
| """开始录音""" | |||||
| if self.controller and not self.controller.is_recording: | |||||
| self.controller.start_record_thread() | |||||
| async def stop_recording( | |||||
| self, websocket: WebSocket, gesture_handler: GestureHandler | |||||
| ) -> None: | |||||
| """停止录音并处理结果""" | |||||
| if self.controller and self.controller.is_recording: | |||||
| self.controller.stop_record() | |||||
| # 获取识别结果并输入 | |||||
| text = self.controller.transcribe_audio() | |||||
| if text: | |||||
| gesture_handler.keyboard.type(text) | |||||
| gesture_handler.keyboard.tap(Key.enter) | |||||
| @router.websocket("/ws_wavecontrol") | |||||
| async def websocket_endpoint(websocket: WebSocket): | |||||
| """WebSocket端点处理函数""" | |||||
| global active_connection | |||||
| await websocket.accept() | |||||
| active_connection = websocket | |||||
| gesture_handler = GestureHandler() | |||||
| voice_handler = VoiceHandler() | |||||
| try: | |||||
| while True: | |||||
| data_str = await websocket.receive_text() | |||||
| await _handle_message(data_str, websocket, gesture_handler, voice_handler) | |||||
| except WebSocketDisconnect: | |||||
| active_connection = None | |||||
| except Exception as e: | |||||
| print(f"WebSocket处理错误: {e}") | |||||
| active_connection = None | |||||
| async def _handle_message( | |||||
| data_str: str, | |||||
| websocket: WebSocket, | |||||
| gesture_handler: GestureHandler, | |||||
| voice_handler: VoiceHandler, | |||||
| ) -> None: | |||||
| """处理WebSocket消息""" | |||||
| try: | |||||
| message = WebSocketMessage(**json.loads(data_str)) | |||||
| data = message.data | |||||
| # 处理鼠标操作 | |||||
| if message.type == WebSocketMessageType.MOUSE_MOVE: | |||||
| gesture_handler.move_mouse(data["x"], data["y"]) | |||||
| elif message.type == WebSocketMessageType.MOUSE_CLICK: | |||||
| gesture_handler.click_mouse() | |||||
| elif message.type == WebSocketMessageType.MOUSE_SCROLL_UP: | |||||
| gesture_handler.scroll_up() | |||||
| elif message.type == WebSocketMessageType.MOUSE_SCROLL_DOWN: | |||||
| gesture_handler.scroll_down() | |||||
| # 处理键盘操作 | |||||
| elif message.type == WebSocketMessageType.SEND_KEYS: | |||||
| gesture_handler.send_keys(data["key_str"]) | |||||
| # 处理语音操作 | |||||
| elif message.type == WebSocketMessageType.VOICE_RECORD: | |||||
| await voice_handler.start_recording(websocket) | |||||
| elif message.type == WebSocketMessageType.VOICE_STOP: | |||||
| await voice_handler.stop_recording(websocket, gesture_handler) | |||||
| except json.JSONDecodeError: | |||||
| print("无效的JSON数据") | |||||
| except Exception as e: | |||||
| print(f"处理消息失败: {e}") | |||||
BIN
src-tauri/.DS_Store
View File
+ 13
- 0
src-tauri/.gitignore
View File
| @ -0,0 +1,13 @@ | |||||
| 2 | |||||
| Cargo.lock | |||||
| bin/ | |||||
| icons/ | |||||
| # Generated by Cargo | |||||
| # will have compiled files and executables | |||||
| /target/ | |||||
| # Generated by Tauri | |||||
| # will have schema files for capabilities auto-completion | |||||
| /gen/schemas | |||||
+ 32
- 0
src-tauri/Cargo.toml
View File
| @ -0,0 +1,32 @@ | |||||
| [package] | |||||
| name = "Lazyeat" | |||||
| version = "0.3.11" | |||||
| description = "Lazyeat 手势识别" | |||||
| authors = ["https://github.com/maplelost"] | |||||
| edition = "2021" | |||||
| # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html | |||||
| [lib] | |||||
| # The `_lib` suffix may seem redundant but it is necessary | |||||
| # to make the lib name unique and wouldn't conflict with the bin name. | |||||
| # This seems to be only an issue on Windows, see https://github.com/rust-lang/cargo/issues/8519 | |||||
| name = "tauri_app_lib" | |||||
| crate-type = ["staticlib", "cdylib", "rlib"] | |||||
| [build-dependencies] | |||||
| tauri-build = { version = "2", features = [] } | |||||
| [dependencies] | |||||
| tauri = { version = "2", features = ["devtools"] } | |||||
| tauri-plugin-opener = "2" | |||||
| serde = { version = "1", features = ["derive"] } | |||||
| serde_json = "1" | |||||
| tauri-plugin-shell = "2" | |||||
| tauri-plugin-store = "2" | |||||
| tauri-plugin-notification = "2" | |||||
| [target.'cfg(not(any(target_os = "android", target_os = "ios")))'.dependencies] | |||||
| tauri-plugin-autostart = "2" | |||||
| tauri-plugin-window-state = "2" | |||||
+ 8
- 0
src-tauri/Info.plist
View File
| @ -0,0 +1,8 @@ | |||||
| <?xml version="1.0" encoding="UTF-8"?> | |||||
| <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> | |||||
| <plist version="1.0"> | |||||
| <dict> | |||||
| <key>NSCameraUsageDescription</key> | |||||
| <string>请允许本程序访问您的摄像头</string> | |||||
| </dict> | |||||
| </plist> | |||||
+ 3
- 0
src-tauri/build.rs
View File
| @ -0,0 +1,3 @@ | |||||
| fn main() { | |||||
| tauri_build::build() | |||||
| } | |||||
+ 41
- 0
src-tauri/capabilities/default.json
View File
| @ -0,0 +1,41 @@ | |||||
| { | |||||
| "$schema": "../gen/schemas/desktop-schema.json", | |||||
| "identifier": "default", | |||||
| "description": "Capability for the main window", | |||||
| "windows": ["main", "NewWindow_2"], | |||||
| "permissions": [ | |||||
| "core:default", | |||||
| "shell:default", | |||||
| "shell:allow-execute", | |||||
| "shell:allow-spawn", | |||||
| "core:path:default", | |||||
| "core:event:default", | |||||
| "core:window:default", | |||||
| "core:app:default", | |||||
| "core:resources:default", | |||||
| "core:menu:default", | |||||
| "core:tray:default", | |||||
| "core:window:allow-destroy", | |||||
| "core:window:allow-set-title", | |||||
| "store:default", | |||||
| "notification:default", | |||||
| "notification:allow-is-permission-granted", | |||||
| "notification:allow-notify", | |||||
| "notification:allow-show", | |||||
| "notification:allow-request-permission", | |||||
| "opener:default", | |||||
| "opener:allow-open-path", | |||||
| "opener:allow-reveal-item-in-dir", | |||||
| "opener:allow-default-urls", | |||||
| "opener:allow-open-url", | |||||
| "core:webview:default", | |||||
| "core:window:allow-show", | |||||
| "core:window:allow-hide", | |||||
| "core:webview:allow-create-webview-window", | |||||
| "core:window:allow-set-position", | |||||
| "core:window:allow-set-size" | |||||
| ] | |||||
| } | |||||
+ 17
- 0
src-tauri/capabilities/desktop.json
View File
| @ -0,0 +1,17 @@ | |||||
| { | |||||
| "identifier": "desktop-capability", | |||||
| "platforms": [ | |||||
| "macOS", | |||||
| "windows", | |||||
| "linux" | |||||
| ], | |||||
| "windows": [ | |||||
| "main" | |||||
| ], | |||||
| "permissions": [ | |||||
| "autostart:allow-enable", | |||||
| "autostart:allow-disable", | |||||
| "autostart:allow-is-enabled", | |||||
| "window-state:default" | |||||
| ] | |||||
| } | |||||
+ 58
- 0
src-tauri/src/lib.rs
View File
| @ -0,0 +1,58 @@ | |||||
| // Learn more about Tauri commands at https://tauri.app/develop/calling-rust/ | |||||
| #[tauri::command] | |||||
| fn greet(name: &str) -> String { | |||||
| format!("Hello, {}! You've been greeted from Rust!", name) | |||||
| } | |||||
| // 提取sidecar启动逻辑到单独的函数 | |||||
| async fn start_sidecar(app: tauri::AppHandle) -> Result<String, String> { | |||||
| let sidecar = app | |||||
| .shell() | |||||
| .sidecar("Lazyeat Backend") | |||||
| .map_err(|e| format!("无法找到sidecar: {}", e))?; | |||||
| let (_rx, _child) = sidecar | |||||
| .spawn() | |||||
| .map_err(|e| format!("无法启动sidecar: {}", e))?; | |||||
| Ok("Sidecar已启动".to_string()) | |||||
| } | |||||
| // 保留命令供可能的手动调用 | |||||
| #[tauri::command] | |||||
| async fn run_sidecar(app: tauri::AppHandle) -> Result<String, String> { | |||||
| start_sidecar(app).await | |||||
| } | |||||
| use tauri_plugin_autostart::MacosLauncher; | |||||
| use tauri_plugin_autostart::ManagerExt; | |||||
| use tauri_plugin_shell::process::CommandEvent; | |||||
| use tauri_plugin_shell::ShellExt; | |||||
| #[cfg_attr(mobile, tauri::mobile_entry_point)] | |||||
| pub fn run() { | |||||
| tauri::Builder::default() | |||||
| .plugin(tauri_plugin_notification::init()) | |||||
| // .plugin(tauri_plugin_window_state::Builder::new().build()) // 窗口状态管理,启用了导致 sub-window 无法设置decorations | |||||
| .plugin(tauri_plugin_store::Builder::new().build()) | |||||
| .plugin(tauri_plugin_autostart::init( | |||||
| MacosLauncher::LaunchAgent, | |||||
| Some(vec!["--flag1", "--flag2"]), | |||||
| )) | |||||
| .plugin(tauri_plugin_shell::init()) | |||||
| .plugin(tauri_plugin_opener::init()) | |||||
| .setup(|app| { | |||||
| // 在应用启动时自动启动sidecar | |||||
| let app_handle = app.handle().clone(); | |||||
| tauri::async_runtime::spawn(async move { | |||||
| match start_sidecar(app_handle).await { | |||||
| Ok(msg) => println!("{}", msg), | |||||
| Err(e) => eprintln!("启动sidecar失败: {}", e), | |||||
| } | |||||
| }); | |||||
| Ok(()) | |||||
| }) | |||||
| .invoke_handler(tauri::generate_handler![greet, run_sidecar]) | |||||
| .run(tauri::generate_context!()) | |||||
| .expect("error while running tauri application"); | |||||
| } | |||||
+ 6
- 0
src-tauri/src/main.rs
View File
| @ -0,0 +1,6 @@ | |||||
| // Prevents additional console window on Windows in release, DO NOT REMOVE!! | |||||
| #![cfg_attr(not(debug_assertions), windows_subsystem = "windows")] | |||||
| fn main() { | |||||
| tauri_app_lib::run() | |||||
| } | |||||
+ 61
- 0
src-tauri/tauri.conf.json
View File
| @ -0,0 +1,61 @@ | |||||
| { | |||||
| "$schema": "https://schema.tauri.app/config/2", | |||||
| "productName": "Lazyeat", | |||||
| "version": "0.3.11", | |||||
| "identifier": "com.Lazyeat.maplelost", | |||||
| "build": { | |||||
| "beforeDevCommand": "npm run dev", | |||||
| "devUrl": "http://localhost:1420", | |||||
| "beforeBuildCommand": "npm run build", | |||||
| "frontendDist": "../dist" | |||||
| }, | |||||
| "app": { | |||||
| "windows": [ | |||||
| { | |||||
| "title": "Lazyeat", | |||||
| "width": 800, | |||||
| "height": 600, | |||||
| "devtools": true | |||||
| } | |||||
| ], | |||||
| "security": { | |||||
| "csp": null | |||||
| } | |||||
| }, | |||||
| "bundle": { | |||||
| "active": true, | |||||
| "externalBin": ["bin/backend-py/Lazyeat Backend"], | |||||
| "resources": { | |||||
| "bin/backend-py/_internal": "_internal/", | |||||
| "../model": "model/", | |||||
| "../debug backend.bat": "/" | |||||
| }, | |||||
| "macOS": { | |||||
| "dmg": { | |||||
| "appPosition": { | |||||
| "x": 180, | |||||
| "y": 170 | |||||
| }, | |||||
| "applicationFolderPosition": { | |||||
| "x": 480, | |||||
| "y": 170 | |||||
| }, | |||||
| "windowSize": { | |||||
| "height": 400, | |||||
| "width": 660 | |||||
| } | |||||
| }, | |||||
| "files": {}, | |||||
| "hardenedRuntime": true, | |||||
| "minimumSystemVersion": "10.13" | |||||
| }, | |||||
| "targets": ["dmg", "msi"], | |||||
| "icon": [ | |||||
| "icons/32x32.png", | |||||
| "icons/128x128.png", | |||||
| "icons/128x128@2x.png", | |||||
| "icons/mac/icon.icns", | |||||
| "icons/icon.ico" | |||||
| ] | |||||
| } | |||||
| } | |||||
+ 13
- 0
src-tauri/tauri.macos.conf.json
View File
| @ -0,0 +1,13 @@ | |||||
| { | |||||
| "$schema": "https://schema.tauri.app/config/2", | |||||
| "identifier": "com.Lazyeat.maplelost", | |||||
| "bundle": { | |||||
| "resources": [], | |||||
| "macOS": { | |||||
| "files": { | |||||
| "Resources/model": "../model", | |||||
| "Frameworks": "./bin/backend-py/_internal" | |||||
| } | |||||
| } | |||||
| } | |||||
| } | |||||
BIN
src/.DS_Store
View File
+ 35
- 0
src/App.vue
View File
| @ -0,0 +1,35 @@ | |||||
| <template> | |||||
| <router-view /> | |||||
| </template> | |||||
| <style lang="scss"> | |||||
| html, | |||||
| body { | |||||
| margin: 0; | |||||
| padding: 0; | |||||
| height: 100%; | |||||
| } | |||||
| #app { | |||||
| height: 100%; | |||||
| } | |||||
| // 链接样式 | |||||
| .contributor-link { | |||||
| color: #2196f3; // 默认使用适中的蓝色 | |||||
| text-decoration: none; | |||||
| transition: all 0.3s ease; // 添加过渡效果 | |||||
| &:visited { | |||||
| color: #2196f3; | |||||
| } | |||||
| &:hover { | |||||
| color: #1976d2; // 悬停时使用深蓝色 | |||||
| } | |||||
| &:active { | |||||
| color: #0d47a1 !important; // 点击时使用更深的蓝色 | |||||
| } | |||||
| } | |||||
| </style> | |||||
+ 34
- 0
src/AppMediaPipe.vue
View File
| @ -0,0 +1,34 @@ | |||||
| <template> | |||||
| <n-message-provider> | |||||
| <div class="app-container"> | |||||
| <h1>MediaPipe</h1> | |||||
| <div class="detection-container"> | |||||
| <hand-landmark-detection /> | |||||
| </div> | |||||
| </div> | |||||
| </n-message-provider> | |||||
| </template> | |||||
| <script setup lang="ts"> | |||||
| import HandLandmarkDetection from "@/hand_landmark/VideoDetector.vue"; | |||||
| </script> | |||||
| <style scoped> | |||||
| .app-container { | |||||
| padding: 20px; | |||||
| display: flex; | |||||
| flex-direction: column; | |||||
| align-items: center; | |||||
| } | |||||
| h1 { | |||||
| color: #333; | |||||
| margin-bottom: 20px; | |||||
| } | |||||
| .detection-container { | |||||
| width: 100%; | |||||
| max-width: 640px; | |||||
| margin: 0 auto; | |||||
| } | |||||
| </style> | |||||
+ 31
- 0
src/components/AutoStart.vue
View File
| @ -0,0 +1,31 @@ | |||||
| <script setup lang="ts"> | |||||
| import { Power } from "@icon-park/vue-next"; | |||||
| import { disable, enable } from "@tauri-apps/plugin-autostart"; | |||||
| import { watch } from "vue"; | |||||
| import { use_app_store } from "@/store/app"; | |||||
| const app_store = use_app_store(); | |||||
| watch( | |||||
| () => app_store.config.auto_start, | |||||
| async (value) => { | |||||
| if (value) { | |||||
| await enable(); | |||||
| } else { | |||||
| await disable(); | |||||
| } | |||||
| } | |||||
| ); | |||||
| </script> | |||||
| <template> | |||||
| <n-space align="center" style="display: flex; align-items: center"> | |||||
| <span style="display: flex; align-items: center"> | |||||
| <n-icon size="20" style="margin-right: 8px"> | |||||
| <Power /> | |||||
| </n-icon> | |||||
| <span>{{ $t("开机自启动") }}</span> | |||||
| </span> | |||||
| <n-switch v-model:value="app_store.config.auto_start" /> | |||||
| </n-space> | |||||
| </template> | |||||
+ 77
- 0
src/components/CircleProgress.vue
View File
| @ -0,0 +1,77 @@ | |||||
| <template> | |||||
| <div class="circle-progress"> | |||||
| <svg :width="size" :height="size" viewBox="0 0 100 100"> | |||||
| <!-- 背景圆环 --> | |||||
| <circle | |||||
| cx="50" | |||||
| cy="50" | |||||
| :r="radius" | |||||
| fill="none" | |||||
| :stroke="backgroundColor" | |||||
| :stroke-width="strokeWidth" | |||||
| /> | |||||
| <!-- 进度圆环 --> | |||||
| <circle | |||||
| cx="50" | |||||
| cy="50" | |||||
| :r="radius" | |||||
| fill="none" | |||||
| :stroke="color" | |||||
| :stroke-width="strokeWidth" | |||||
| :stroke-dasharray="circumference" | |||||
| :stroke-dashoffset="dashOffset" | |||||
| class="progress" | |||||
| /> | |||||
| <!-- 中心文本 --> | |||||
| <slot> | |||||
| <text x="50" y="50" text-anchor="middle" dominant-baseline="middle" class="progress-text"> | |||||
| {{ text }} | |||||
| {{ percentage }}% | |||||
| </text> | |||||
| </slot> | |||||
| </svg> | |||||
| </div> | |||||
| </template> | |||||
| <script setup lang="ts"> | |||||
| import { computed } from 'vue'; | |||||
| interface Props { | |||||
| percentage: number; | |||||
| size?: number; | |||||
| strokeWidth?: number; | |||||
| color?: string; | |||||
| backgroundColor?: string; | |||||
| text?: string; | |||||
| } | |||||
| const props = withDefaults(defineProps<Props>(), { | |||||
| size: 100, | |||||
| strokeWidth: 6, | |||||
| color: '#409eff', | |||||
| backgroundColor: '#e5e9f2', | |||||
| text: '' | |||||
| }); | |||||
| const radius = computed(() => 50 - props.strokeWidth / 2); | |||||
| const circumference = computed(() => 2 * Math.PI * radius.value); | |||||
| const dashOffset = computed(() => | |||||
| circumference.value * (1 - props.percentage / 100) | |||||
| ); | |||||
| </script> | |||||
| <style lang="scss" scoped> | |||||
| .circle-progress { | |||||
| display: inline-block; | |||||
| .progress { | |||||
| transform: rotate(-90deg); | |||||
| transform-origin: center; | |||||
| } | |||||
| .progress-text { | |||||
| font-size: 14px; | |||||
| fill: #606266; | |||||
| } | |||||
| } | |||||
| </style> | |||||
+ 81
- 0
src/components/DevTool.vue
View File
| @ -0,0 +1,81 @@ | |||||
| <template> | |||||
| <div | |||||
| v-if="is_dev" | |||||
| class="dev-tool" | |||||
| :class="{ 'dev-tool--expanded': isExpanded }" | |||||
| > | |||||
| <div class="dev-tool__toggle" @click="toggleToolbox"> | |||||
| <span class="dev-tool__icon">🔧</span> | |||||
| </div> | |||||
| <!-- <div class="dev-tool__content" v-if="isExpanded"> | |||||
| <div class="dev-tool__item" @click="createSubWindowClick">创建子窗口</div> | |||||
| </div> --> | |||||
| </div> | |||||
| </template> | |||||
| <script setup lang="ts"> | |||||
| import { createSubWindow } from "@/utils/subWindow"; | |||||
| import { ref } from "vue"; | |||||
| const is_dev = import.meta.env.DEV; | |||||
| const isExpanded = ref(false); | |||||
| const toggleToolbox = () => { | |||||
| isExpanded.value = !isExpanded.value; | |||||
| }; | |||||
| const createSubWindowClick = () => { | |||||
| createSubWindow("/sub-window", "subWindow"); | |||||
| }; | |||||
| </script> | |||||
| <style scoped lang="scss"> | |||||
| .dev-tool { | |||||
| position: fixed; | |||||
| top: 50%; | |||||
| left: 0; | |||||
| transform: translateY(-50%); | |||||
| background-color: #fff; | |||||
| border-radius: 0 8px 8px 0; | |||||
| box-shadow: 0 2px 8px rgba(0, 0, 0, 0.1); | |||||
| transition: all 0.3s ease; | |||||
| z-index: 9999; | |||||
| &--expanded { | |||||
| width: 200px; | |||||
| } | |||||
| &__toggle { | |||||
| padding: 10px; | |||||
| cursor: pointer; | |||||
| display: flex; | |||||
| align-items: center; | |||||
| justify-content: center; | |||||
| border-bottom: 1px solid #eee; | |||||
| } | |||||
| &__icon { | |||||
| font-size: 20px; | |||||
| } | |||||
| &__content { | |||||
| padding: 10px; | |||||
| } | |||||
| &__item { | |||||
| padding: 8px 12px; | |||||
| cursor: pointer; | |||||
| border-radius: 4px; | |||||
| margin-bottom: 5px; | |||||
| transition: background-color 0.2s ease; | |||||
| &:hover { | |||||
| background-color: #f5f5f5; | |||||
| } | |||||
| &:last-child { | |||||
| margin-bottom: 0; | |||||
| } | |||||
| } | |||||
| } | |||||
| </style> | |||||
+ 82
- 0
src/components/GestureCard.vue
View File
| @ -0,0 +1,82 @@ | |||||
| <template> | |||||
| <n-card class="gesture-card" :bordered="false"> | |||||
| <n-space align="center" class="gesture-content"> | |||||
| <div class="gesture-icon" :class="{ 'double-hand': isDoubleHand }"> | |||||
| <slot name="icon"></slot> | |||||
| </div> | |||||
| <div class="gesture-info"> | |||||
| <h3>{{ title }}</h3> | |||||
| <p>{{ description }}</p> | |||||
| <slot name="extra"></slot> | |||||
| </div> | |||||
| </n-space> | |||||
| </n-card> | |||||
| </template> | |||||
| <script setup lang="ts"> | |||||
| defineProps<{ | |||||
| title: string; | |||||
| description: string; | |||||
| isDoubleHand?: boolean; | |||||
| }>(); | |||||
| </script> | |||||
| <style scoped lang="scss"> | |||||
| .gesture-card { | |||||
| transition: all 0.3s ease; | |||||
| background: linear-gradient(145deg, #f8faff, #ffffff); | |||||
| box-shadow: 0 4px 12px rgba(0, 0, 0, 0.05); | |||||
| border: 1px solid #e5e9f2; | |||||
| border-radius: 12px; | |||||
| &:hover { | |||||
| transform: translateY(-2px); | |||||
| box-shadow: 0 8px 24px rgba(64, 152, 252, 0.15); | |||||
| border-color: #4098fc; | |||||
| } | |||||
| } | |||||
| .gesture-content { | |||||
| padding: 12px; | |||||
| } | |||||
| .gesture-icon { | |||||
| background: rgba(64, 152, 252, 0.1); | |||||
| padding: 16px; | |||||
| border-radius: 12px; | |||||
| display: flex; | |||||
| align-items: center; | |||||
| justify-content: center; | |||||
| border: 1px solid rgba(64, 152, 252, 0.2); | |||||
| } | |||||
| .gesture-info { | |||||
| flex: 1; | |||||
| h3 { | |||||
| margin: 0 0 4px 0; | |||||
| font-size: 1.1rem; | |||||
| color: #2c3e50; | |||||
| } | |||||
| p { | |||||
| margin: 0; | |||||
| color: #666; | |||||
| font-size: 0.9rem; | |||||
| } | |||||
| } | |||||
| .double-hand { | |||||
| display: flex; | |||||
| gap: 8px; | |||||
| :deep(svg) { | |||||
| width: 32px; | |||||
| height: 32px; | |||||
| } | |||||
| } | |||||
| .flipped { | |||||
| transform: scaleX(-1); | |||||
| } | |||||
| </style> | |||||
+ 23
- 0
src/components/GestureIcon.vue
View File
| @ -0,0 +1,23 @@ | |||||
| <template> | |||||
| <component | |||||
| :is="icon" | |||||
| theme="outline" | |||||
| size="40" | |||||
| fill="#4098fc" | |||||
| :stroke-width="3" | |||||
| :class="{ flipped }" | |||||
| /> | |||||
| </template> | |||||
| <script setup lang="ts"> | |||||
| defineProps<{ | |||||
| icon: any; | |||||
| flipped?: boolean; | |||||
| }>(); | |||||
| </script> | |||||
| <style scoped> | |||||
| .flipped { | |||||
| transform: scaleX(-1); | |||||
| } | |||||
| </style> | |||||
+ 41
- 0
src/components/Menu.vue
View File
| @ -0,0 +1,41 @@ | |||||
| <script setup lang="ts"> | |||||
| import type { MenuOption } from "naive-ui"; | |||||
| import { NMenu } from "naive-ui"; | |||||
| import { ref } from "vue"; | |||||
| import { useRouter } from "vue-router"; | |||||
| const router = useRouter(); | |||||
| const menuOptions: MenuOption[] = [ | |||||
| { | |||||
| label: "首页", | |||||
| key: "/", | |||||
| }, | |||||
| { | |||||
| label: "操作指南", | |||||
| key: "/guide", | |||||
| }, | |||||
| // { | |||||
| // label: "开发", | |||||
| // key: "/update", | |||||
| // } | |||||
| ]; | |||||
| const activeKey = ref("/"); | |||||
| const handleUpdateValue = (key: string) => { | |||||
| activeKey.value = key; | |||||
| router.push(key); | |||||
| }; | |||||
| </script> | |||||
| <template> | |||||
| <n-menu | |||||
| :options="menuOptions" | |||||
| v-model:value="activeKey" | |||||
| mode="vertical" | |||||
| @update:value="handleUpdateValue" | |||||
| /> | |||||
| </template> | |||||
| <style scoped></style> | |||||
+ 259
- 0
src/hand_landmark/VideoDetector.vue
View File
| @ -0,0 +1,259 @@ | |||||
| <template> | |||||
| <div v-if="!camera_premission"> | |||||
| <n-alert title="获取摄像头权限失败" type="error"> | |||||
| <p>请尝试以下步骤解决:</p> | |||||
| <ol> | |||||
| <!-- <li> | |||||
| 删除文件夹 | |||||
| <n-tag size="small"> | |||||
| %LOCALAPPDATA%\com.Lazyeat.maplelost\EBWebView | |||||
| </n-tag> | |||||
| </li> --> | |||||
| <li> | |||||
| 进入<n-tag size="small">%LOCALAPPDATA%\com.Lazyeat.maplelost</n-tag> | |||||
| </li> | |||||
| <li>删除<n-tag size="small">EBWebView</n-tag>文件夹</li> | |||||
| <li>重新启动程序</li> | |||||
| </ol> | |||||
| </n-alert> | |||||
| </div> | |||||
| <div v-else> | |||||
| <span>FPS: {{ FPS }}</span> | |||||
| <div class="hand-detection"> | |||||
| <video | |||||
| ref="videoElement" | |||||
| class="input-video" | |||||
| :width="app_store.VIDEO_WIDTH" | |||||
| :height="app_store.VIDEO_HEIGHT" | |||||
| autoplay | |||||
| style="display: none" | |||||
| ></video> | |||||
| <canvas | |||||
| ref="canvasElement" | |||||
| class="output-canvas" | |||||
| :width="app_store.VIDEO_WIDTH" | |||||
| :height="app_store.VIDEO_HEIGHT" | |||||
| ></canvas> | |||||
| </div> | |||||
| <n-space> | |||||
| <n-form-item label="识别框x"> | |||||
| <n-input-number | |||||
| v-model:value="app_store.config.boundary_left" | |||||
| :min="0" | |||||
| :max="app_store.VIDEO_WIDTH - app_store.config.boundary_width - 10" | |||||
| style="width: 150px" | |||||
| /> | |||||
| </n-form-item> | |||||
| <n-form-item label="识别框y"> | |||||
| <n-input-number | |||||
| v-model:value="app_store.config.boundary_top" | |||||
| :min="0" | |||||
| :max="app_store.VIDEO_HEIGHT - app_store.config.boundary_height - 10" | |||||
| style="width: 150px" | |||||
| /> | |||||
| </n-form-item> | |||||
| <n-form-item label="识别框宽"> | |||||
| <n-input-number | |||||
| v-model:value="app_store.config.boundary_width" | |||||
| :min="0" | |||||
| :max="app_store.VIDEO_WIDTH - app_store.config.boundary_left - 10" | |||||
| style="width: 150px" | |||||
| /> | |||||
| </n-form-item> | |||||
| <n-form-item label="识别框高"> | |||||
| <n-input-number | |||||
| v-model:value="app_store.config.boundary_height" | |||||
| :min="0" | |||||
| :max="app_store.VIDEO_HEIGHT - app_store.config.boundary_top - 10" | |||||
| style="width: 150px" | |||||
| /> | |||||
| </n-form-item> | |||||
| </n-space> | |||||
| </div> | |||||
| </template> | |||||
| <script setup> | |||||
| import { Detector } from "@/hand_landmark/detector"; | |||||
| import { use_app_store } from "@/store/app"; | |||||
| import { onBeforeUnmount, onMounted, ref, watch } from "vue"; | |||||
| // 常量定义 | |||||
| const app_store = use_app_store(); | |||||
| // 组件状态 | |||||
| const videoElement = ref(null); | |||||
| const canvasElement = ref(null); | |||||
| const detector = ref(new Detector()); | |||||
| const lastVideoTime = ref(-1); | |||||
| const currentStream = ref(null); | |||||
| const FPS = ref(0); | |||||
| const camera_premission = ref(false); | |||||
| onMounted(() => { | |||||
| navigator.mediaDevices.getUserMedia({ video: true }).then((stream) => { | |||||
| camera_premission.value = true; | |||||
| stream.getTracks().forEach((track) => track.stop()); | |||||
| }); | |||||
| }); | |||||
| // 绘制相关方法 | |||||
| const drawMouseMoveBox = (ctx) => { | |||||
| ctx.strokeStyle = "rgb(255, 0, 255)"; | |||||
| ctx.lineWidth = 2; | |||||
| ctx.strokeRect( | |||||
| app_store.config.boundary_left, | |||||
| app_store.config.boundary_top, | |||||
| app_store.config.boundary_width, | |||||
| app_store.config.boundary_height | |||||
| ); | |||||
| }; | |||||
| const frameCount = ref(0); | |||||
| const fpsUpdateInterval = 1000; // 每秒更新一次 FPS | |||||
| const lastFpsTime = ref(0); | |||||
| const drawFPS = (ctx) => { | |||||
| const now = performance.now(); | |||||
| frameCount.value++; | |||||
| if (now - lastFpsTime.value >= fpsUpdateInterval) { | |||||
| FPS.value = frameCount.value; | |||||
| frameCount.value = 0; | |||||
| lastFpsTime.value = now; | |||||
| } | |||||
| }; | |||||
| const drawHandLandmarks = (ctx, hand, color) => { | |||||
| hand.landmarks.forEach((landmark) => { | |||||
| ctx.beginPath(); | |||||
| ctx.arc( | |||||
| landmark.x * app_store.VIDEO_WIDTH, | |||||
| landmark.y * app_store.VIDEO_HEIGHT, | |||||
| 5, | |||||
| 0, | |||||
| 2 * Math.PI | |||||
| ); | |||||
| ctx.fillStyle = color; | |||||
| ctx.fill(); | |||||
| }); | |||||
| }; | |||||
| // 主要检测逻辑 | |||||
| const predictWebcam = async () => { | |||||
| const video = videoElement.value; | |||||
| const canvas = canvasElement.value; | |||||
| const ctx = canvas.getContext("2d"); | |||||
| if (video.currentTime !== lastVideoTime.value) { | |||||
| lastVideoTime.value = video.currentTime; | |||||
| const detection = await detector.value.detect(video); | |||||
| if (app_store.config.show_window) { | |||||
| // 绘制视频帧 | |||||
| ctx.clearRect(0, 0, canvas.width, canvas.height); | |||||
| // 翻转绘制 | |||||
| ctx.save(); | |||||
| ctx.translate(canvas.width, 0); | |||||
| ctx.scale(-1, 1); | |||||
| ctx.drawImage(video, 0, 0, canvas.width, canvas.height); | |||||
| // 绘制手势点 | |||||
| if (detection.leftHand) { | |||||
| drawHandLandmarks(ctx, detection.leftHand, "red"); | |||||
| } | |||||
| if (detection.rightHand) { | |||||
| drawHandLandmarks(ctx, detection.rightHand, "blue"); | |||||
| } | |||||
| // 恢复绘制状态 | |||||
| ctx.restore(); | |||||
| // 绘制鼠标移动框 | |||||
| drawMouseMoveBox(ctx); | |||||
| } | |||||
| // 手势处理 | |||||
| await detector.value.process(detection); | |||||
| // 绘制FPS | |||||
| drawFPS(ctx); | |||||
| } | |||||
| requestAnimationFrame(predictWebcam); | |||||
| }; | |||||
| const initializeCamera = async () => { | |||||
| try { | |||||
| const stream = await navigator.mediaDevices.getUserMedia({ | |||||
| video: { | |||||
| deviceId: app_store.config.selected_camera_id | |||||
| ? { exact: app_store.config.selected_camera_id } | |||||
| : undefined, | |||||
| width: app_store.VIDEO_WIDTH, | |||||
| height: app_store.VIDEO_HEIGHT, | |||||
| }, | |||||
| audio: false, | |||||
| }); | |||||
| currentStream.value = stream; | |||||
| videoElement.value.srcObject = stream; | |||||
| videoElement.value.addEventListener("loadeddata", predictWebcam); | |||||
| } catch (error) { | |||||
| console.error("无法访问摄像头:", error); | |||||
| } | |||||
| }; | |||||
| const stopCamera = () => { | |||||
| if (videoElement.value?.srcObject) { | |||||
| videoElement.value.srcObject.getTracks().forEach((track) => track.stop()); | |||||
| } | |||||
| }; | |||||
| watch( | |||||
| () => app_store.config.selected_camera_id, | |||||
| async () => { | |||||
| stopCamera(); | |||||
| } | |||||
| ); | |||||
| // 监听 mission_running 的变化 | |||||
| watch( | |||||
| () => app_store.mission_running, | |||||
| async (newValue) => { | |||||
| if (newValue) { | |||||
| await initializeCamera(); | |||||
| app_store.flag_detecting = true; | |||||
| } else { | |||||
| stopCamera(); | |||||
| } | |||||
| } | |||||
| ); | |||||
| // 生命周期钩子 | |||||
| onMounted(async () => { | |||||
| await detector.value.initialize(); | |||||
| // 如果 mission_running 为 true，则初始化摄像头 | |||||
| if (app_store.mission_running) { | |||||
| await initializeCamera(); | |||||
| } | |||||
| }); | |||||
| onBeforeUnmount(() => { | |||||
| stopCamera(); | |||||
| }); | |||||
| </script> | |||||
| <style scoped> | |||||
| .hand-detection { | |||||
| width: v-bind('app_store.VIDEO_WIDTH + "px"'); | |||||
| height: v-bind('app_store.VIDEO_HEIGHT + "px"'); | |||||
| } | |||||
| .output-canvas { | |||||
| position: absolute; | |||||
| } | |||||
| </style> | |||||
+ 291
- 0
src/hand_landmark/detector.ts
View File
| @ -0,0 +1,291 @@ | |||||
| import { GestureHandler } from "@/hand_landmark/gesture_handler"; | |||||
| import { | |||||
| FilesetResolver, | |||||
| GestureRecognizer, | |||||
| GestureRecognizerOptions, | |||||
| } from "@mediapipe/tasks-vision"; | |||||
| // 手势枚举 | |||||
| export enum HandGesture { | |||||
| // 食指举起，移动鼠标 | |||||
| ONLY_INDEX_UP = "only_index_up", | |||||
| // 食指和拇指同时竖起 - 鼠标左键点击 | |||||
| INDEX_AND_THUMB_UP = "index_and_thumb_up", | |||||
| ROCK_GESTURE = "rock_gesture", | |||||
| // 三根手指同时竖起 - 滚动屏幕 | |||||
| THREE_FINGERS_UP = "three_fingers_up", | |||||
| SCROLL_GESTURE_2 = "scroll_gesture_2", | |||||
| // 四根手指同时竖起 | |||||
| FOUR_FINGERS_UP = "four_fingers_up", | |||||
| // 五根手指同时竖起 - 暂停/开始 识别 | |||||
| STOP_GESTURE = "stop_gesture", | |||||
| // 其他手势 | |||||
| DELETE_GESTURE = "delete_gesture", | |||||
| OTHER = "other", | |||||
| } | |||||
| interface HandLandmark { | |||||
| x: number; | |||||
| y: number; | |||||
| z: number; | |||||
| } | |||||
| export interface HandInfo { | |||||
| landmarks: HandLandmark[]; | |||||
| handedness: "Left" | "Right"; | |||||
| score: number; | |||||
| categoryName?: string; | |||||
| } | |||||
| interface DetectionResult { | |||||
| leftHand?: HandInfo; | |||||
| rightHand?: HandInfo; | |||||
| // 原始检测结果，以防需要访问其他数据 | |||||
| rawResult: any; | |||||
| } | |||||
| /** | |||||
| * 检测器类 - 负责手势识别和手势分类 | |||||
| * 主要职责: | |||||
| * 1. 初始化和管理MediaPipe HandLandmarker | |||||
| * 2. 检测视频帧中的手部 | |||||
| * 3. 分析手势类型(手指竖起等) | |||||
| * 4. 提供手部关键点查询方法 | |||||
| */ | |||||
| export class Detector { | |||||
| private detector: GestureRecognizer | null = null; | |||||
| private gestureHandler: GestureHandler | null = null; | |||||
| async initialize(useCanvas = false) { | |||||
| const vision = await FilesetResolver.forVisionTasks("/mediapipe/wasm"); | |||||
| try { | |||||
| const params = { | |||||
| baseOptions: { | |||||
| modelAssetPath: "/mediapipe/gesture_recognizer.task", | |||||
| delegate: "GPU", | |||||
| }, | |||||
| runningMode: "VIDEO", | |||||
| numHands: 1, | |||||
| } as GestureRecognizerOptions; | |||||
| if (useCanvas) { | |||||
| params.canvas = document.createElement("canvas"); | |||||
| } | |||||
| this.detector = await GestureRecognizer.createFromOptions(vision, params); | |||||
| this.gestureHandler = new GestureHandler(); | |||||
| } catch (error: any) { | |||||
| // macos 旧设备的 wkwebview 对 webgl 兼容性不好，需要手动创建 canvas | |||||
| if (error.toString().includes("kGpuService")) { | |||||
| await this.initialize(true); | |||||
| } else { | |||||
| throw error; | |||||
| } | |||||
| } | |||||
| } | |||||
| /** | |||||
| * 从视频帧检测手部 | |||||
| */ | |||||
| async detect(video: HTMLVideoElement): Promise<DetectionResult> { | |||||
| if (!this.detector) { | |||||
| throw new Error("检测器未初始化"); | |||||
| } | |||||
| const result = await this.detector.recognize(video); | |||||
| const detection: DetectionResult = { | |||||
| rawResult: result, | |||||
| }; | |||||
| if (result.landmarks && result.handedness) { | |||||
| for (let i = 0; i < result.landmarks.length; i++) { | |||||
| const hand: HandInfo = { | |||||
| landmarks: result.landmarks[i], | |||||
| handedness: result.handedness[i][0].categoryName as "Left" | "Right", | |||||
| score: result.handedness[i][0].score, | |||||
| }; | |||||
| if (result.gestures.length > 0) { | |||||
| hand.categoryName = result.gestures[0][0].categoryName; | |||||
| } | |||||
| if (hand.handedness === "Left") { | |||||
| detection.leftHand = hand; | |||||
| } else { | |||||
| detection.rightHand = hand; | |||||
| } | |||||
| } | |||||
| } | |||||
| return detection; | |||||
| } | |||||
| /** | |||||
| * 便捷方法：获取特定手指的关键点 | |||||
| */ | |||||
| static getFingerLandmarks( | |||||
| hand: HandInfo | undefined, | |||||
| fingerIndex: number | |||||
| ): HandLandmark[] | null { | |||||
| if (!hand) return null; | |||||
| const fingerIndices = { | |||||
| thumb: [1, 2, 3, 4], | |||||
| index: [5, 6, 7, 8], | |||||
| middle: [9, 10, 11, 12], | |||||
| ring: [13, 14, 15, 16], | |||||
| pinky: [17, 18, 19, 20], | |||||
| }; | |||||
| const indices = Object.values(fingerIndices)[fingerIndex]; | |||||
| return indices.map((i) => hand.landmarks[i]); | |||||
| } | |||||
| /** | |||||
| * 获取手指尖点 | |||||
| */ | |||||
| static getFingerTip( | |||||
| hand: HandInfo | undefined, | |||||
| fingerIndex: number | |||||
| ): HandLandmark | null { | |||||
| if (!hand) return null; | |||||
| const tipIndices = [4, 8, 12, 16, 20]; | |||||
| return hand.landmarks[tipIndices[fingerIndex]]; | |||||
| } | |||||
| /** | |||||
| * 检测手指是否竖起 | |||||
| */ | |||||
| static _fingersUp(hand: HandInfo): number[] { | |||||
| const fingers: number[] = []; | |||||
| const tipIds = [4, 8, 12, 16, 20]; // 从大拇指开始，依次为每个手指指尖 | |||||
| // 检测大拇指 | |||||
| if (hand.handedness === "Right") { | |||||
| if (hand.landmarks[tipIds[0]].x < hand.landmarks[tipIds[0] - 1].x) { | |||||
| fingers.push(0); | |||||
| } else { | |||||
| fingers.push(1); | |||||
| } | |||||
| } else { | |||||
| if (hand.landmarks[tipIds[0]].x > hand.landmarks[tipIds[0] - 1].x) { | |||||
| fingers.push(0); | |||||
| } else { | |||||
| fingers.push(1); | |||||
| } | |||||
| } | |||||
| // 检测其他四个手指 | |||||
| for (let id = 1; id < 5; id++) { | |||||
| if (hand.landmarks[tipIds[id]].y < hand.landmarks[tipIds[id] - 2].y) { | |||||
| fingers.push(1); | |||||
| } else { | |||||
| fingers.push(0); | |||||
| } | |||||
| } | |||||
| return fingers; | |||||
| } | |||||
| /** | |||||
| * 获取单个手的手势类型 | |||||
| */ | |||||
| public static getSingleHandGesture(hand: HandInfo): HandGesture { | |||||
| const fingers = this._fingersUp(hand); | |||||
| const fingerState = fingers.join(","); | |||||
| // 定义手势映射表 | |||||
| const gestureMap = new Map<string, HandGesture>([ | |||||
| // 食指举起，移动鼠标 | |||||
| ["0,1,0,0,0", HandGesture.ONLY_INDEX_UP], | |||||
| // 鼠标左键点击手势 | |||||
| ["1,1,0,0,0", HandGesture.INDEX_AND_THUMB_UP], | |||||
| ["0,1,0,0,1", HandGesture.ROCK_GESTURE], | |||||
| ["1,1,0,0,1", HandGesture.ROCK_GESTURE], | |||||
| // 滚动屏幕手势 | |||||
| ["0,1,1,1,0", HandGesture.THREE_FINGERS_UP], | |||||
| ["1,0,1,1,1", HandGesture.SCROLL_GESTURE_2], | |||||
| ["0,0,1,1,1", HandGesture.SCROLL_GESTURE_2], | |||||
| // 四根手指同时竖起 | |||||
| ["0,1,1,1,1", HandGesture.FOUR_FINGERS_UP], | |||||
| // 五根手指同时竖起 - 暂停/开始 识别 | |||||
| ["1,1,1,1,1", HandGesture.STOP_GESTURE], | |||||
| // 拇指和食指同时竖起 - 语音识别 | |||||
| ["1,0,0,0,1", HandGesture.VOICE_GESTURE_START], | |||||
| // 其他手势 | |||||
| ["0,0,0,0,0", HandGesture.VOICE_GESTURE_STOP], | |||||
| ]); | |||||
| if (gestureMap.has(fingerState)) { | |||||
| return gestureMap.get(fingerState) as HandGesture; | |||||
| } | |||||
| // 检查删除手势 | |||||
| if (this._isDeleteGesture(hand, fingers)) { | |||||
| return HandGesture.DELETE_GESTURE; | |||||
| } | |||||
| // 返回默认值 | |||||
| return HandGesture.OTHER; | |||||
| } | |||||
| /** | |||||
| * 检查是否为删除手势 | |||||
| */ | |||||
| private static _isDeleteGesture(hand: HandInfo, fingers: number[]): boolean { | |||||
| const THUMB_INDEX = 4; | |||||
| const FINGER_TIPS = [8, 12, 16, 20]; | |||||
| const distance_threshold = 0.05; | |||||
| const isThumbExtended = fingers[0] === 1; | |||||
| const areOtherFingersClosed = fingers | |||||
| .slice(1) | |||||
| .every((finger) => finger === 0); | |||||
| const isThumbLeftmost = FINGER_TIPS.every( | |||||
| (tipIndex) => | |||||
| hand.landmarks[THUMB_INDEX].x > | |||||
| hand.landmarks[tipIndex].x + distance_threshold | |||||
| ); | |||||
| return isThumbExtended && areOtherFingersClosed && isThumbLeftmost; | |||||
| } | |||||
| /** | |||||
| * 处理检测结果并执行相应动作 | |||||
| */ | |||||
| async process(detection: DetectionResult): Promise<void> { | |||||
| const rightHandGesture = detection.rightHand | |||||
| ? Detector.getSingleHandGesture(detection.rightHand) | |||||
| : HandGesture.OTHER; | |||||
| const leftHandGesture = detection.leftHand | |||||
| ? Detector.getSingleHandGesture(detection.leftHand) | |||||
| : HandGesture.OTHER; | |||||
| // 优先使用右手 | |||||
| let effectiveGesture = rightHandGesture; | |||||
| if (detection.rightHand) { | |||||
| effectiveGesture = rightHandGesture; | |||||
| } else if (detection.leftHand) { | |||||
| effectiveGesture = leftHandGesture; | |||||
| } | |||||
| // 将手势处理交给GestureHandler | |||||
| if (detection.rightHand) { | |||||
| this.gestureHandler?.handleGesture(effectiveGesture, detection.rightHand); | |||||
| } else if (detection.leftHand) { | |||||
| this.gestureHandler?.handleGesture(effectiveGesture, detection.leftHand); | |||||
| } | |||||
| } | |||||
| } | |||||
+ 507
- 0
src/hand_landmark/gesture_handler.ts
View File
| @ -0,0 +1,507 @@ | |||||
| import { HandGesture, HandInfo } from "@/hand_landmark/detector"; | |||||
| import i18n from "@/locales/i18n"; | |||||
| import use_app_store from "@/store/app"; | |||||
| // WebSocket数据类型定义 | |||||
| enum WsDataType { | |||||
| // 系统消息类型 | |||||
| INFO = "info", | |||||
| SUCCESS = "success", | |||||
| WARNING = "warning", | |||||
| ERROR = "error", | |||||
| // 鼠标操作类型 | |||||
| MOUSE_MOVE = "mouse_move", | |||||
| MOUSE_CLICK = "mouse_click", | |||||
| MOUSE_SCROLL_UP = "mouse_scroll_up", | |||||
| MOUSE_SCROLL_DOWN = "mouse_scroll_down", | |||||
| // 键盘操作类型 | |||||
| SEND_KEYS = "send_keys", | |||||
| // 语音操作类型 | |||||
| VOICE_RECORD = "voice_record", | |||||
| VOICE_STOP = "voice_stop", | |||||
| } | |||||
| interface WsData { | |||||
| type: WsDataType; | |||||
| msg?: string; | |||||
| duration?: number; | |||||
| title?: string; | |||||
| data?: { | |||||
| x?: number; | |||||
| y?: number; | |||||
| key_str?: string; | |||||
| }; | |||||
| } | |||||
| /** | |||||
| * 动作触发器类 - 负责发送操作命令到系统 | |||||
| * 主要职责： | |||||
| * 1. 维护WebSocket连接 | |||||
| * 2. 提供各种操作方法（移动鼠标、点击等） | |||||
| * 3. 发送通知 | |||||
| */ | |||||
| export class TriggerAction { | |||||
| private ws: WebSocket | null = null; | |||||
| constructor() { | |||||
| this.connectWebSocket(); | |||||
| } | |||||
| private connectWebSocket() { | |||||
| try { | |||||
| this.ws = new WebSocket("ws://127.0.0.1:62334/ws_wavecontrol"); | |||||
| this.ws.onmessage = (event: MessageEvent) => { | |||||
| const response: WsData = JSON.parse(event.data); | |||||
| const app_store = use_app_store(); | |||||
| app_store.sub_window_info(response.msg || ""); | |||||
| }; | |||||
| this.ws.onopen = () => { | |||||
| console.log("ws_wavecontrol connected"); | |||||
| }; | |||||
| this.ws.onclose = () => { | |||||
| console.log("ws_wavecontrol closed, retrying..."); | |||||
| this.ws = null; | |||||
| setTimeout(() => this.connectWebSocket(), 3000); | |||||
| }; | |||||
| this.ws.onerror = (error) => { | |||||
| console.error("ws_wavecontrol error:", error); | |||||
| this.ws?.close(); | |||||
| }; | |||||
| } catch (error) { | |||||
| console.error("Failed to create WebSocket instance:", error); | |||||
| this.ws = null; | |||||
| setTimeout(() => this.connectWebSocket(), 1000); | |||||
| } | |||||
| } | |||||
| private send(data: { type: WsDataType } & Partial<Omit<WsData, "type">>) { | |||||
| const message: WsData = { | |||||
| type: data.type, | |||||
| msg: data.msg || "", | |||||
| title: data.title || "Lazyeat", | |||||
| duration: data.duration || 1, | |||||
| data: data.data || {}, | |||||
| }; | |||||
| this.ws?.send(JSON.stringify(message)); | |||||
| } | |||||
| moveMouse(x: number, y: number) { | |||||
| this.send({ | |||||
| type: WsDataType.MOUSE_MOVE, | |||||
| data: { x, y }, | |||||
| }); | |||||
| } | |||||
| clickMouse() { | |||||
| this.send({ | |||||
| type: WsDataType.MOUSE_CLICK, | |||||
| }); | |||||
| } | |||||
| scrollUp() { | |||||
| this.send({ | |||||
| type: WsDataType.MOUSE_SCROLL_UP, | |||||
| }); | |||||
| } | |||||
| scrollDown() { | |||||
| this.send({ | |||||
| type: WsDataType.MOUSE_SCROLL_DOWN, | |||||
| }); | |||||
| } | |||||
| sendKeys(key_str: string) { | |||||
| this.send({ | |||||
| type: WsDataType.SEND_KEYS, | |||||
| data: { key_str }, | |||||
| }); | |||||
| } | |||||
| voiceRecord() { | |||||
| this.send({ | |||||
| type: WsDataType.VOICE_RECORD, | |||||
| }); | |||||
| } | |||||
| voiceStop() { | |||||
| this.send({ | |||||
| type: WsDataType.VOICE_STOP, | |||||
| }); | |||||
| } | |||||
| } | |||||
| /** | |||||
| * 手势处理器类 - 负责将手势转换为具体操作 | |||||
| * 主要职责: | |||||
| * 1. 接收识别到的手势类型 | |||||
| * 2. 根据手势执行相应动作 | |||||
| * 3. 处理防抖和连续手势确认 | |||||
| */ | |||||
| export class GestureHandler { | |||||
| private triggerAction: TriggerAction; | |||||
| private previousGesture: HandGesture | null = null; | |||||
| private previousGestureCount: number = 0; | |||||
| private minGestureCount: number = 5; | |||||
| // 鼠标移动参数 | |||||
| private screen_width: number = window.screen.width; | |||||
| private screen_height: number = window.screen.height; | |||||
| private smoothening = 7; // 平滑系数 | |||||
| private prev_loc_x: number = 0; | |||||
| private prev_loc_y: number = 0; | |||||
| private prev_three_fingers_y: number = 0; // 添加三根手指上一次的 Y 坐标 | |||||
| private prev_scroll2_y: number = 0; | |||||
| // 时间间隔参数 | |||||
| private lastClickTime: number = 0; | |||||
| private lastScrollTime: number = 0; | |||||
| private lastFullScreenTime: number = 0; | |||||
| private lastDeleteTime: number = 0; | |||||
| // 时间间隔常量（毫秒） | |||||
| private readonly CLICK_INTERVAL = 500; // 点击间隔 | |||||
| private readonly SCROLL_INTERVAL = 100; // 滚动间隔 | |||||
| private readonly FULL_SCREEN_INTERVAL = 1500; // 全屏切换间隔 | |||||
| // 语音识别参数 | |||||
| private voice_recording: boolean = false; | |||||
| private app_store: any; | |||||
| constructor() { | |||||
| this.triggerAction = new TriggerAction(); | |||||
| this.app_store = use_app_store(); | |||||
| } | |||||
| /** | |||||
| * 处理食指上举手势 - 鼠标移动 | |||||
| */ | |||||
| private handleIndexFingerUp(hand: HandInfo) { | |||||
| const indexTip = this.getFingerTip(hand, 1); // 食指指尖 | |||||
| if (!indexTip) return; | |||||
| try { | |||||
| // 将 hand 的坐标转换为视频坐标 | |||||
| const video_x = | |||||
| this.app_store.VIDEO_WIDTH - indexTip.x * this.app_store.VIDEO_WIDTH; | |||||
| const video_y = indexTip.y * this.app_store.VIDEO_HEIGHT; | |||||
| /** | |||||
| * 辅助方法：将值从一个范围映射到另一个范围 | |||||
| */ | |||||
| function mapRange( | |||||
| value: number, | |||||
| fromMin: number, | |||||
| fromMax: number, | |||||
| toMin: number, | |||||
| toMax: number | |||||
| ): number { | |||||
| return ( | |||||
| ((value - fromMin) * (toMax - toMin)) / (fromMax - fromMin) + toMin | |||||
| ); | |||||
| } | |||||
| // 将视频坐标映射到屏幕坐标 | |||||
| // 由于 x 轴方向相反，所以需要翻转 | |||||
| let screenX = mapRange( | |||||
| video_x, | |||||
| this.app_store.config.boundary_left, | |||||
| this.app_store.config.boundary_left + | |||||
| this.app_store.config.boundary_width, | |||||
| 0, | |||||
| this.screen_width | |||||
| ); | |||||
| let screenY = mapRange( | |||||
| video_y, | |||||
| this.app_store.config.boundary_top, | |||||
| this.app_store.config.boundary_top + | |||||
| this.app_store.config.boundary_height, | |||||
| 0, | |||||
| this.screen_height | |||||
| ); | |||||
| // 应用平滑处理 | |||||
| screenX = | |||||
| this.prev_loc_x + (screenX - this.prev_loc_x) / this.smoothening; | |||||
| screenY = | |||||
| this.prev_loc_y + (screenY - this.prev_loc_y) / this.smoothening; // 消除抖动 | |||||
| this.prev_loc_x = screenX; | |||||
| this.prev_loc_y = screenY; | |||||
| // 移动鼠标 | |||||
| this.app_store.sub_windows.x = screenX + 10; | |||||
| this.app_store.sub_windows.y = screenY; | |||||
| this.triggerAction.moveMouse(screenX, screenY); | |||||
| } catch (error) { | |||||
| console.error("处理鼠标移动失败:", error); | |||||
| } | |||||
| } | |||||
| /** | |||||
| * 处理食指和中指同时竖起手势 - 鼠标左键点击 | |||||
| */ | |||||
| private handleMouseClick() { | |||||
| const now = Date.now(); | |||||
| if (now - this.lastClickTime < this.CLICK_INTERVAL) { | |||||
| return; | |||||
| } | |||||
| this.lastClickTime = now; | |||||
| this.triggerAction.clickMouse(); | |||||
| } | |||||
| /** | |||||
| * 处理三根手指同时竖起手势 - 滚动屏幕 | |||||
| */ | |||||
| private handleScroll(hand: HandInfo) { | |||||
| const indexTip = this.getFingerTip(hand, 1); | |||||
| const middleTip = this.getFingerTip(hand, 2); | |||||
| const ringTip = this.getFingerTip(hand, 3); | |||||
| if (!indexTip || !middleTip || !ringTip) { | |||||
| this.prev_three_fingers_y = 0; | |||||
| return; | |||||
| } | |||||
| const now = Date.now(); | |||||
| if (now - this.lastScrollTime < this.SCROLL_INTERVAL) { | |||||
| return; | |||||
| } | |||||
| this.lastScrollTime = now; | |||||
| // 计算三根手指的平均 Y 坐标 | |||||
| const currentY = (indexTip.y + middleTip.y + ringTip.y) / 3; | |||||
| // 如果是第一次检测到手势，记录当前 Y 坐标 | |||||
| if (this.prev_three_fingers_y === 0) { | |||||
| this.prev_three_fingers_y = currentY; | |||||
| return; | |||||
| } | |||||
| // 计算 Y 坐标的变化 | |||||
| const deltaY = currentY - this.prev_three_fingers_y; | |||||
| // 如果变化超过阈值，则触发滚动 | |||||
| if (Math.abs(deltaY) > 0.008) { | |||||
| if (deltaY < 0) { | |||||
| // 手指向上移动，向上滚动 | |||||
| this.triggerAction.scrollUp(); | |||||
| } else { | |||||
| // 手指向下移动，向下滚动 | |||||
| this.triggerAction.scrollDown(); | |||||
| } | |||||
| // 更新上一次的 Y 坐标 | |||||
| this.prev_three_fingers_y = currentY; | |||||
| } | |||||
| } | |||||
| // 拇指和食指捏合，滚动屏幕 | |||||
| private handleScroll2(hand: HandInfo) { | |||||
| const indexTip = this.getFingerTip(hand, 1); | |||||
| const thumbTip = this.getFingerTip(hand, 0); | |||||
| if (!indexTip || !thumbTip) { | |||||
| this.prev_scroll2_y = 0; | |||||
| return; | |||||
| } | |||||
| const now = Date.now(); | |||||
| if (now - this.lastScrollTime < this.SCROLL_INTERVAL) { | |||||
| return; | |||||
| } | |||||
| this.lastScrollTime = now; | |||||
| // 计算食指和拇指的距离 | |||||
| const distance = Math.sqrt( | |||||
| (indexTip.x - thumbTip.x) ** 2 + (indexTip.y - thumbTip.y) ** 2 | |||||
| ); | |||||
| console.log(i18n.global.t("当前食指和拇指距离"), distance); | |||||
| // 如果距离大于阈值，说明没有捏合，重置上一次的 Y 坐标 | |||||
| if ( | |||||
| distance > | |||||
| this.app_store.config.scroll_gesture_2_thumb_and_index_threshold | |||||
| ) { | |||||
| this.prev_scroll2_y = 0; | |||||
| return; | |||||
| } | |||||
| // 如果是第一次检测到捏合，记录当前 Y 坐标 | |||||
| if (this.prev_scroll2_y === 0) { | |||||
| this.prev_scroll2_y = indexTip.y; | |||||
| return; | |||||
| } | |||||
| // 计算 Y | |||||
| const deltaY = indexTip.y - this.prev_scroll2_y; | |||||
| // 如果变化超过阈值，则触发滚动 | |||||
| if (Math.abs(deltaY) > 0.008) { | |||||
| if (deltaY < 0) { | |||||
| // 手指向上移动，向上滚动 | |||||
| this.triggerAction.scrollUp(); | |||||
| } else { | |||||
| // 手指向下移动，向下滚动 | |||||
| this.triggerAction.scrollDown(); | |||||
| } | |||||
| // 更新上一次的 Y 坐标 | |||||
| this.prev_scroll2_y = indexTip.y; | |||||
| } | |||||
| } | |||||
| /** | |||||
| * 处理四根手指同时竖起手势 - 发送快捷键 | |||||
| */ | |||||
| private handleFourFingers() { | |||||
| try { | |||||
| const key_str = this.app_store.config.four_fingers_up_send || "f"; | |||||
| const now = Date.now(); | |||||
| if (now - this.lastFullScreenTime < this.FULL_SCREEN_INTERVAL) { | |||||
| return; | |||||
| } | |||||
| this.lastFullScreenTime = now; | |||||
| this.triggerAction.sendKeys(key_str); | |||||
| } catch (error) { | |||||
| console.error("处理四指手势失败:", error); | |||||
| } | |||||
| } | |||||
| /** | |||||
| * 处理拇指和小指同时竖起手势 - 开始语音识别 | |||||
| */ | |||||
| async handleVoiceStart() { | |||||
| if (this.voice_recording) { | |||||
| return; | |||||
| } | |||||
| await this.app_store.sub_window_info("开始语音识别"); | |||||
| this.voice_recording = true; | |||||
| this.triggerAction.voiceRecord(); | |||||
| } | |||||
| /** | |||||
| * 处理拳头手势 - 停止语音识别 | |||||
| */ | |||||
| async handleVoiceStop() { | |||||
| if (!this.voice_recording) { | |||||
| return; | |||||
| } | |||||
| await this.app_store.sub_window_success("停止语音识别"); | |||||
| this.voice_recording = false; | |||||
| this.triggerAction.voiceStop(); | |||||
| } | |||||
| /** | |||||
| * 处理删除手势 | |||||
| */ | |||||
| private handleDelete() { | |||||
| const now = Date.now(); | |||||
| if (now - this.lastDeleteTime < 300) { | |||||
| return; | |||||
| } | |||||
| this.lastDeleteTime = now; | |||||
| this.triggerAction.sendKeys("backspace"); | |||||
| } | |||||
| /** | |||||
| * 处理停止手势 | |||||
| */ | |||||
| async handleStopGesture(): Promise<void> { | |||||
| const toogle_detect = () => { | |||||
| this.app_store.flag_detecting = !this.app_store.flag_detecting; | |||||
| }; | |||||
| if (this.previousGestureCount >= 60) { | |||||
| toogle_detect(); | |||||
| this.previousGestureCount = 0; | |||||
| // 暂停手势识别后，更新 sub-window 进度条 | |||||
| this.app_store.sub_windows.progress = 0; | |||||
| } else { | |||||
| this.previousGestureCount++; | |||||
| if (this.previousGestureCount >= 20) { | |||||
| // 更新 sub-window 进度条 | |||||
| this.app_store.sub_windows.progress = Math.floor( | |||||
| (this.previousGestureCount / 60) * 100 | |||||
| ); | |||||
| } | |||||
| } | |||||
| } | |||||
| /** | |||||
| * 获取手指尖点 | |||||
| */ | |||||
| private getFingerTip(hand: HandInfo, fingerIndex: number) { | |||||
| if (!hand) return null; | |||||
| const tipIndices = [4, 8, 12, 16, 20]; | |||||
| return hand.landmarks[tipIndices[fingerIndex]]; | |||||
| } | |||||
| /** | |||||
| * 处理手势 | |||||
| */ | |||||
| handleGesture(gesture: HandGesture, hand: HandInfo) { | |||||
| // 更新手势连续性计数 | |||||
| if (gesture === this.previousGesture) { | |||||
| this.previousGestureCount++; | |||||
| } else { | |||||
| this.previousGesture = gesture; | |||||
| this.previousGestureCount = 1; | |||||
| } | |||||
| // 首先处理停止手势 | |||||
| if (gesture === HandGesture.STOP_GESTURE) { | |||||
| if (hand.categoryName === "Open_Palm") { | |||||
| this.handleStopGesture(); | |||||
| } | |||||
| return; | |||||
| } | |||||
| // 如果手势识别已暂停，则不处理 | |||||
| if (!this.app_store.flag_detecting) { | |||||
| return; | |||||
| } | |||||
| // 只要切换手势就停止语音识别 | |||||
| if (gesture !== HandGesture.VOICE_GESTURE_START && this.voice_recording) { | |||||
| this.handleVoiceStop(); | |||||
| return; | |||||
| } | |||||
| // 鼠标移动手势直接执行，不需要连续确认 | |||||
| if (gesture === HandGesture.ONLY_INDEX_UP) { | |||||
| this.handleIndexFingerUp(hand); | |||||
| return; | |||||
| } | |||||
| // 其他手势需要连续确认才执行 | |||||
| if (this.previousGestureCount >= this.minGestureCount) { | |||||
| switch (gesture) { | |||||
| case HandGesture.ROCK_GESTURE: | |||||
| case HandGesture.INDEX_AND_THUMB_UP: | |||||
| this.handleMouseClick(); | |||||
| break; | |||||
| case HandGesture.SCROLL_GESTURE_2: | |||||
| this.handleScroll2(hand); | |||||
| break; | |||||
| // case HandGesture.THREE_FINGERS_UP: | |||||
| // this.handleScroll(hand); | |||||
| // break; | |||||
| case HandGesture.FOUR_FINGERS_UP: | |||||
| this.handleFourFingers(); | |||||
| break; | |||||
| case HandGesture.VOICE_GESTURE_START: | |||||
| this.handleVoiceStart(); | |||||
| break; | |||||
| case HandGesture.DELETE_GESTURE: | |||||
| this.handleDelete(); | |||||
| break; | |||||
| } | |||||
| } | |||||
| } | |||||
| } | |||||
+ 49
- 0
src/locales/en.ts
View File
| @ -0,0 +1,49 @@ | |||||
| export default { | |||||
| "手势识别控制": "Gesture Recognition Control", | |||||
| "运行中": "Running", | |||||
| "已停止": "Stopped", | |||||
| "开机自启动": "Auto Start", | |||||
| "显示识别窗口": "Show Recognition Window", | |||||
| "摄像头选择": "Camera Selection", | |||||
| "手势操作指南": "Gesture Guide", | |||||
| "光标控制": "Cursor Control", | |||||
| "竖起食指滑动控制光标位置": "Slide with index finger to control cursor position", | |||||
| "单击操作": "Click Operation", | |||||
| "双指举起执行鼠标单击": "Raise two fingers to perform mouse click", | |||||
| "Rock手势执行鼠标单击": "Rock gesture to perform mouse click", | |||||
| "滚动控制": "Scroll Control", | |||||
| "三指上下滑动控制页面滚动": "Slide three fingers up/down to control page scrolling", | |||||
| "（okay手势）食指和拇指捏合滚动页面": "（okay gesture）Pinch with index and thumb to scroll page", | |||||
| "食指和拇指距离小于": "Index and thumb distance less than", | |||||
| "触发捏合": "Trigger pinch", | |||||
| "默认值0.02": "Default value 0.02", | |||||
| "可以通过右键->检查->控制台->捏合手势->查看当前距离": "Can check current distance by right-click -> inspect -> console -> pinch gesture", | |||||
| "全屏控制": "Full Screen Control", | |||||
| "四指并拢发送按键": "Four fingers together to send key", | |||||
| "点击设置快捷键": "Click to set shortcut", | |||||
| "请按下按键...": "Please press keys...", | |||||
| "点击设置": "Click to set", | |||||
| "退格": "Backspace", | |||||
| "发送退格键": "Send backspace key", | |||||
| "开始语音识别": "Start Voice Recognition", | |||||
| "六指手势开始语音识别": "Six fingers gesture to start voice recognition", | |||||
| "结束语音识别": "End Voice Recognition", | |||||
| "拳头手势结束语音识别": "Fist gesture to end voice recognition", | |||||
| "暂停/继续": "Pause/Resume", | |||||
| "单手张开1.5秒 暂停/继续 手势识别": "Open one hand for 1.5 seconds to pause/resume gesture recognition", | |||||
| "识别框x": "Recognition box x", | |||||
| "识别框y": "Recognition box y", | |||||
| "识别框宽": "Recognition box width", | |||||
| "识别框高": "Recognition box height", | |||||
| // 通知 | |||||
| "Lazyeat": "Lazyeat", | |||||
| "提示": "Tip", | |||||
| "停止语音识别": "Stop Voice Recognition", | |||||
| "手势识别": "Gesture Recognition", | |||||
| "继续手势识别": "Continue Gesture Recognition", | |||||
| "暂停手势识别": "Pause Gesture Recognition", | |||||
| }; | |||||
+ 18
- 0
src/locales/i18n.ts
View File
| @ -0,0 +1,18 @@ | |||||
| import { createI18n } from "vue-i18n"; | |||||
| // 导入语言包 | |||||
| import en from "./en"; | |||||
| import zh from "./zh"; | |||||
| // 创建 i18n 实例 | |||||
| const i18n = createI18n({ | |||||
| locale: navigator.language.split("-")[0], // 使用系统语言作为默认语言 | |||||
| // locale: "en", // 使用系统语言作为默认语言 | |||||
| fallbackLocale: "zh", // 回退语言 | |||||
| messages: { | |||||
| en, | |||||
| zh, | |||||
| }, | |||||
| }); | |||||
| export default i18n; | |||||
+ 47
- 0
src/locales/zh.ts
View File
| @ -0,0 +1,47 @@ | |||||
| export default { | |||||
| "手势识别控制": "手势识别控制", | |||||
| "运行中": "运行中", | |||||
| "已停止": "已停止", | |||||
| "显示识别窗口": "显示识别窗口", | |||||
| "摄像头选择": "摄像头选择", | |||||
| "手势操作指南": "手势操作指南", | |||||
| "开机自启动": "开机自启动", | |||||
| "光标控制": "光标控制", | |||||
| "竖起食指滑动控制光标位置": "竖起食指滑动控制光标位置", | |||||
| "单击操作": "单击操作", | |||||
| "双指举起执行鼠标单击": "双指举起执行鼠标单击", | |||||
| "Rock手势执行鼠标单击": "Rock手势执行鼠标单击", | |||||
| "滚动控制": "滚动控制", | |||||
| "三指上下滑动控制页面滚动": "三指上下滑动控制页面滚动", | |||||
| "（okay手势）食指和拇指捏合滚动页面": "（okay手势）食指和拇指捏合滚动页面", | |||||
| "食指和拇指距离小于": "食指和拇指距离小于", | |||||
| "触发捏合": "触发捏合", | |||||
| "全屏控制": "全屏控制", | |||||
| "四指并拢发送按键": "四指并拢发送按键", | |||||
| "点击设置快捷键": "点击设置快捷键", | |||||
| "请按下按键...": "请按下按键...", | |||||
| "点击设置": "点击设置", | |||||
| "退格": "退格", | |||||
| "发送退格键": "发送退格键", | |||||
| "开始语音识别": "开始语音识别", | |||||
| "六指手势开始语音识别": "六指手势开始语音识别", | |||||
| "结束语音识别": "结束语音识别", | |||||
| "拳头手势结束语音识别": "拳头手势结束语音识别", | |||||
| "暂停/继续": "暂停/继续", | |||||
| "单手张开1.5秒 暂停/继续 手势识别": "单手张开1.5秒 暂停/继续 手势识别", | |||||
| "识别框x": "识别框x", | |||||
| "识别框y": "识别框y", | |||||
| "识别框宽": "识别框宽", | |||||
| "识别框高": "识别框高", | |||||
| "默认值0.02": "默认值0.02", | |||||
| "可以通过右键->检查->控制台->捏合手势->查看当前距离": "可以通过右键->检查->控制台->捏合手势->查看当前距离", | |||||
| // 通知 | |||||
| "Lazyeat": "Lazyeat", | |||||
| "提示": "提示", | |||||
| "停止语音识别": "停止语音识别", | |||||
| "手势识别": "手势识别", | |||||
| "继续手势识别": "继续手势识别", | |||||
| "暂停手势识别": "暂停手势识别", | |||||
| }; | |||||
+ 85
- 0
src/main.ts
View File
| @ -0,0 +1,85 @@ | |||||
| import i18n from "@/locales/i18n"; | |||||
| import router from "@/router"; | |||||
| import { createPinia } from "pinia"; | |||||
| import { PiniaSharedState } from "pinia-shared-state"; | |||||
| import { createApp } from "vue"; | |||||
| import App from "@/App.vue"; | |||||
| import { | |||||
| create, | |||||
| NAlert, | |||||
| NButton, | |||||
| NCard, | |||||
| NCheckbox, | |||||
| NDivider, | |||||
| NForm, | |||||
| NFormItem, | |||||
| NIcon, | |||||
| NImage, | |||||
| NInput, | |||||
| NLayout, | |||||
| NLayoutContent, | |||||
| NLayoutFooter, | |||||
| NLayoutHeader, | |||||
| NMenu, | |||||
| NMessageProvider, | |||||
| NSelect, | |||||
| NSpace, | |||||
| NInputNumber, | |||||
| NSpin, | |||||
| NSwitch, | |||||
| NProgress, | |||||
| NTag, | |||||
| } from "naive-ui"; | |||||
| // 引入element-plus | |||||
| import "element-plus/dist/index.css"; | |||||
| const naive = create({ | |||||
| components: [ | |||||
| NButton, | |||||
| NLayout, | |||||
| NLayoutHeader, | |||||
| NLayoutContent, | |||||
| NLayoutFooter, | |||||
| NMenu, | |||||
| NSpace, | |||||
| NImage, | |||||
| NDivider, | |||||
| NSwitch, | |||||
| NSelect, | |||||
| NSpin, | |||||
| NIcon, | |||||
| NInput, | |||||
| NInputNumber, | |||||
| NForm, | |||||
| NFormItem, | |||||
| NCheckbox, | |||||
| NCard, | |||||
| NMessageProvider, | |||||
| NAlert, | |||||
| NTag, | |||||
| NProgress, | |||||
| ], | |||||
| }); | |||||
| const app = createApp(App); | |||||
| const pinia = createPinia(); | |||||
| // Pass the plugin to your application's pinia plugin | |||||
| pinia.use( | |||||
| PiniaSharedState({ | |||||
| // Enables the plugin for all stores. Defaults to true. | |||||
| enable: true, | |||||
| // If set to true this tab tries to immediately recover the shared state from another tab. Defaults to true. | |||||
| initialize: false, | |||||
| // Enforce a type. One of native, idb, localstorage or node. Defaults to native. | |||||
| type: "localstorage", | |||||
| }) | |||||
| ); | |||||
| app.use(naive); | |||||
| app.use(pinia); | |||||
| app.use(i18n); | |||||
| app.use(router); | |||||
| app.mount("#app"); | |||||
+ 30
- 0
src/py_api.ts
View File
| @ -0,0 +1,30 @@ | |||||
| const port = 62334; | |||||
| const base_url = `http://localhost:${port}`; | |||||
| class PyApi { | |||||
| async ready(): Promise<boolean> { | |||||
| try { | |||||
| await fetch(`${base_url}/`, { | |||||
| signal: AbortSignal.timeout(1000), | |||||
| }); | |||||
| return true; | |||||
| } catch (error) { | |||||
| return false; | |||||
| } | |||||
| } | |||||
| async shutdown() { | |||||
| try { | |||||
| await fetch(`${base_url}/shutdown`, { | |||||
| method: "GET", | |||||
| signal: AbortSignal.timeout(500), | |||||
| }); | |||||
| } catch (error) { | |||||
| console.error("关闭服务失败:", error); | |||||
| } | |||||
| } | |||||
| } | |||||
| const pyApi = new PyApi(); | |||||
| export default pyApi; | |||||
+ 38
- 0
src/router/index.ts
View File
| @ -0,0 +1,38 @@ | |||||
| import MainWindow from "@/view/mainWindow/MainWindow.vue"; | |||||
| import SubWindow from "@/view/subWindow/SubWindow.vue"; | |||||
| import Home from "@/view/mainWindow/Home.vue"; | |||||
| import Update from "@/view/mainWindow/Update.vue"; | |||||
| import Guide from "@/view/mainWindow/Guide.vue"; | |||||
| import { createRouter, createWebHistory } from "vue-router"; | |||||
| const routes = [ | |||||
| { | |||||
| path: "/", | |||||
| name: "mainWindow", | |||||
| component: MainWindow, | |||||
| children: [ | |||||
| { | |||||
| path: "", | |||||
| name: "home", | |||||
| component: Home, | |||||
| }, | |||||
| { | |||||
| path: "update", | |||||
| name: "update", | |||||
| component: Update, | |||||
| }, | |||||
| { | |||||
| path: "guide", | |||||
| name: "guide", | |||||
| component: Guide, | |||||
| } | |||||
| ] | |||||
| }, | |||||
| ]; | |||||
| const router = createRouter({ | |||||
| history: createWebHistory(import.meta.env.BASE_URL), | |||||
| routes, | |||||
| }); | |||||
| export default router; | |||||
+ 87
- 0
src/store/app.ts
View File
| @ -0,0 +1,87 @@ | |||||
| import { defineStore } from "pinia"; | |||||
| interface Camera { | |||||
| deviceId: string; | |||||
| label: string; | |||||
| kind: string; | |||||
| } | |||||
| enum NotiType { | |||||
| INFO = "info", | |||||
| SUCCESS = "success", | |||||
| WARNING = "warning", | |||||
| ERROR = "error", | |||||
| } | |||||
| export const use_app_store = defineStore("app-store", { | |||||
| state: () => ({ | |||||
| config: { | |||||
| auto_start: false, | |||||
| show_window: false, | |||||
| four_fingers_up_send: "f", | |||||
| selected_camera_id: "", | |||||
| // 识别框 | |||||
| boundary_left: 300, | |||||
| boundary_top: 150, | |||||
| boundary_width: 280, | |||||
| boundary_height: 200, | |||||
| // 手势识别 | |||||
| scroll_gesture_2_thumb_and_index_threshold: 0.02, // 食指和拇指距离阈值 | |||||
| }, | |||||
| sub_windows: { | |||||
| x: 0, | |||||
| y: 0, | |||||
| progress: 0, | |||||
| notification: "", | |||||
| noti_type: NotiType.INFO, | |||||
| }, | |||||
| mission_running: false, | |||||
| cameras: [] as Camera[], | |||||
| VIDEO_WIDTH: 640, | |||||
| VIDEO_HEIGHT: 480, | |||||
| flag_detecting: false, | |||||
| }), | |||||
| // PiniaSharedState 来共享不同 tauri 窗口之间的状态 | |||||
| share: { | |||||
| // Override global config for this store. | |||||
| enable: true, | |||||
| initialize: true, | |||||
| }, | |||||
| actions: { | |||||
| is_macos() { | |||||
| return navigator.userAgent.includes("Mac"); | |||||
| }, | |||||
| is_windows() { | |||||
| return navigator.userAgent.includes("Windows"); | |||||
| }, | |||||
| is_linux() { | |||||
| return navigator.userAgent.includes("Linux"); | |||||
| }, | |||||
| async sub_window_info(body: string) { | |||||
| this.sub_windows.notification = body; | |||||
| this.sub_windows.noti_type = NotiType.INFO; | |||||
| }, | |||||
| async sub_window_success(body: string) { | |||||
| this.sub_windows.notification = body; | |||||
| this.sub_windows.noti_type = NotiType.SUCCESS; | |||||
| }, | |||||
| async sub_window_warning(body: string) { | |||||
| this.sub_windows.notification = body; | |||||
| this.sub_windows.noti_type = NotiType.WARNING; | |||||
| }, | |||||
| async sub_window_error(body: string) { | |||||
| this.sub_windows.notification = body; | |||||
| this.sub_windows.noti_type = NotiType.ERROR; | |||||
| }, | |||||
| }, | |||||
| }); | |||||
| export default use_app_store; | |||||
+ 46
- 0
src/utils/subWindow.ts
View File
| @ -0,0 +1,46 @@ | |||||
| import { | |||||
| WebviewWindow, | |||||
| getAllWebviewWindows, | |||||
| } from "@tauri-apps/api/webviewWindow"; | |||||
| export const SUB_WINDOW_WIDTH = 130; | |||||
| export const SUB_WINDOW_HEIGHT = 130; | |||||
| export async function createSubWindow(url: string, title: string) { | |||||
| let message = ""; | |||||
| let success = true; | |||||
| try { | |||||
| const allWindows = await getAllWebviewWindows(); | |||||
| const windownsLen = allWindows.length; | |||||
| const label = `NewWindow_${windownsLen + 1}`; | |||||
| const openUrl = url || "index.html"; | |||||
| const newTitle = title || "新窗口"; | |||||
| const openTitle = `${newTitle}-${windownsLen + 1}`; | |||||
| const webview_window = new WebviewWindow(label, { | |||||
| url: openUrl, | |||||
| title: openTitle, | |||||
| parent: "main", | |||||
| zoomHotkeysEnabled: false, | |||||
| width: SUB_WINDOW_WIDTH, | |||||
| height: SUB_WINDOW_HEIGHT, | |||||
| minWidth: SUB_WINDOW_WIDTH, | |||||
| minHeight: SUB_WINDOW_HEIGHT, | |||||
| alwaysOnTop: true, | |||||
| decorations: false, // 隐藏窗口边框 | |||||
| visible: false, | |||||
| resizable: false, | |||||
| }); | |||||
| webview_window.once("tauri://created", async () => { | |||||
| message = "打开成功"; | |||||
| }); | |||||
| webview_window.once("tauri://error", function (e) { | |||||
| message = `打开${openTitle}报错: ${e}`; | |||||
| success = false; | |||||
| }); | |||||
| return { success: success, message: message, webview: webview_window }; | |||||
| } catch (error) { | |||||
| return { success: false, message: error }; | |||||
| } | |||||
| } | |||||
BIN
src/view/.DS_Store
View File
+ 348
- 0
src/view/mainWindow/Guide.vue
View File
| @ -0,0 +1,348 @@ | |||||
| <template> | |||||
| <div class="guide-container"> | |||||
| <div class="header-bar"> | |||||
| <div class="title-container"> | |||||
| <h1 class="view-title">{{ $t('手势操作指南') }}</h1> | |||||
| <div class="view-description">{{ $t('学习使用各种手势控制您的设备') }}</div> | |||||
| </div> | |||||
| </div> | |||||
| <div class="main-content"> | |||||
| <n-card class="guide-card"> | |||||
| <div class="gesture-grid"> | |||||
| <!-- 卡片1: 光标控制 --> | |||||
| <div class="card"> | |||||
| <div class="card-icon"> | |||||
| <div class="icon-circle"> | |||||
| <span class="icon-symbol">👆</span> | |||||
| </div> | |||||
| </div> | |||||
| <div class="card-content"> | |||||
| <h3 class="card-title">{{ $t('光标控制') }}</h3> | |||||
| <p class="card-description">{{ $t('竖起食指滑动控制光标位置') }}</p> | |||||
| </div> | |||||
| </div> | |||||
| <!-- 卡片2: 单击操作（双指） --> | |||||
| <div class="card"> | |||||
| <div class="card-icon"> | |||||
| <div class="icon-circle"> | |||||
| <span class="icon-symbol">✌️</span> | |||||
| </div> | |||||
| </div> | |||||
| <div class="card-content"> | |||||
| <h3 class="card-title">{{ $t('单击操作') }}</h3> | |||||
| <p class="card-description">{{ $t('双指举起执行鼠标单击') }}</p> | |||||
| </div> | |||||
| </div> | |||||
| <!-- 卡片3: 单击操作（Rock手势） --> | |||||
| <div class="card"> | |||||
| <div class="card-icon"> | |||||
| <div class="icon-circle"> | |||||
| <span class="icon-symbol">🤘</span> | |||||
| </div> | |||||
| </div> | |||||
| <div class="card-content"> | |||||
| <h3 class="card-title">{{ $t('单击操作') }}</h3> | |||||
| <p class="card-description">{{ $t('Rock手势执行鼠标单击') }}</p> | |||||
| <div class="card-extra"> | |||||
| </div> | |||||
| </div> | |||||
| </div> | |||||
| <!-- 卡片4: 滚动控制 --> | |||||
| <div class="card"> | |||||
| <div class="card-icon"> | |||||
| <div class="icon-circle"> | |||||
| <span class="icon-symbol">👌</span> | |||||
| </div> | |||||
| </div> | |||||
| <div class="card-content"> | |||||
| <h3 class="card-title">{{ $t('滚动控制') }}</h3> | |||||
| <p class="card-description">{{ $t('(okay手势)食指和拇指捏合滚动页面') }}</p> | |||||
| <div class="card-extra"> | |||||
| <div class="setting-control"> | |||||
| <span>{{ $t("食指和拇指距离小于") }}</span> | |||||
| <n-input-number | |||||
| v-model:value="app_store.config.scroll_gesture_2_thumb_and_index_threshold" | |||||
| :min="0" | |||||
| :step="0.01" | |||||
| size="small" | |||||
| /> | |||||
| <span>{{ $t("触发捏合") }}</span> | |||||
| </div> | |||||
| <div class="hint-tags"> | |||||
| <n-tag size="small" type="info">{{ $t("默认值0.02") }}</n-tag> | |||||
| <n-tag size="small" type="info">{{ $t("右键->检查->控制台->查看当前距离") }}</n-tag> | |||||
| </div> | |||||
| </div> | |||||
| </div> | |||||
| </div> | |||||
| <!-- 卡片5: 全屏控制 --> | |||||
| <div class="card"> | |||||
| <div class="card-icon"> | |||||
| <div class="icon-circle"> | |||||
| <span class="icon-symbol">🖐️</span> | |||||
| </div> | |||||
| </div> | |||||
| <div class="card-content"> | |||||
| <h3 class="card-title">{{ $t('全屏控制') }}</h3> | |||||
| <p class="card-description">{{ $t('四指并拢发送按键') }}</p> | |||||
| <div class="card-extra"> | |||||
| <div class="keyboard-input"> | |||||
| <n-input | |||||
| :value="app_store.config.four_fingers_up_send || 'f'" | |||||
| readonly | |||||
| :placeholder="$t('点击设置快捷键')" | |||||
| @click="listenForKey" | |||||
| :status="isListening ? 'warning' : undefined" | |||||
| size="small" | |||||
| > | |||||
| <template #suffix> | |||||
| <span>{{ isListening ? $t("请按下按键...") : $t("点击设置") }}</span> | |||||
| </template> | |||||
| </n-input> | |||||
| </div> | |||||
| </div> | |||||
| </div> | |||||
| </div> | |||||
| <!-- 卡片6: 退格 --> | |||||
| <div class="card"> | |||||
| <div class="card-icon"> | |||||
| <div class="icon-circle"> | |||||
| <span class="icon-symbol">🔙</span> | |||||
| </div> | |||||
| </div> | |||||
| <div class="card-content"> | |||||
| <h3 class="card-title">{{ $t('退格') }}</h3> | |||||
| <p class="card-description">{{ $t('发送退格键') }}</p> | |||||
| </div> | |||||
| </div> | |||||
| <!-- 卡片7: 开始语音识别 --> | |||||
| <div class="card"> | |||||
| <div class="card-icon"> | |||||
| <div class="icon-circle"> | |||||
| <span class="icon-symbol">🎤</span> | |||||
| </div> | |||||
| </div> | |||||
| <div class="card-content"> | |||||
| <h3 class="card-title">{{ $t('开始语音识别') }}</h3> | |||||
| <p class="card-description">{{ $t('六指手势开始语音识别') }}</p> | |||||
| </div> | |||||
| </div> | |||||
| <!-- 卡片8: 结束语音识别 --> | |||||
| <div class="card"> | |||||
| <div class="card-icon"> | |||||
| <div class="icon-circle"> | |||||
| <span class="icon-symbol">👊</span> | |||||
| </div> | |||||
| </div> | |||||
| <div class="card-content"> | |||||
| <h3 class="card-title">{{ $t('结束语音识别') }}</h3> | |||||
| <p class="card-description">{{ $t('拳头手势结束语音识别') }}</p> | |||||
| </div> | |||||
| </div> | |||||
| <!-- 卡片9: 暂停/继续 --> | |||||
| <div class="card"> | |||||
| <div class="card-icon"> | |||||
| <div class="icon-circle"> | |||||
| <span class="icon-symbol">✋</span> | |||||
| </div> | |||||
| </div> | |||||
| <div class="card-content"> | |||||
| <h3 class="card-title">{{ $t('暂停/继续') }}</h3> | |||||
| <p class="card-description">{{ $t('单手张开1.5秒 暂停/继续 手势识别') }}</p> | |||||
| </div> | |||||
| </div> | |||||
| </div> | |||||
| </n-card> | |||||
| </div> | |||||
| </div> | |||||
| </template> | |||||
| <script setup lang="ts"> | |||||
| import { use_app_store } from "@/store/app"; | |||||
| import { ref } from "vue"; | |||||
| const app_store = use_app_store(); | |||||
| const isListening = ref(false); | |||||
| const listenForKey = () => { | |||||
| isListening.value = true; | |||||
| // ...（原有监听键盘的代码） | |||||
| }; | |||||
| </script> | |||||
| <style scoped lang="scss"> | |||||
| .guide-container { | |||||
| padding: 16px; | |||||
| box-sizing: border-box; | |||||
| } | |||||
| .header-bar { | |||||
| margin-bottom: 24px; | |||||
| } | |||||
| .title-container { | |||||
| padding: 0 16px; | |||||
| } | |||||
| .view-title { | |||||
| font-size: 24px; | |||||
| font-weight: 700; | |||||
| margin: 0; | |||||
| color: #333; | |||||
| } | |||||
| .view-description { | |||||
| font-size: 14px; | |||||
| color: #666; | |||||
| margin-top: 8px; | |||||
| } | |||||
| .main-content { | |||||
| display: flex; | |||||
| flex-direction: column; | |||||
| } | |||||
| .guide-card { | |||||
| background-color: #fff; | |||||
| border-radius: 8px; | |||||
| box-shadow: 0 2px 8px rgba(0, 0, 0, 0.1); | |||||
| overflow: hidden; | |||||
| } | |||||
| .gesture-grid { | |||||
| display: grid; | |||||
| grid-template-columns: repeat(3, 1fr); | |||||
| gap: 16px; | |||||
| padding: 16px; | |||||
| } | |||||
| .card { | |||||
| display: flex; | |||||
| flex-direction: column; | |||||
| background: #fff; | |||||
| border: 1px solid #f0f0f0; | |||||
| border-radius: 8px; | |||||
| overflow: hidden; | |||||
| transition: all 0.3s ease; | |||||
| &:hover { | |||||
| box-shadow: 0 4px 12px rgba(0, 0, 0, 0.15); | |||||
| transform: translateY(-2px); | |||||
| } | |||||
| } | |||||
| .card-icon { | |||||
| padding: 16px; | |||||
| display: flex; | |||||
| justify-content: center; | |||||
| } | |||||
| .icon-circle { | |||||
| width: 64px; | |||||
| height: 64px; | |||||
| border-radius: 50%; | |||||
| background: #e6f7ff; | |||||
| display: flex; | |||||
| align-items: center; | |||||
| justify-content: center; | |||||
| } | |||||
| .icon-symbol { | |||||
| font-size: 32px; | |||||
| color: #1890ff; | |||||
| } | |||||
| .card-content { | |||||
| padding: 0 16px 16px; | |||||
| } | |||||
| .card-title { | |||||
| font-size: 16px; | |||||
| font-weight: 600; | |||||
| color: #333; | |||||
| margin: 8px 0 4px; | |||||
| } | |||||
| .card-description { | |||||
| font-size: 14px; | |||||
| color: #666; | |||||
| margin: 4px 0 8px; | |||||
| line-height: 1.4; | |||||
| } | |||||
| .card-extra { | |||||
| border-top: 1px solid #f0f0f0; | |||||
| padding-top: 12px; | |||||
| margin-top: 12px; | |||||
| a { | |||||
| color: #1890ff; | |||||
| text-decoration: none; | |||||
| margin-right: 12px; | |||||
| &:hover { | |||||
| text-decoration: underline; | |||||
| } | |||||
| } | |||||
| } | |||||
| .setting-control { | |||||
| display: flex; | |||||
| align-items: center; | |||||
| flex-wrap: wrap; | |||||
| gap: 8px; | |||||
| margin-bottom: 8px; | |||||
| .n-input-number { | |||||
| width: 100px; | |||||
| } | |||||
| } | |||||
| .hint-tags { | |||||
| display: flex; | |||||
| flex-wrap: wrap; | |||||
| gap: 8px; | |||||
| margin-top: 8px; | |||||
| .n-tag { | |||||
| padding: 4px 8px; | |||||
| background: #f0f9ff; | |||||
| border-color: #91d5ff; | |||||
| color: #1890ff; | |||||
| } | |||||
| } | |||||
| .keyboard-input { | |||||
| width: 100%; | |||||
| .n-input { | |||||
| width: 100%; | |||||
| } | |||||
| } | |||||
| @media (max-width: 992px) { | |||||
| .gesture-grid { | |||||
| grid-template-columns: repeat(2, 1fr); | |||||
| } | |||||
| } | |||||
| @media (max-width: 576px) { | |||||
| .gesture-grid { | |||||
| grid-template-columns: 1fr; | |||||
| } | |||||
| .setting-control { | |||||
| flex-direction: column; | |||||
| align-items: flex-start; | |||||
| } | |||||
| } | |||||
| </style> | |||||
+ 367
- 0
src/view/mainWindow/Home.vue
View File
| @ -0,0 +1,367 @@ | |||||
| <template> | |||||
| <div class="home-container"> | |||||
| <!-- 顶部统计数据卡片 --> | |||||
| <div class="stats-row"> | |||||
| <div class="stat-card"> | |||||
| <div class="stat-icon">👋</div> | |||||
| <div class="stat-data"> | |||||
| <div class="value">92%</div> | |||||
| <div class="label">识别准确率</div> | |||||
| </div> | |||||
| </div> | |||||
| <div class="stat-card"> | |||||
| <div class="stat-icon">⚡</div> | |||||
| <div class="stat-data"> | |||||
| <div class="value">38ms</div> | |||||
| <div class="label">平均响应时间</div> | |||||
| </div> | |||||
| </div> | |||||
| <div class="stat-card"> | |||||
| <div class="stat-icon">🔍</div> | |||||
| <div class="stat-data"> | |||||
| <div class="value">312</div> | |||||
| <div class="label">今日识别次数</div> | |||||
| </div> | |||||
| </div> | |||||
| </div> | |||||
| <!-- 主内容区域 --> | |||||
| <div class="main-content"> | |||||
| <!-- 左侧摄像头预览区域 --> | |||||
| <div class="preview-card"> | |||||
| <div class="preview-header"> | |||||
| <h3>手势识别预览</h3> | |||||
| <!-- 运行开关移到预览区域顶部 --> | |||||
| <n-switch | |||||
| v-model:value="app_store.mission_running" | |||||
| size="large" | |||||
| :rail-style="railStyle" | |||||
| > | |||||
| <template #checked>{{ $t("运行中") }}</template> | |||||
| <template #unchecked>{{ $t("已停止") }}</template> | |||||
| </n-switch> | |||||
| </div> | |||||
| <!-- 摄像头预览组件 --> | |||||
| <VideoDetector /> | |||||
| </div> | |||||
| <!-- 右侧控制面板 --> | |||||
| <div class="control-panel"> | |||||
| <!-- 系统设置 --> | |||||
| <div class="settings-section"> | |||||
| <h3 class="section-title">手势识别控制</h3> | |||||
| <div class="setting-item"> | |||||
| <div class="setting-label"> | |||||
| <n-icon size="20"> | |||||
| <Browser /> | |||||
| </n-icon> | |||||
| <span>{{ $t("显示识别窗口") }}</span> | |||||
| </div> | |||||
| <n-switch v-model:value="app_store.config.show_window" /> | |||||
| </div> | |||||
| <div class="setting-item"> | |||||
| <div class="setting-label"> | |||||
| <AutoStart /> | |||||
| </div> | |||||
| </div> | |||||
| <div class="setting-item"> | |||||
| <div class="setting-label"> | |||||
| <n-icon size="20"> | |||||
| <Camera /> | |||||
| </n-icon> | |||||
| <span>{{ $t("摄像头选择") }}</span> | |||||
| </div> | |||||
| <n-select | |||||
| v-model:value="app_store.config.selected_camera_id" | |||||
| :options="camera_options" | |||||
| :disabled="app_store.mission_running" | |||||
| style="width: 100%" | |||||
| /> | |||||
| </div> | |||||
| </div> | |||||
| <!-- 手势库展示 --> | |||||
| <div class="gesture-gallery"> | |||||
| <div class="gallery-header"> | |||||
| <div>手势库</div> | |||||
| <button class="add-btn">+ 添加手势</button> | |||||
| </div> | |||||
| <div class="gesture-grid"> | |||||
| <div class="gesture-item" v-for="i in 6" :key="i"> | |||||
| <div class="gesture-icon">👌</div> | |||||
| <div class="gesture-name">OK手势</div> | |||||
| <div class="gesture-action">暂停媒体</div> | |||||
| </div> | |||||
| </div> | |||||
| </div> | |||||
| </div> | |||||
| </div> | |||||
| </div> | |||||
| </template> | |||||
| <script setup lang="ts"> | |||||
| import AutoStart from "@/components/AutoStart.vue"; | |||||
| import VideoDetector from "@/hand_landmark/VideoDetector.vue"; | |||||
| import { use_app_store } from "@/store/app"; | |||||
| import { Browser, Camera } from "@icon-park/vue-next"; | |||||
| import { computed, onMounted } from "vue"; | |||||
| const is_dev = computed(() => import.meta.env.DEV); | |||||
| const app_store = use_app_store(); | |||||
| // 计算属性：摄像头选项 | |||||
| const camera_options = computed(() => { | |||||
| return app_store.cameras.map((camera) => ({ | |||||
| label: camera.label || `摄像头 ${camera.deviceId.slice(0, 4)}`, | |||||
| value: camera.deviceId, | |||||
| })); | |||||
| }); | |||||
| const getCameras = async () => { | |||||
| try { | |||||
| // 申请获取摄像头权限 | |||||
| const stream = await navigator.mediaDevices.getUserMedia({ | |||||
| video: true, | |||||
| }); | |||||
| stream.getTracks().forEach((track) => track.stop()); | |||||
| const devices = await navigator.mediaDevices.enumerateDevices(); | |||||
| app_store.cameras = devices.filter( | |||||
| (device) => device.kind === "videoinput" | |||||
| ); | |||||
| } catch (error) { | |||||
| console.error("获取摄像头列表失败:", error); | |||||
| } | |||||
| }; | |||||
| onMounted(async () => { | |||||
| await getCameras(); | |||||
| }); | |||||
| </script> | |||||
| <style scoped lang="scss"> | |||||
| .home-container { | |||||
| display: flex; | |||||
| flex-direction: column; | |||||
| gap: 24px; | |||||
| height: 100%; | |||||
| } | |||||
| .stats-row { | |||||
| display: grid; | |||||
| grid-template-columns: repeat(3, 1fr); | |||||
| gap: 16px; | |||||
| } | |||||
| .stat-card { | |||||
| background: var(--card-bg); | |||||
| backdrop-filter: blur(10px); | |||||
| border-radius: 16px; | |||||
| padding: 20px; | |||||
| display: flex; | |||||
| align-items: center; | |||||
| gap: 16px; | |||||
| border: 1px solid var(--border-color); | |||||
| transition: transform 0.2s, box-shadow 0.2s; | |||||
| &:hover { | |||||
| transform: translateY(-5px); | |||||
| box-shadow: 0 10px 20px var(--shadow-color); | |||||
| } | |||||
| } | |||||
| .stat-icon { | |||||
| width: 56px; | |||||
| height: 56px; | |||||
| background: rgba(14, 165, 233, 0.1); | |||||
| border-radius: 50%; | |||||
| display: flex; | |||||
| align-items: center; | |||||
| justify-content: center; | |||||
| font-size: 24px; | |||||
| color: var(--accent-primary); | |||||
| } | |||||
| .stat-data { | |||||
| flex: 1; | |||||
| } | |||||
| .value { | |||||
| font-size: 24px; | |||||
| font-weight: 700; | |||||
| color: var(--text-primary); | |||||
| } | |||||
| .label { | |||||
| font-size: 14px; | |||||
| color: var(--text-secondary); | |||||
| margin-top: 4px; | |||||
| } | |||||
| .main-content { | |||||
| display: grid; | |||||
| grid-template-columns: 1.5fr 1fr; | |||||
| gap: 24px; | |||||
| height: calc(100% - 120px); | |||||
| } | |||||
| .preview-card { | |||||
| background: var(--card-bg); | |||||
| backdrop-filter: blur(10px); | |||||
| border-radius: 16px; | |||||
| padding: 24px; | |||||
| border: 1px solid var(--border-color); | |||||
| display: flex; | |||||
| flex-direction: column; | |||||
| gap: 16px; | |||||
| } | |||||
| .preview-header { | |||||
| display: flex; | |||||
| justify-content: space-between; | |||||
| align-items: center; | |||||
| h3 { | |||||
| font-size: 18px; | |||||
| font-weight: 600; | |||||
| color: var(--text-primary); | |||||
| margin: 0; | |||||
| } | |||||
| } | |||||
| .control-panel { | |||||
| display: flex; | |||||
| flex-direction: column; | |||||
| gap: 24px; | |||||
| } | |||||
| .settings-section { | |||||
| background: var(--card-bg); | |||||
| backdrop-filter: blur(10px); | |||||
| border-radius: 16px; | |||||
| padding: 24px; | |||||
| border: 1px solid var(--border-color); | |||||
| } | |||||
| .section-title { | |||||
| font-size: 18px; | |||||
| font-weight: 600; | |||||
| color: var(--text-primary); | |||||
| margin-top: 0; | |||||
| margin-bottom: 20px; | |||||
| } | |||||
| .setting-item { | |||||
| display: flex; | |||||
| justify-content: space-between; | |||||
| align-items: center; | |||||
| padding: 12px 0; | |||||
| border-bottom: 1px solid var(--border-color); | |||||
| &:last-child { | |||||
| border-bottom: none; | |||||
| } | |||||
| } | |||||
| .setting-label { | |||||
| display: flex; | |||||
| align-items: center; | |||||
| gap: 8px; | |||||
| font-size: 14px; | |||||
| color: var(--text-primary); | |||||
| } | |||||
| .gesture-gallery { | |||||
| background: var(--card-bg); | |||||
| backdrop-filter: blur(10px); | |||||
| border-radius: 16px; | |||||
| padding: 24px; | |||||
| border: 1px solid var(--border-color); | |||||
| } | |||||
| .gallery-header { | |||||
| display: flex; | |||||
| justify-content: space-between; | |||||
| align-items: center; | |||||
| margin-bottom: 20px; | |||||
| div { | |||||
| font-size: 18px; | |||||
| font-weight: 600; | |||||
| color: var(--text-primary); | |||||
| } | |||||
| } | |||||
| .add-btn { | |||||
| background: rgba(14, 165, 233, 0.15); | |||||
| border: none; | |||||
| border-radius: 8px; | |||||
| padding: 8px 15px; | |||||
| color: var(--accent-primary); | |||||
| font-weight: 500; | |||||
| cursor: pointer; | |||||
| transition: all 0.2s; | |||||
| &:hover { | |||||
| background: rgba(14, 165, 233, 0.25); | |||||
| } | |||||
| } | |||||
| .gesture-grid { | |||||
| display: grid; | |||||
| grid-template-columns: repeat(2, 1fr); | |||||
| gap: 15px; | |||||
| margin-top: 10px; | |||||
| } | |||||
| .gesture-item { | |||||
| background: var(--bg-tertiary); | |||||
| border-radius: 12px; | |||||
| padding: 15px; | |||||
| display: flex; | |||||
| flex-direction: column; | |||||
| align-items: center; | |||||
| transition: all 0.2s; | |||||
| &:hover { | |||||
| transform: translateY(-3px); | |||||
| box-shadow: 0 4px 10px var(--shadow-color); | |||||
| } | |||||
| } | |||||
| .gesture-icon { | |||||
| font-size: 32px; | |||||
| margin-bottom: 10px; | |||||
| } | |||||
| .gesture-name { | |||||
| font-weight: 500; | |||||
| color: var(--text-primary); | |||||
| } | |||||
| .gesture-action { | |||||
| font-size: 13px; | |||||
| color: var(--text-secondary); | |||||
| text-align: center; | |||||
| margin-top: 5px; | |||||
| } | |||||
| .ad-container { | |||||
| height: 200px; | |||||
| background: var(--card-bg); | |||||
| backdrop-filter: blur(10px); | |||||
| border-radius: 16px; | |||||
| overflow: hidden; | |||||
| iframe { | |||||
| border: none; | |||||
| background-color: var(--bg-secondary); | |||||
| border-radius: 16px; | |||||
| } | |||||
| } | |||||
| </style> | |||||
+ 607
- 0
src/view/mainWindow/MainWindow.vue
View File
| @ -0,0 +1,607 @@ | |||||
| <script setup lang="ts"> | |||||
| import DevTool from "@/components/DevTool.vue"; | |||||
| import pyApi from "@/py_api"; | |||||
| import use_app_store from "@/store/app"; | |||||
| import { getVersion } from "@tauri-apps/api/app"; | |||||
| import { | |||||
| getCurrentWindow, | |||||
| LogicalPosition, | |||||
| LogicalSize, | |||||
| } from "@tauri-apps/api/window"; | |||||
| import { LazyStore } from "@tauri-apps/plugin-store"; | |||||
| import { onMounted, ref, watch, computed } from "vue"; | |||||
| import { useRoute, useRouter } from "vue-router"; | |||||
| const route = useRoute(); | |||||
| const router = useRouter(); | |||||
| const is_dev = import.meta.env.DEV; | |||||
| const appVersion = ref(""); | |||||
| const app_store = use_app_store(); | |||||
| const ready = ref(false); | |||||
| const darkTheme = ref(false); | |||||
| const randomIconPath = ref(""); | |||||
| // 新UI中需要的数据 | |||||
| const settings = ref({ | |||||
| autostart: true, | |||||
| showWindow: true | |||||
| }); | |||||
| // 当前视图计算 | |||||
| const currentView = computed(() => route.name || 'home'); | |||||
| // 视图标题计算 | |||||
| const viewTitle = computed(() => { | |||||
| switch (currentView.value) { | |||||
| case 'home': return '控制面板'; | |||||
| case 'gestures': return '手势管理'; | |||||
| case 'settings': return '设置中心'; | |||||
| case 'help': return '帮助文档'; | |||||
| default: return 'WaveControl'; | |||||
| } | |||||
| }); | |||||
| // 在组件挂载时生成随机图标 | |||||
| onMounted(() => { | |||||
| const randomIndex = Math.floor(Math.random() * 9) + 1; | |||||
| randomIconPath.value = `/wavecontrol${randomIndex}.jpg`; | |||||
| }); | |||||
| // 初始化逻辑 | |||||
| onMounted(async () => { | |||||
| ready.value = await pyApi.ready(); | |||||
| const timer = setInterval(async () => { | |||||
| ready.value = await pyApi.ready(); | |||||
| if (ready.value) { | |||||
| clearInterval(timer); | |||||
| } | |||||
| }, 5000); | |||||
| await getCurrentWindow().onCloseRequested(async () => { | |||||
| const factor = await getCurrentWindow().scaleFactor(); | |||||
| const position = (await getCurrentWindow().innerPosition()).toLogical(factor); | |||||
| const size = (await getCurrentWindow().innerSize()).toLogical(factor); | |||||
| await window_store_json.set("window_state", { | |||||
| x: position.x, | |||||
| y: position.y, | |||||
| width: size.width, | |||||
| height: size.height, | |||||
| }); | |||||
| if (!is_dev) { | |||||
| await pyApi.shutdown(); | |||||
| } | |||||
| }); | |||||
| }); | |||||
| // 窗口恢复上一次位置 | |||||
| const window_store_json = new LazyStore("window_state.json"); | |||||
| onMounted(async () => { | |||||
| appVersion.value = await getVersion(); | |||||
| const window_state = await window_store_json.get("window_state"); | |||||
| if (window_state) { | |||||
| let new_x = window_state.x; | |||||
| let new_y = window_state.y; | |||||
| const screen_width = window.screen.width; | |||||
| const screen_height = window.screen.height; | |||||
| // 如果窗口位置超出屏幕，则将窗口位置设置为100，100 | |||||
| if (new_x <= 0) new_x = 100; | |||||
| if (new_x >= screen_width) new_x = 100 | |||||
| if (new_y <= 0) new_y = 100; | |||||
| if (new_y >= screen_height) new_y = 100; | |||||
| getCurrentWindow().setPosition(new LogicalPosition(new_x, new_y)); | |||||
| getCurrentWindow().setSize( | |||||
| new LogicalSize(window_state.width, window_state.height) | |||||
| ); | |||||
| } | |||||
| }); | |||||
| // app_store 数据加载 | |||||
| const app_store_json = new LazyStore("settings.json"); | |||||
| onMounted(async () => { | |||||
| const config_data = await app_store_json.get("config"); | |||||
| console.log("config_data", config_data); | |||||
| if (config_data) { | |||||
| Object.assign(app_store.config, JSON.parse(JSON.stringify(config_data))); | |||||
| } | |||||
| }); | |||||
| watch( | |||||
| () => app_store.config, | |||||
| async (value) => { | |||||
| await app_store_json.set("config", value); | |||||
| app_store_json.save(); | |||||
| }, | |||||
| { deep: true } | |||||
| ); | |||||
| // 通知 | |||||
| import { | |||||
| isPermissionGranted, | |||||
| requestPermission, | |||||
| } from "@tauri-apps/plugin-notification"; | |||||
| onMounted(async () => { | |||||
| let permissionGranted = await isPermissionGranted(); | |||||
| if (!permissionGranted) { | |||||
| const permission = await requestPermission(); | |||||
| permissionGranted = permission === "granted"; | |||||
| } | |||||
| }); | |||||
| // 使用默认浏览器打开 iframe 中的 <a> 标签 | |||||
| import { openUrl } from "@tauri-apps/plugin-opener"; | |||||
| window.addEventListener("message", async function (e) { | |||||
| const url = e.data; | |||||
| if (url) { | |||||
| await openUrl(url); | |||||
| } | |||||
| }); | |||||
| // 创建子窗口 | |||||
| import { createSubWindow } from "@/utils/subWindow"; | |||||
| const subWindow = ref(null); | |||||
| onMounted(async () => { | |||||
| if (!subWindow.value) { | |||||
| subWindow.value = await createSubWindow("/sub-window", "subWindow"); | |||||
| } | |||||
| }); | |||||
| // 操作方法 | |||||
| const toggleTheme = () => { | |||||
| darkTheme.value = !darkTheme.value; | |||||
| }; | |||||
| const switchView = (viewName: string) => { | |||||
| router.push({ name: viewName }); | |||||
| }; | |||||
| const toggleSetting = (setting: string) => { | |||||
| settings.value[setting] = !settings.value[setting]; | |||||
| }; | |||||
| // 状态计算属性 | |||||
| const statusClass = computed(() => ({ | |||||
| 'status-active': ready.value, | |||||
| 'status-inactive': !ready.value | |||||
| })); | |||||
| const statusText = computed(() => { | |||||
| return ready.value ? '运行中' : '已停止'; | |||||
| }); | |||||
| </script> | |||||
| <template> | |||||
| <div class="app-container" :class="{'dark-theme': darkTheme}"> | |||||
| <!-- 加载状态 --> | |||||
| <div v-if="!ready" class="loading-overlay"> | |||||
| <div class="loader-container"> | |||||
| <div class="spinner"></div> | |||||
| <div class="loading-text">手势识别模块加载中...</div> | |||||
| </div> | |||||
| </div> | |||||
| <!-- 主界面 --> | |||||
| <div v-else class="main-layout"> | |||||
| <!-- 左侧导航 --> | |||||
| <div class="sidebar"> | |||||
| <div class="branding"> | |||||
| <!-- 使用随机图标路径 --> | |||||
| <img :src="randomIconPath" alt="WaveControl Logo" class="logo" /> | |||||
| <div class="app-info"> | |||||
| <div class="app-name">WaveControl</div> | |||||
| </div> | |||||
| </div> | |||||
| <div class="nav-items"> | |||||
| <div class="nav-item" :class="{active: currentView === 'home'}" @click="switchView('home')"> | |||||
| <i class="icon">🏠</i> | |||||
| <span>主控台</span> | |||||
| </div> | |||||
| <div class="nav-item" :class="{active: currentView === 'guide'}" @click="switchView('guide')"> | |||||
| <i class="icon">👋</i> | |||||
| <span>手势管理</span> | |||||
| </div> | |||||
| <div class="nav-item" :class="{active: currentView === 'settings'}" @click="switchView('settings')"> | |||||
| <i class="icon">⚙️</i> | |||||
| <span>设置</span> | |||||
| </div> | |||||
| <div class="nav-item" :class="{active: currentView === 'help'}" @click="switchView('help')"> | |||||
| <i class="icon">❓</i> | |||||
| <span>帮助</span> | |||||
| </div> | |||||
| </div> | |||||
| <div class="status-card"> | |||||
| <div class="status-indicator" :class="statusClass"> | |||||
| {{ statusText }} | |||||
| </div> | |||||
| <div class="fps-counter"> | |||||
| <span>FPS</span> | |||||
| <span class="value">60</span> | |||||
| </div> | |||||
| </div> | |||||
| <div class="contributors"> | |||||
| <div v-if="app_store.is_macos()"> | |||||
| <a | |||||
| class="contributor-link" | |||||
| href="https://gitee.com/wydhhh/software-engineering" | |||||
| target="_blank" | |||||
| > | |||||
| @backpack</a | |||||
| > | |||||
| </div> | |||||
| <div v-else class="contributor">@ninjakelly</div> | |||||
| </div> | |||||
| </div> | |||||
| <!-- 主内容区 --> | |||||
| <div class="content-area"> | |||||
| <div class="header-bar"> | |||||
| <div class="title-container"> | |||||
| <h1 class="view-title">{{ viewTitle }}</h1> | |||||
| <div class="view-description">隔空手势控制系统</div> | |||||
| </div> | |||||
| <div class="controls"> | |||||
| <button class="theme-toggle" @click="toggleTheme"> | |||||
| <i class="icon">{{ darkTheme ? '☀️' : '🌙' }}</i> | |||||
| </button> | |||||
| </div> | |||||
| </div> | |||||
| <div class="main-content"> | |||||
| <router-view /> | |||||
| </div> | |||||
| </div> | |||||
| </div> | |||||
| <DevTool v-if="is_dev" /> | |||||
| </div> | |||||
| </template> | |||||
| <style scoped lang="scss"> | |||||
| .app-container { | |||||
| font-family: 'Segoe UI', system-ui, sans-serif; | |||||
| height: 100vh; | |||||
| width: 100vw; | |||||
| overflow: hidden; | |||||
| transition: background-color 0.3s, color 0.3s; | |||||
| &.dark-theme { | |||||
| --bg-primary: #0f172a; | |||||
| --bg-secondary: #1e293b; | |||||
| --bg-tertiary: #334155; | |||||
| --text-primary: #f1f5f9; | |||||
| --text-secondary: #94a3b8; | |||||
| --accent-primary: #0ea5e9; | |||||
| --accent-secondary: #7dd3fc; | |||||
| --hover-bg: rgba(148, 163, 184, 0.12); | |||||
| --border-color: #334155; | |||||
| --card-bg: rgba(30, 41, 59, 0.7); | |||||
| --shadow-color: rgba(0, 0, 0, 0.4); | |||||
| --status-active: #0ea5e9; | |||||
| --status-inactive: #475569; | |||||
| } | |||||
| &:not(.dark-theme) { | |||||
| --bg-primary: #f8fafc; | |||||
| --bg-secondary: #f1f5f9; | |||||
| --bg-tertiary: #e2e8f0; | |||||
| --text-primary: #0f172a; | |||||
| --text-secondary: #475569; | |||||
| --accent-primary: #0284c7; | |||||
| --accent-secondary: #0ea5e9; | |||||
| --hover-bg: rgba(15, 23, 42, 0.05); | |||||
| --border-color: #cbd5e1; | |||||
| --card-bg: rgba(241, 245, 249, 0.7); | |||||
| --shadow-color: rgba(0, 0, 0, 0.08); | |||||
| --status-active: #0284c7; | |||||
| --status-inactive: #94a3b8; | |||||
| } | |||||
| background-color: var(--bg-primary); | |||||
| color: var(--text-primary); | |||||
| } | |||||
| .loading-overlay { | |||||
| position: absolute; | |||||
| top: 0; | |||||
| left: 0; | |||||
| width: 100%; | |||||
| height: 100%; | |||||
| display: flex; | |||||
| align-items: center; | |||||
| justify-content: center; | |||||
| background-color: var(--bg-primary); | |||||
| z-index: 100; | |||||
| } | |||||
| .loader-container { | |||||
| display: flex; | |||||
| flex-direction: column; | |||||
| align-items: center; | |||||
| gap: 20px; | |||||
| } | |||||
| .spinner { | |||||
| width: 50px; | |||||
| height: 50px; | |||||
| border: 4px solid rgba(14, 165, 233, 0.2); | |||||
| border-radius: 50%; | |||||
| border-top: 4px solid var(--accent-primary); | |||||
| animation: spin 1s linear infinite; | |||||
| } | |||||
| @keyframes spin { | |||||
| 0% { transform: rotate(0deg); } | |||||
| 100% { transform: rotate(360deg); } | |||||
| } | |||||
| .loading-text { | |||||
| font-size: 18px; | |||||
| font-weight: 500; | |||||
| color: var(--text-primary); | |||||
| } | |||||
| .main-layout { | |||||
| display: flex; | |||||
| height: 100vh; | |||||
| } | |||||
| /* 左侧导航样式 */ | |||||
| .sidebar { | |||||
| width: 260px; | |||||
| background-color: var(--bg-secondary); | |||||
| display: flex; | |||||
| flex-direction: column; | |||||
| padding: 25px 0; | |||||
| border-right: 1px solid var(--border-color); | |||||
| z-index: 10; | |||||
| } | |||||
| .branding { | |||||
| display: flex; | |||||
| align-items: center; | |||||
| padding: 0 20px 20px; | |||||
| } | |||||
| .logo { | |||||
| width: 42px; | |||||
| height: 42px; | |||||
| border-radius: 10px; | |||||
| margin-right: 12px; | |||||
| background: linear-gradient(135deg, var(--accent-primary), #7dd3fc); | |||||
| padding: 5px; | |||||
| } | |||||
| .app-info { | |||||
| display: flex; | |||||
| flex-direction: column; | |||||
| } | |||||
| .app-name { | |||||
| font-size: 18px; | |||||
| font-weight: 700; | |||||
| letter-spacing: 0.5px; | |||||
| color: var(--text-primary); | |||||
| } | |||||
| .app-version { | |||||
| font-size: 13px; | |||||
| color: var(--text-secondary); | |||||
| font-weight: 500; | |||||
| opacity: 0.8; | |||||
| } | |||||
| .nav-items { | |||||
| display: flex; | |||||
| flex-direction: column; | |||||
| gap: 8px; | |||||
| padding: 0 15px; | |||||
| flex: 1; | |||||
| } | |||||
| .nav-item { | |||||
| display: flex; | |||||
| align-items: center; | |||||
| padding: 12px 20px; | |||||
| border-radius: 10px; | |||||
| cursor: pointer; | |||||
| transition: all 0.2s; | |||||
| gap: 14px; | |||||
| font-size: 16px; | |||||
| color: var(--text-secondary); | |||||
| &:hover { | |||||
| background-color: var(--hover-bg); | |||||
| color: var(--text-primary); | |||||
| } | |||||
| &.active { | |||||
| background: rgba(14, 165, 233, 0.15); | |||||
| color: var(--accent-primary); | |||||
| font-weight: 500; | |||||
| .icon { | |||||
| color: var(--accent-primary); | |||||
| } | |||||
| } | |||||
| } | |||||
| .icon { | |||||
| font-size: 18px; | |||||
| } | |||||
| .status-card { | |||||
| background: var(--card-bg); | |||||
| border-radius: 12px; | |||||
| padding: 20px; | |||||
| margin: 20px; | |||||
| backdrop-filter: blur(10px); | |||||
| border: 1px solid var(--border-color); | |||||
| box-shadow: 0 4px 6px var(--shadow-color); | |||||
| } | |||||
| .status-indicator { | |||||
| display: inline-flex; | |||||
| align-items: center; | |||||
| font-size: 14px; | |||||
| font-weight: 500; | |||||
| padding: 6px 12px; | |||||
| border-radius: 20px; | |||||
| &::before { | |||||
| content: ''; | |||||
| display: inline-block; | |||||
| width: 8px; | |||||
| height: 8px; | |||||
| border-radius: 50%; | |||||
| margin-right: 8px; | |||||
| } | |||||
| &.status-active { | |||||
| background: rgba(14, 165, 233, 0.2); | |||||
| color: var(--accent-primary); | |||||
| &::before { | |||||
| background-color: var(--accent-primary); | |||||
| } | |||||
| } | |||||
| &.status-inactive { | |||||
| background: rgba(148, 163, 184, 0.2); | |||||
| color: var(--text-secondary); | |||||
| &::before { | |||||
| background-color: var(--text-secondary); | |||||
| } | |||||
| } | |||||
| } | |||||
| .fps-counter { | |||||
| margin-top: 15px; | |||||
| display: flex; | |||||
| align-items: baseline; | |||||
| gap: 8px; | |||||
| span { | |||||
| font-size: 14px; | |||||
| color: var(--text-secondary); | |||||
| } | |||||
| .value { | |||||
| font-size: 24px; | |||||
| font-weight: 600; | |||||
| color: var(--accent-primary); | |||||
| } | |||||
| } | |||||
| .contributors { | |||||
| padding: 20px; | |||||
| color: var(--text-secondary); | |||||
| font-size: 14px; | |||||
| display: flex; | |||||
| flex-direction: column; | |||||
| gap: 15px; | |||||
| .contributor { | |||||
| padding: 5px 12px; | |||||
| border-radius: 6px; | |||||
| background-color: var(--bg-tertiary); | |||||
| } | |||||
| .contributor-link { | |||||
| color: var(--accent-primary); | |||||
| text-decoration: none; | |||||
| &:hover { | |||||
| text-decoration: underline; | |||||
| } | |||||
| } | |||||
| } | |||||
| /* 主内容区域样式 */ | |||||
| .content-area { | |||||
| flex: 1; | |||||
| display: flex; | |||||
| flex-direction: column; | |||||
| overflow: hidden; | |||||
| } | |||||
| .header-bar { | |||||
| display: flex; | |||||
| justify-content: space-between; | |||||
| align-items: center; | |||||
| padding: 25px 30px 15px; | |||||
| border-bottom: 1px solid var(--border-color); | |||||
| } | |||||
| .title-container { | |||||
| display: flex; | |||||
| flex-direction: column; | |||||
| } | |||||
| .view-title { | |||||
| font-size: 28px; | |||||
| font-weight: 700; | |||||
| margin: 0; | |||||
| color: var(--text-primary); | |||||
| line-height: 1.2; | |||||
| } | |||||
| .view-description { | |||||
| font-size: 15px; | |||||
| color: var(--text-secondary); | |||||
| margin-top: 6px; | |||||
| } | |||||
| .controls { | |||||
| display: flex; | |||||
| gap: 15px; | |||||
| } | |||||
| .theme-toggle { | |||||
| width: 40px; | |||||
| height: 40px; | |||||
| border-radius: 50%; | |||||
| background: var(--bg-secondary); | |||||
| border: none; | |||||
| display: flex; | |||||
| align-items: center; | |||||
| justify-content: center; | |||||
| cursor: pointer; | |||||
| font-size: 18px; | |||||
| color: var(--text-primary); | |||||
| transition: all 0.2s; | |||||
| &:hover { | |||||
| background: var(--hover-bg); | |||||
| transform: scale(1.05); | |||||
| } | |||||
| } | |||||
| .main-content { | |||||
| flex: 1; | |||||
| overflow: auto; | |||||
| padding: 30px; | |||||
| } | |||||
| /* 添加路由视图过渡效果 */ | |||||
| .router-view-container { | |||||
| transition: opacity 0.3s ease; | |||||
| > * { | |||||
| animation: fadeIn 0.5s ease; | |||||
| } | |||||
| } | |||||
| @keyframes fadeIn { | |||||
| from { opacity: 0; transform: translateY(10px); } | |||||
| to { opacity: 1; transform: translateY(0); } | |||||
| } | |||||
| </style> | |||||
+ 95
- 0
src/view/mainWindow/Update.vue
View File
| @ -0,0 +1,95 @@ | |||||
| <template> | |||||
| <div class="update-container"> | |||||
| <n-card title="开发" hoverable> | |||||
| <n-space vertical size="large"> | |||||
| <div class="dev-section"> | |||||
| <n-alert type="info" title="开发者招募"> | |||||
| <template #icon> | |||||
| <n-icon><code /></n-icon> | |||||
| </template> | |||||
| <n-space vertical> | |||||
| <span>有安卓开发的大佬吗，可以联系我。</span> | |||||
| <span>欢迎加入QQ群 | |||||
| <n-button text type="primary" tag="a" href="https://jq.qq.com/?_wv=1027&k=452246065" target="_blank"> | |||||
| 452246065 | |||||
| </n-button> | |||||
| 讨论。 | |||||
| </span> | |||||
| </n-space> | |||||
| </n-alert> | |||||
| </div> | |||||
| <div class="version-section"> | |||||
| <h3>最新版本</h3> | |||||
| <n-card embedded> | |||||
| <n-space vertical> | |||||
| <div class="version-header"> | |||||
| <n-tag type="success" size="small">v0.3.9</n-tag> | |||||
| <n-button text type="primary" tag="a" href="https://github.com/maplelost/lazyeat/releases/tag/v0.3.9" target="_blank"> | |||||
| 下载地址 | |||||
| </n-button> | |||||
| </div> | |||||
| <n-list> | |||||
| <n-list-item> | |||||
| <n-text type="success">新增</n-text> 捏合手势滚动功能 | |||||
| </n-list-item> | |||||
| <n-list-item> | |||||
| <n-text type="info">优化</n-text> 自定义手势识别区域 | |||||
| </n-list-item> | |||||
| <n-list-item> | |||||
| <n-text type="info">优化</n-text> 手势识别的准确性 | |||||
| </n-list-item> | |||||
| </n-list> | |||||
| </n-space> | |||||
| </n-card> | |||||
| </div> | |||||
| </n-space> | |||||
| </n-card> | |||||
| </div> | |||||
| </template> | |||||
| <script setup lang="ts"> | |||||
| import { NCard, NSpace, NList, NListItem, NAlert, NButton, NTag, NText, NIcon } from 'naive-ui' | |||||
| import { Code } from '@icon-park/vue-next' | |||||
| </script> | |||||
| <style scoped> | |||||
| .update-container { | |||||
| padding: 16px; | |||||
| max-width: 800px; | |||||
| margin: 0 auto; | |||||
| } | |||||
| .dev-section { | |||||
| margin-bottom: 24px; | |||||
| } | |||||
| .version-section { | |||||
| margin-bottom: 24px; | |||||
| } | |||||
| .version-section h3 { | |||||
| margin: 0 0 16px 0; | |||||
| color: #2c3e50; | |||||
| font-size: 1.2em; | |||||
| } | |||||
| .version-header { | |||||
| display: flex; | |||||
| align-items: center; | |||||
| gap: 12px; | |||||
| margin-bottom: 16px; | |||||
| } | |||||
| :deep(.n-alert .n-alert-body) { | |||||
| align-items: flex-start; | |||||
| } | |||||
| :deep(.n-card) { | |||||
| transition: all 0.3s ease; | |||||
| } | |||||
| :deep(.n-card:hover) { | |||||
| box-shadow: 0 4px 12px rgba(0, 0, 0, 0.1); | |||||
| } | |||||
| </style> | |||||
+ 93
- 0
src/view/subWindow/SubWindow.vue
View File
| @ -0,0 +1,93 @@ | |||||
| <template> | |||||
| <div class="container-sub-window"> | |||||
| <CircleProgress | |||||
| :percentage="app_store.sub_windows.progress" | |||||
| :size="100" | |||||
| :text="app_store.flag_detecting ? '暂停检测' : '继续检测'" | |||||
| :color="app_store.flag_detecting ? '#F56C6C' : '#67C23A'" | |||||
| /> | |||||
| <div style="height: 30px"> | |||||
| <span>{{ app_store.sub_windows.notification }}</span> | |||||
| </div> | |||||
| </div> | |||||
| </template> | |||||
| <script setup lang="ts"> | |||||
| import CircleProgress from "@/components/CircleProgress.vue"; | |||||
| import use_app_store from "@/store/app"; | |||||
| import { getCurrentWindow, LogicalPosition } from "@tauri-apps/api/window"; | |||||
| import { computed, ref, watch } from "vue"; | |||||
| const app_store = use_app_store(); | |||||
| const display_progress = ref(false); | |||||
| const display_notification = computed(() => { | |||||
| return !display_progress.value; | |||||
| }); | |||||
| let hideTimer: number | null = null; | |||||
| async function show_window() { | |||||
| await getCurrentWindow().show(); | |||||
| } | |||||
| async function hide_window() { | |||||
| await getCurrentWindow().hide(); | |||||
| } | |||||
| watch( | |||||
| () => app_store.sub_windows.x, | |||||
| (newVal) => { | |||||
| getCurrentWindow().setPosition( | |||||
| new LogicalPosition(newVal, app_store.sub_windows.y) | |||||
| ); | |||||
| } | |||||
| ); | |||||
| // 显示 sub-window | |||||
| watch( | |||||
| () => app_store.sub_windows.progress, | |||||
| (newVal) => { | |||||
| if (newVal) { | |||||
| display_progress.value = true; | |||||
| show_window(); | |||||
| // 清除之前的定时器 | |||||
| if (hideTimer) { | |||||
| clearTimeout(hideTimer); | |||||
| } | |||||
| // 设置新的定时器 | |||||
| hideTimer = setTimeout(() => { | |||||
| hide_window(); | |||||
| }, 300); | |||||
| } | |||||
| } | |||||
| ); | |||||
| watch( | |||||
| () => app_store.sub_windows.notification, | |||||
| (newVal) => { | |||||
| if (newVal) { | |||||
| display_progress.value = false; | |||||
| show_window(); | |||||
| // 清除之前的定时器 | |||||
| if (hideTimer) { | |||||
| clearTimeout(hideTimer); | |||||
| } | |||||
| // 设置新的定时器 | |||||
| hideTimer = setTimeout(() => { | |||||
| hide_window(); | |||||
| app_store.sub_windows.notification = ""; | |||||
| }, 1000); | |||||
| } | |||||
| } | |||||
| ); | |||||
| </script> | |||||
| <style lang="scss" scoped> | |||||
| .container-sub-window { | |||||
| width: 100%; | |||||
| height: 100%; | |||||
| display: flex; | |||||
| justify-content: center; | |||||
| align-items: center; | |||||
| flex-direction: column; | |||||
| } | |||||
| </style> | |||||
+ 7
- 0
src/vite-env.d.ts
View File
| @ -0,0 +1,7 @@ | |||||
| /// <reference types="vite/client" /> | |||||
| declare module "*.vue" { | |||||
| import type { DefineComponent } from "vue"; | |||||
| const component: DefineComponent<{}, {}, any>; | |||||
| export default component; | |||||
| } | |||||
+ 30
- 0
tsconfig.json
View File
| @ -0,0 +1,30 @@ | |||||
| { | |||||
| "compilerOptions": { | |||||
| "target": "ES2020", | |||||
| "useDefineForClassFields": true, | |||||
| "module": "ESNext", | |||||
| "lib": ["ES2020", "DOM", "DOM.Iterable"], | |||||
| "skipLibCheck": true, | |||||
| /* Bundler mode */ | |||||
| "moduleResolution": "bundler", | |||||
| "allowImportingTsExtensions": true, | |||||
| "resolveJsonModule": true, | |||||
| "isolatedModules": true, | |||||
| "noEmit": true, | |||||
| "jsx": "preserve", | |||||
| /* Linting */ | |||||
| "strict": true, | |||||
| "noUnusedLocals": true, | |||||
| "noUnusedParameters": true, | |||||
| "noFallthroughCasesInSwitch": true, | |||||
| "baseUrl": ".", | |||||
| "paths": { | |||||
| "@/*": ["src/*"] // 将 @ 映射到 src 目录 | |||||
| } | |||||
| }, | |||||
| "include": ["src/**/*.ts", "src/**/*.d.ts", "src/**/*.tsx", "src/**/*.vue"], | |||||
| "references": [{ "path": "./tsconfig.node.json" }] | |||||
| } | |||||
+ 10
- 0
tsconfig.node.json
View File
| @ -0,0 +1,10 @@ | |||||
| { | |||||
| "compilerOptions": { | |||||
| "composite": true, | |||||
| "skipLibCheck": true, | |||||
| "module": "ESNext", | |||||
| "moduleResolution": "bundler", | |||||
| "allowSyntheticDefaultImports": true | |||||
| }, | |||||
| "include": ["vite.config.ts"] | |||||
| } | |||||
+ 43
- 0
vite.config.ts
View File
| @ -0,0 +1,43 @@ | |||||
| import { defineConfig } from "vite"; | |||||
| import vue from "@vitejs/plugin-vue"; | |||||
| import path from "path"; | |||||
| import { fileURLToPath } from "url"; | |||||
| import { dirname } from "path"; | |||||
| // @ts-expect-error process is a nodejs global | |||||
| const host = process.env.TAURI_DEV_HOST; | |||||
| const __filename = fileURLToPath(import.meta.url); | |||||
| const __dirname = dirname(__filename); | |||||
| // https://vitejs.dev/config/ | |||||
| export default defineConfig(async () => ({ | |||||
| plugins: [vue()], | |||||
| // Vite options tailored for Tauri development and only applied in `tauri dev` or `tauri build` | |||||
| // | |||||
| // 1. prevent vite from obscuring rust errors | |||||
| clearScreen: false, | |||||
| // 2. tauri expects a fixed port, fail if that port is not available | |||||
| server: { | |||||
| port: 1420, | |||||
| strictPort: true, | |||||
| host: host || false, | |||||
| hmr: host | |||||
| ? { | |||||
| protocol: "ws", | |||||
| host, | |||||
| port: 1421, | |||||
| } | |||||
| : undefined, | |||||
| watch: { | |||||
| // 3. tell vite to ignore watching `src-tauri` | |||||
| ignored: ["**/src-tauri/**"], | |||||
| }, | |||||
| }, | |||||
| resolve: { | |||||
| alias: { | |||||
| "@": path.resolve(__dirname, "src"), // 使用绝对路径 | |||||
| }, | |||||
| }, | |||||
| })); | |||||
+ 0
- 306
项目测试文档.md
View File
| @ -1,306 +0,0 @@ | |||||
| # 📋 WaveControl 隔空手势控制系统 - 测试文档 | |||||
| ## 一、测试概述 | |||||
| ### 1.1 测试目标 | |||||
| 确保系统三个核心子模块功能完整、交互稳定、性能可靠，满足多场景下的非接触式人机交互需求，包括主控制平台、手语通平台、游戏控制模块。 | |||||
| ### 1.2 测试对象 | |||||
| - 主控制系统（控制面板 + 手势库管理 + 语音识别） | |||||
| - WaveSign 手语通系统（教学、评分、社区） | |||||
| - 虚拟赛车手柄系统（游戏中控制响应） | |||||
| ### 1.3 测试类型 | |||||
| | 测试类型 | 说明 | | |||||
| | ------------ | ------------------------------------------ | | |||||
| | 功能测试 | 各模块是否能完成核心功能 | | |||||
| | 集成测试 | 各子模块之间的数据流与交互是否正确 | | |||||
| | 性能测试 | 系统是否在高帧率下稳定运行、响应及时 | | |||||
| | 边界测试 | 识别模糊手势、断网、摄像头断连等异常情况 | | |||||
| | 用户体验测试 | 普通用户是否能流畅使用、易上手、有清晰反馈 | | |||||
| ## 二、测试环境 | |||||
| | 项目 | 配置 | | |||||
| | ----------- | --------------------------------- | | |||||
| | 操作系统 | Windows 10 / 11、MacOS | | |||||
| | Python 版本 | Python 3.8+ | | |||||
| | 浏览器 | Chrome / Edge | | |||||
| | 识别设备 | USB 外接摄像头 / 笔记本自带摄像头 | | |||||
| | 游戏平台 | Steam 平台《Rush Rally Origins》 | | |||||
| ## 三、测试用例设计 | |||||
| ### ✅ 主控制系统 | |||||
| | 序号 | 手势名称 | 手势动作说明 | 所属类型 | | |||||
| | ---- | ------------ | --------------------------------------------- | -------- | | |||||
| | 01 | 光标控制 | 竖起食指滑动控制光标位置 | 通用控制 | | |||||
| | 02 | 鼠标左键点击 | 食指 + 大拇指上举执行点击 | 通用控制 | | |||||
| | 03 | 滚动控制 | okay 手势（食指+拇指捏合），上下移动滚动页面 | 通用控制 | | |||||
| | 04 | 全屏控制 | 四指并拢向上 → 触发设定键（默认 f 键） | 通用控制 | | |||||
| | 05 | 退格 | 特定手势触发退格键 | 通用控制 | | |||||
| | 06 | 开始语音识别 | 六指手势触发语音识别启动 | 通用控制 | | |||||
| | 07 | 结束语音识别 | 拳头手势触发语音识别停止 | 通用控制 | | |||||
| | 08 | 暂停/继续 | 单手张开保持 1.5 秒触发暂停/继续识别 | 通用控制 | | |||||
| | 09 | 向右移动 | 拇指上抬，其余手指收回 → 控制游戏角色向右移动 | 游戏控制 | | |||||
| | 10 | 跳跃 | 食指、中指上举 → 控制跳跃动作 | 游戏控制 | | |||||
| | 11 | 右跳跃 | 拇指 + 食指 + 中指上举 → 控制右跳跃 | 游戏控制 | | |||||
| | 12 | 上一首 | 大拇指向左摆动 → 上一首音乐 | 音乐控制 | | |||||
| | 13 | 下一首 | 大拇指向右摆动 → 下一首音乐 | 音乐控制 | | |||||
| | 14 | 暂停/播放 | 比耶手势（✌️ ） → 暂停或播放音乐 | 音乐控制 | | |||||
| | 15 | 切换音乐模式 | rock 手势（🤘）→ 切换音乐/普通控制模式 | 模式切换 | | |||||
| ### 🤟 手语通 WaveSign | |||||
| #### ✅ 1. **SLClassroom（手语教室）模块** | |||||
| | 用例编号 | 用例名称 | 测试点 | 预期结果 | | |||||
| | -------- | ------------------ | --------------------------- | --------------------------- | | |||||
| | TC-SL-01 | 摄像头实时识别 | 摄像头接通后手势是否被识别 | 返回手语内容 + 实时评分动画 | | |||||
| | TC-SL-02 | 视频上传评分 | 上传手语视频后是否正常评分 | 返回分数、标准建议 | | |||||
| | TC-SL-03 | 视频课程学习流程 | 是否能顺序播放、标记已学 | 视频播放正常，课程解锁 | | |||||
| | TC-SL-04 | 翻转卡片练习 | 卡片是否翻转 + 显示正确答案 | 点击翻面后显示预设解释 | | |||||
| | TC-SL-05 | 任务式课程地图跳转 | 点击课程节点是否正确跳转 | 跳转至对应课程页 | | |||||
| #### ✅ 2. **Community（社区系统）模块** | |||||
| | 用例编号 | 用例名称 | 测试点 | 预期结果 | | |||||
| | --------- | ------------- | -------------------------- | -------------------------- | | |||||
| | TC-COM-01 | 发帖功能 | 输入文字/图片/视频发帖 | 帖子成功展示 + ID 唯一标识 | | |||||
| | TC-COM-02 | 评论功能 | 帖子下评论 + 删除 | 评论正常显示/删除 | | |||||
| | TC-COM-03 | 点赞机制 | 点赞后数值变化 | 点赞数+1，重复点则取消 | | |||||
| | TC-COM-04 | 热门话题显示 | 帖子互动数高时是否上热门区 | 热门区出现帖子 | | |||||
| | TC-COM-05 | 内容审核机制 | 敏感词是否被拦截/提示 | 给出“内容不合规”提示 | | |||||
| | TC-COM-06 | 标签推荐 | 选择话题是否推荐相关内容 | 推荐结果合理、及时加载 | | |||||
| | TC-COM-07 | 关注/取关系统 | 关注后是否成功建立关注关系 | 动态展示更新 | | |||||
| #### ✅ 3. **Schedule（日程与任务模块）** | |||||
| | 用例编号 | 用例名称 | 测试点 | 预期结果 | | |||||
| | --------- | -------------------- | -------------------------- | ------------------------- | | |||||
| | TC-SCH-01 | 添加任务清单 | 是否可设置日期/优先级 | 列表显示任务 + 状态可勾选 | | |||||
| | TC-SCH-02 | 事件提醒触发 | 设置提醒是否能按时通知 | 到时弹出提醒/响铃提示 | | |||||
| | TC-SCH-03 | 日/周/月视图切换 | 是否能无误切换不同日历视图 | 各视图正常显示 | | |||||
| | TC-SCH-04 | 删除任务是否更新视图 | 删除后是否同步更新 | 日历图/列表同步清除 | | |||||
| #### ✅ 4. **LifeServing（生活服务）模块** | |||||
| | 用例编号 | 用例名称 | 测试点 | 预期结果 | | |||||
| | -------- | ---------------- | ---------------------------------- | ---------------------------- | | |||||
| | TC-LS-01 | 发布内容管理 | 发布好物推荐/活动等信息 | 内容展示无误 | | |||||
| | TC-LS-02 | 辅助器具推荐 | 推荐列表是否分类清晰/加载正确 | 分类显示 + 图片正常 | | |||||
| | TC-LS-03 | 学习设备推荐 | 展示硬件学习工具列表 | 内容图文加载完整 | | |||||
| | TC-LS-04 | 就业信息推送 | 职位内容、公司信息展示是否完整 | 包含岗位名称、描述、联系方式 | | |||||
| | TC-LS-05 | 残障友好企业标识 | 是否加V / 标签区分 | 有“友好企业”提示 | | |||||
| | TC-LS-06 | 无障碍路线规划 | 是否能绘制无台阶/电梯路线 | 地图路径合理 | | |||||
| | TC-LS-07 | 实时公交提醒 | 路线是否加载正确/更新是否及时 | 实时展示公交进站时间 | | |||||
| | TC-LS-08 | 活动预告展示 | 亲子/文娱/演出分类信息加载是否完整 | 可报名 + 有时间地点说明 | | |||||
| #### ✅ 5. **MyPage（个人中心）模块** | |||||
| | 用例编号 | 用例名称 | 测试点 | 预期结果 | | |||||
| | -------- | ------------------ | ------------------------------- | ---------------------------- | | |||||
| | TC-MY-01 | 修改头像 | 上传头像 + 预览功能是否生效 | 显示新头像 | | |||||
| | TC-MY-02 | 修改资料 | 昵称/简介/联系方式可修改 | 保存后页面同步更新 | | |||||
| | TC-MY-03 | 收藏管理 | 收藏帖子/课程后是否能查看 | 我的收藏页显示内容 | | |||||
| | TC-MY-04 | 账号注册/登录/登出 | 是否可注册新用户 + 正确跳转状态 | 新用户进入首页，旧账号可退出 | | |||||
| | TC-MY-05 | 权限角色识别 | 是否区分普通用户/审核员角色 | 页面显示不同选项 | | |||||
| #### ✅ 6. **技术实现相关（稳定性/架构）测试** | |||||
| | 用例编号 | 用例名称 | 测试点 | 预期结果 | | |||||
| | ---------- | --------------------- | ----------------------------- | ------------------------ | | |||||
| | TC-TECH-01 | SQLite 数据读写测试 | 批量操作课程/帖子是否存取正常 | 数据不丢失，响应时间正常 | | |||||
| | TC-TECH-02 | MediaPipe模型崩溃恢复 | 强行关闭摄像头后是否重连 | 显示重连提示/自动恢复 | | |||||
| | TC-TECH-03 | Tailwind 前端样式响应 | 各分辨率下页面是否响应式变化 | 不溢出，元素自适应 | | |||||
| ### 🏎️ 游戏控制模块 | |||||
| | 用例编号 | 模块 | 用例名称 | 前置条件 | 测试步骤 | 预期结果 | 优先级 | | |||||
| | ---------- | -------- | ---------------- | ------------------------ | -------------------------------- | ---------------------------------- | ------ | | |||||
| | TC-GAME-01 | 游戏控制 | 加速动作识别 | 摄像头运行正常，程序启动 | 举起右手拇指上扬 | 车辆开始加速 | 高 | | |||||
| | TC-GAME-02 | 游戏控制 | 刹车动作识别 | 同上 | 举起左手拇指上扬 | 车辆开始减速 | 高 | | |||||
| | TC-GAME-03 | 游戏控制 | 左右转向动作识别 | 同上 | 向左倾手掌 | 车辆向左转弯 | 高 | | |||||
| | TC-GAME-04 | 游戏控制 | 手势连续识别切换 | 程序运行中 | 快速从加速 → 左转 → 刹车切换手势 | 每步操作都有反馈，游戏响应流畅 | 高 | | |||||
| | TC-GAME-05 | 游戏控制 | 识别抖动干扰测试 | 程序运行中，抖动手指 | 快速小幅度摆动手指 | 系统不误触操作，保持稳定 | 中 | | |||||
| | TC-GAME-06 | 游戏控制 | 虚拟手柄断连恢复 | 手柄模拟开启中 | 断开 vgamepad 端口 → 重连 | 程序检测中断并尝试自动恢复 | 中 | | |||||
| | TC-GAME-07 | 游戏控制 | 低帧率下性能表现 | 模拟20fps摄像头 | 尝试完成左右转、加速等动作 | 出现识别滞后，界面给出低帧警告 | 中 | | |||||
| | TC-GAME-08 | 游戏控制 | UI反馈准确性 | 摄像头运行中 | 做出加速动作，观察 UI 面板反馈 | 显示“当前动作：加速”，图像区域标亮 | 中 | | |||||
| ## 四、测试程序实现与技术支撑 | |||||
| 本项目测试除使用主系统 UI 操作外，亦开发了两套独立测试程序用于识别稳定性、输入准确性与边界场景的验证，覆盖核心逻辑路径，支撑高频回归测试与离线分析。 | |||||
| ### 4.1 手势识别测试程序（wavecontrol-test 模块） | |||||
| 项目测试脚本集中存放于 `wavecontrol-test/src/` （gesture分支）路径下，采用 TypeScript + Vue3 框架实现，通过 MediaPipe 实时检测与手势逻辑模块协同，实现系统功能验证。 | |||||
| #### 4.1.1 hand_landmark 模块 | |||||
| - **detector.ts** | |||||
| 核心手部关键点检测模块，封装对 MediaPipe 的调用逻辑，统一输出手部21个关键点的坐标、置信度等数据。 | |||||
| - 功能点：初始化摄像头流、绑定回调、封装模型参数。 | |||||
| - 用于：为 `VideoDetector.vue` 和 `gesture_handler.ts` 提供关键点数据源。 | |||||
| - **gesture_handler.ts** | |||||
| 手势解析与事件派发模块，将 landmark 数据解析为具体手势动作（如光标控制、点击、跳跃等）。 | |||||
| - 支持自定义手势库扩展。 | |||||
| - 与游戏控制模块或系统控制指令绑定。 | |||||
| - **VideoDetector.vue** | |||||
| Vue 组件封装，展示摄像头实时画面 + 可视化 landmark 点位（调试模式用）。 | |||||
| - 提供测试 UI 面板，便于调试每个手势识别过程。 | |||||
| - 集成 FPS 状态、实时识别手势结果反馈。 | |||||
| #### 4.1.2 独立运行说明与调试提示 | |||||
| - **模块定位：** | |||||
| `wavecontrol-test` 为独立测试工程，当前未集成至主项目的 UI 页面路由体系，主要用于**手势识别逻辑的单元测试与调试验证**。 | |||||
| - **运行状态说明：** | |||||
| 启动后页面加载为空白（白屏），属**正常表现**，原因如下： | |||||
| - 项目尚未渲染任何主界面组件； | |||||
| - 测试逻辑运行主要依赖控制台输出来验证关键逻辑（如 landmark 检测、手势判断等）。 | |||||
| - **调试建议：** | |||||
| 启动后打开浏览器开发者工具（快捷键 `F12` 或 `Ctrl+Shift+I`），查看： | |||||
| - 控制台（Console）：打印识别结果、错误日志、手势状态； | |||||
| - 网络（Network）：确认模型文件是否成功加载； | |||||
| - 元素（Elements）：手动挂载 `VideoDetector.vue`注入测试组件进行临时渲染调试。 | |||||
| ### 4.2 游戏控制模块测试（`test.py` 等） | |||||
| ### 4.2.1 模块定位与结构 | |||||
| 该模块为游戏控制核心动作识别的测试环境，主要用于模拟真实场景下用户的手势输入，评估系统能否准确识别特定动作（如加速、转弯、刹车等），并通过 OpenCV 实时可视化手势状态与角度变化。 | |||||
| - **所在目录：** `wavecontrol/test_py/`（`game_control` 分支） | |||||
| - **相关脚本：** | |||||
| - `test.py`：主测试入口，执行实时手势识别与状态展示。 | |||||
| - `gesture_detector.py`：封装具体的手势识别逻辑。 | |||||
| - `utils.py`：通用工具类，如角度计算、数据格式处理等。 | |||||
| - `test_xbox.py`：拟用于连接虚拟 Xbox 控制器，模拟游戏输入。 | |||||
| - `test_gui.py`：图形界面测试入口。 | |||||
| ### 4.2.2 核心测试逻辑（基于 `test.py`） | |||||
| **功能点解析：** | |||||
| - **手势状态追踪：** | |||||
| 通过 `gesture_status` 字典记录五类状态： | |||||
| - `Left Fist`、`Right Fist`：用于映射刹车/加速等动作 | |||||
| - `Left Thumb`、`Right Thumb`：映射转向或音量等功能 | |||||
| - `Angle`：表示当前手势的偏转角度，用于模拟方向盘行为或特殊动作 | |||||
| - **视觉反馈：** | |||||
| 使用 `OpenCV` (`cv2`) 实时将识别结果叠加在摄像头图像上，显示五种状态的当前值，方便快速人工验证是否识别准确。 | |||||
| - **模块化调用：** | |||||
| `GestureDetector` 实例通过 `gesture_detector.py` 导入，使得识别逻辑可独立测试、方便集成到主游戏模块。 | |||||
| ------ | |||||
| #### 4.2.3 使用说明 | |||||
| - **启动方法：** | |||||
| ``` | |||||
| python test.py | |||||
| ``` | |||||
| **运行效果：** | |||||
| - 打开摄像头窗口； | |||||
| - 实时显示识别到的五项手势状态； | |||||
| - 开发者可通过手部动作验证识别准确性。 | |||||
| - **测试目标：** | |||||
| - 验证关键手势在自然使用状态下的识别准确率； | |||||
| - 评估系统在不同光照/距离/角度条件下对“加速”“刹车”“左右转”的响应稳定性； | |||||
| - 检查 OpenCV 图像反馈是否与真实操作一致，便于回归对比。 | |||||
| ## 五、测试结果分析 | |||||
| #### 项目整体手动测试结果： | |||||
| [⌛️项目测试结果](./项目测试结果.xlsx) | |||||
| #### 主平台自动化测试结果： | |||||
| 全部测试通过，覆盖率达79% | |||||
|  | |||||
| #### WaveSign平台自动化测试结果： | |||||
| 全部测试通过，覆盖率达93% | |||||
| 具体测试设计和结果见以下链接： | |||||
| [WaveSign测试文档](https://gitee.com/wydhhh/software-engineering/blob/wavesign/WaveSign%E6%B5%8B%E8%AF%95%E6%96%87%E6%A1%A3.md) | |||||
| [WaveSign测试结果html报告](https://gitee.com/wydhhh/software-engineering/tree/wavesign/htmlcov) | |||||
| ### **5.1 测试通过率** | |||||
| - **总体通过率：** | |||||
| 在所有模块（主控制系统 + WaveSign + 游戏控制模块）共计 **50+ 条用例 × 10次测试 = 500+ 次执行记录**中，约 **88%** 的测试结果为“通过”，**12%** 的测试记录显示“未通过”。 | |||||
| - **通过率较高的模块：** | |||||
| - **主控制系统：**通过率 **>90%**，大部分基础手势（如光标控制、鼠标点击、滚动控制）表现稳定。 | |||||
| - **WaveSign 社区与个人中心模块：**通过率 **约95%**，发帖、评论、关注、资料修改等常规操作无明显问题。 | |||||
| - **通过率较低的模块：** | |||||
| - **WaveSign SLClassroom（手语教室）：**实时识别和视频上传评分测试中偶尔出现 **识别延迟** 或 **评分不稳定**，通过率约 **80%**。 | |||||
| - **游戏控制模块：**“低帧率下性能表现”与“识别抖动干扰测试”未通过率稍高（**20% 左右**），主要原因是低帧率摄像头模拟下识别滞后明显。 | |||||
| ------ | |||||
| ### **5.2 典型问题分析** | |||||
| - **（1）识别类问题：** | |||||
| - 当背景复杂或光照不足时，MediaPipe 模型识别准确率下降，个别测试记录显示 **响应时间超过0.7s**，甚至未正确识别手势。 | |||||
| - “六指手势启动语音识别”在部分测试中未触发，需要优化手势阈值。 | |||||
| - **（2）性能问题：** | |||||
| - 游戏模块在模拟 **20fps 低帧率**摄像头时，出现识别滞后，UI延迟反馈。 | |||||
| - 部分 UI 动画在高分辨率（4K）设备上存在轻微卡顿，需要进一步优化前端渲染。 | |||||
| - **（3）社区系统：** | |||||
| - 敏感词拦截逻辑偶尔误报，如普通词语被识别为敏感词。 | |||||
| - **（4）任务提醒功能：** | |||||
| - **日/周/月视图切换**测试中发现，在多任务快速切换时，少数场景下界面刷新不完全。 | |||||
| ------ | |||||
| ### **5.3 综合结论** | |||||
| - WaveControl 系统整体功能 **满足预期目标**，大部分核心功能已稳定实现。 | |||||